Multithreaded Downloader
A browser based multi-threaded downloader implemented in vanilla JavaScript.
Fetches parts of a file using the HTTP Range header and downloads those pieces in parallel. When the pieces have all been downloaded, the original file is re-assembled and saved in the browser's Downloads folder.
Requirements
The downloader should fetch the file directly from the web browser without a server needed to proxy the file. The download process should not need any client software or browser plugin to be installed. It should allow for resuming an interrupted download, or at least retrying a part of the file that was interrupted.
This project will allow us to specify the number of download threads and the size of each request... so we can tune it for specific network conditions, if that is necessary.
- Sends HTTP HEAD request to get the file info and calculate number of chunks
- Sends HTTP GET requests with "Range: bytes=start-end" header for each chunk
- Monitor the progress of each response stream
100% client side JavaScript, no plug-ins or proxy required
Methods
a[download]
- WhatWG Web Streams
- Memory limit
- Blob
StreamSaver.js
- Uses StreamSaver.js to simplify downloading the output stream.
- Concatenates each response stream (in order) into a final output stream
- Cross-origin dependency (mitm.html)
- Service worker timeout
- WhatWG Web Streams
- Web Streams Polyfill
HTML5 FileSystem API
- Uses Bro-fs (HTML5 Filesystem api) to temporarily save each chunk.
- Concatenates all chunks once complete and triggers a[download] with final file.
- HTML5 Filesystem is marked deprecated but is still commonly in use.
Usage
Constructor
The Multithread constructor accepts a single object parameter:
new MultiThread({
url: 'http://some-url/', // The request url
headers: { // Request headers to pass-though
'Authorization': `Bearer ${accessToken}`
},
fileName: 'some-file.ext', // The final output fileName
chunkSize: 4, // Size of each chunk in MB
threads: 6, // Number of concurrent request threads
retries: 2, // Number of retry attempts
retryDelay: 1000 // Delay before another retry attempt in ms
})
Callbacks
Multithread triggers several events that you can attach callbacks to. The callbacks will be called with a single object consisting of the following keys:
- onStart({ contentLength, chunks })
- onFinish({})
- onProgress({ contentLength, loaded })
- onError({ error })
Each individual chunk will also trigger it's own events:
- onChunkStart({ id })
- onChunkFinish({ id })
- onChunkProgress({ contentLength, loaded, id })
- onChunkError({ id, error })
PromiseQueue provides concurrency by using a queue of promises. It will automatically retry a configurable amount of times before triggering either an "onFinish" or "onError" event.
Demo
Source
