Multithreaded Downloader

A browser based multi-threaded downloader implemented in vanilla JavaScript. Fetches parts of a file using the HTTP Range header and downloads those pieces in parallel. When the pieces have all been downloaded, the original file is re-assembled and saved in the browser’s Downloads folder.

Requirements

The downloader should fetch the file directly from the web browser without a server needed to proxy the file. The download process should not need any client software or browser plugin to be installed. It should allow for resuming an interrupted download, or at least retrying a part of the file that was interrupted.

This project will allow us to specify the number of download threads and the size of each request… so we can tune it for specific network conditions, if that is necessary.

Sends HTTP HEAD request to get the file info and calculate number of chunks
Sends HTTP GET requests with “Range: bytes=start-end” header for each chunk
Monitor the progress of each response stream

100% client side JavaScript, no plug-ins or proxy required

Methods

a[download]

WhatWG Web Streams
Memory limit
Blob

StreamSaver.js

Uses StreamSaver.js to simplify downloading the output stream.
Concatenates each response stream (in order) into a final output stream
Cross-origin dependency (mitm.html)
Service worker timeout
WhatWG Web Streams
Web Streams Polyfill

HTML5 FileSystem API

Uses Bro-fs (HTML5 Filesystem api) to temporarily save each chunk.
Concatenates all chunks once complete and triggers a[download] with final file.
HTML5 Filesystem is marked deprecated but is still commonly in use.

Usage

Constructor

The Multithread constructor accepts a single object parameter:

new MultiThread({
  url: 'http://some-url/',    // The request url
  headers: {                  // Request headers to pass-though
    'Authorization': `Bearer ${accessToken}`
  },
  fileName: 'some-file.ext',  // The final output fileName
  chunkSize: 4,               // Size of each chunk in MB
  threads: 6,                 // Number of concurrent request threads
  retries: 2,                 // Number of retry attempts
  retryDelay: 1000            // Delay before another retry attempt in ms
})

Callbacks

Multithread triggers several events that you can attach callbacks to. The callbacks will be called with a single object consisting of the following keys:

onStart({ contentLength, chunks })
onFinish({})
onProgress({ contentLength, loaded })
onError({ error })

Each individual chunk will also trigger it’s own events:

onChunkStart({ id })
onChunkFinish({ id })
onChunkProgress({ contentLength, loaded, id })
onChunkError({ id, error })

PromiseQueue provides concurrency by using a queue of promises. It will automatically retry a configurable amount of times before triggering either an “onFinish” or “onError” event.