Download attempts should be retried

[mikix]

I've had the situation recently where Cerner likes to give many many tiny files as the bulk export result. Each file has no more than 200 lines and no more than one patient. (So some files are very short, like one line, if the patient doesn't have many resources.) In one recent example, the server wanted me to download over 20k files.

In this situation, the bulk-data-client starts tearing through those files as fast as it can (and by default that means 5 files at once). But that means the server quickly gets mad at it, giving 429 errors and the like.

To avoid 429 errors, I've had to reduce the parallelDownloads setting to 1. But even then, I get the occasional http hiccup (20k requests after all) like a 502 gateway error.

So my feature request is to retry download attempts in general, with some backoff for 429 results specifically.

[mikix]

I know this is heinous code, but this is what I've been using to get around this. In BulkDataClient.ts, after line 566 (inside worker()):

                        for (const i of [1, 2, 3, 4, 5]) {
                            try {
                                await this.downloadFile({
                                    file: f,
                                    fileName,
                                    onProgress: state => {
                                        Object.assign(status, state)
                                        this.emit("downloadProgress", downloadJobs.map(j => j.status))
                                    },
                                    authorize: manifest.requiresAccessToken,
                                    subFolder: status.exportType == "output" ? "" : status.exportType,
                                    exportType: status.exportType
                                })
                                break
                            }
                            catch (e: any) {
                                if (i < 5) {
                                    await wait(5000)
                                }
                                else {
                                    throw e
                                }
                            }
                        }

[mikix]

Downloads for DocRef attachments should also be retried in the same way, while we're here.