[feature request] Modernize file_package.py: chunking; promise-based loading and file creation; caching/decompression -> library-js; JSON-based index

[vadimkantorov]

Grouping earlier issues:

• [question] How to await all run dependencies (data-files) to be satisfied? #14362
• Loading data packages into MODULARIZE'd modules dynamically without --pre-js #12347
• [feature request] Automatic chunking of data files by file_packager and FS (and potentially wasm files as well?) #12342
• file_packager.py to accept arguments from a file #18445

Currently it uses timers / dependenciesFullfilled callback which are not controlled by the user.

Maybe good first steps for an implementor:

• move out helper methods (mainly IDBFS caching / LZ4 decompression) produced by file_packager.py into a JS file that can be included with the main JS library
• file_packager.py should also export a JSON containing file-list and data offsets

[vadimkantorov]

Chunking becomes even more important, as github pages reject files larger than 100 Mb: https://docs.github.com/en/github/managing-large-files/working-with-large-files

E.g. TexLive small profile distribution takes 136Mb

[vadimkantorov]

It also seems that Chrome will relaunch a request for a large file, even if it's already in flight with a link-preload tag, so html-triggered preload doesn't work well... So chunking should help with that as well