INT-790: Remove extra await

[ntrevino-virtru]

This is to fix large file downloads from what I understand from @ivanovSPvirtru 😅

[dmihalcik-virtru]

Hmm. This means if there is a network error in this function, it won't propagate anymore. Instead of throwing, you will have to explicitly call a _reject handler on the chunk.

• In this method: add a corresponding _reject parameter, in addition to the _resolve handler below. This then propagate an error in chunk processing to the stream assembly code here by throwing it out on the await on line 1085

      if (!chunk.decryptedChunk) {
        await new Promise((resolve, reject) => {
          chunk._resolve = resolve;
          chunk._reject = reject;
        });
      }

• Above, in updateChunkQueue, instead of throwing the error, send it to the offending slice. It is kinda hard, since you won't know which slice the error happened on in the current double-looped code. So you'll have to refactor it a bit so the handler happens within the innermost loop, and add another handler for the zip format check which just attaches it to the first item in the stride.

[ntrevino-virtru]

I think it should be hooked up properly, though not sure exactly which zip format check you meant.

[ivanovSPvirtru]

Little context

1. Without await its starts in parallel with file writing stream (decrypt and download) and population of chunks map with file chunks
2. If they run in parallel stream eats basically those chunks from map as soon as they populate so memory never overlflow. Its dowloading, populating map writing stream instantly
3. If we wait that function, its just polulate map and when its populated with file chunks then it gets writed to hard drive
4. Its fine if file less then 5 gb but if more than its reaches limit of size for browser keeping stuff in memory
   Why it didnt failed before - because it was instantly written to harddrive and removed from browser memory

[github-actions]

If these changes look good, signoff on them with:

git pull && git commit --amend --signoff && git push --force-with-lease origin

If they aren't any good, please remove them with:

git pull && git reset --hard HEAD~1 && git push --force-with-lease origin

[ivanovSPvirtru]

Unfortunetly it wont change anything. When you do Promise.All on decrypt its not paralelise decryption. I tried it donesnt give any peformnace boost.

https://github.com/opentdf/client-web/pull/256/files#diff-5c6d9ea24cc7a3470e79efacc4f7aed59b657798a7cba23499d0cfd6476b9935R970

here work that im doing for that uses workers. But without limiting promise.all of lot of chunks can freeze ui so i use pLimit.

Could you remove Promise.all for now since changes is on the way? and leave just _reject part?

[ntrevino-virtru]

Done.

[github-actions]

If these changes look good, signoff on them with:

git pull && git commit --amend --signoff && git push --force-with-lease origin

If they aren't any good, please remove them with:

git pull && git reset --hard HEAD~1 && git push --force-with-lease origin

[ivanovSPvirtru]

We dont need await here. Dowload (getPayloadSegment) doesnt need to wait for dowloaded chunk to decrypt to start to dowload next chunk. Its just slowedown file download.

stream waits for new chunk in chun map, slice and decrypt provide it. Dowload doesnt have to stop.

[ntrevino-virtru]

Thanks for the explanation! Updated!

[ntrevino-virtru]

There was still an issue with decrypts, but I was able to test in SS and it seems fine to me so far.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

100.0% Coverage
0.0% Duplication

[ivanovSPvirtru]

10gb file successfully decrypted. No memory leak

[dmihalcik-virtru]

You could rewrite it to not use .then, but the logic seems correct as is if you'd rather not