Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement mpu file upload #1190

Merged
merged 1 commit into from
Nov 11, 2024
Merged

implement mpu file upload #1190

merged 1 commit into from
Nov 11, 2024

Conversation

MariusWirtz
Copy link
Collaborator

@MariusWirtz MariusWirtz commented Nov 6, 2024

Fixes TM1 v12 (in PA aaS) constraint of 200 MB max file size in file update / create endpoints

Speeds up file upload through parallelization of multi-part upload.

@MariusWirtz
Copy link
Collaborator Author

Fixes #1186

Copy link

github-actions bot commented Nov 6, 2024

Tests completed for environment: tm1-11-cloud. Check artifacts for details.

@Hubert-Heijkers
Copy link
Collaborator

Nice one @MariusWirtz!

Quick question though, just because I don't know enough of the details, what happens if one of the requests, specifically the POST of a part, fails? Does the underlying POST repeat that request until it doesn't?

Asking because MPU, other then speeding up things, also helps deal with issues of uploading large files which could still happen whilst uploading a part. Even if the upload of such part succeeds but, due to some network glitch, that is never returned to the initiator of that request. You don't need to clean up extra parts or anything, I'll do that for you in the end, but you might need to retry uploading a part in some of these cases???

PS There is absolutely no constraint on the TM1 v12 side that limits the maximum file size, this is something that's imposed by the PA SaaS environment apparently (just saying;-).

@MariusWirtz
Copy link
Collaborator Author

Good catch @Hubert-Heijkers. Thank you

I will make sure to include a retry on the part uploads.

Copy link

github-actions bot commented Nov 8, 2024

Tests completed for environment: tm1-11-cloud. Check artifacts for details.

@MariusWirtz MariusWirtz merged commit 3c3f69a into master Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants