Add MD5 checksum integrity check

[yanchenm]

Ticket(s) Closed

• Closes #

Description

Deals with https://sentry.io/organizations/whist/issues/2963782777/events/?project=5461239&referrer=slack. Error given in Sentry is that the salt header is missing (an indicator of a malformed config file). However, upon manual inspection of all the possible candidate files for both occurrences of the error, the salt header was present in all of them. I suspect then that the cause could be because of corrupted downloads which we do not currently check for.

This PR adds MD5 checksum validation to the config download process and allows up to 3 retries on checksum mismatches.

Implementation

Adds an md5 key to the object metadata in S3.

Documentation & Tests Added

Testing Instructions

• Test that configs still persist correctly
• Check that S3 object has an md5 tag

PR Checklist

• Did the PR author fully test this PR end-to-end?
• Did one PR reviewer fully test this PR end-to-end?
• Did one PR reviewer conduct a thorough code design review?

[djsavvy]

Great catch @yanchenm. Hopefully this sheds some light onto what's going on.

Code looks great as always.

[codecov]

Codecov Report

Merging #5790 (f4163e4) into dev (a0bf931) will decrease coverage by 0.09%.
The diff coverage is 64.28%.

@@            Coverage Diff             @@
##              dev    #5790      +/-   ##
==========================================
- Coverage   54.22%   54.12%   -0.10%     
==========================================
  Files         190      190              
  Lines       24752    24696      -56     
  Branches       31       31              
==========================================
- Hits        13422    13367      -55     
- Misses      11116    11117       +1     
+ Partials      214      212       -2     

Flag	Coverage Δ		*Carryforward flag
backend-services	51.10% <64.28%> (-0.77%)	⬇️
backend-webserver	88.15% <ø> (ø)		Carriedforward from adf09ab
frontend-core-ts	69.13% <ø> (ø)		Carriedforward from adf09ab
protocol	51.81% <ø> (ø)		Carriedforward from adf09ab

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files	Coverage Δ
...ices/host-service/mandelbox/configutils/storage.go	0.00% <0.00%> (ø)
...nd/services/host-service/mandelbox/user_configs.go	58.88% <81.81%> (+1.99%)	⬆️
backend/services/host-service/httpserver.go	80.10% <0.00%> (-2.05%)	⬇️
backend/services/host-service/host-service.go	41.30% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0bf931...f4163e4. Read the comment docs.

[MauAraujo]

I tested this PR and got this warning while connecting the client app to my dev instance:

WARN Response has no supported checksum. Not validating response payload.

What causes the checksum to not be supported? Does this mean that, in this case, there is still a possibility for a mismatch that will not be detected?

Aside from that, the code is great and everything else works nice!

[yanchenm]

Yeah I saw this warning too and it's from the AWS SDK not our code. I'll look into what it means and why it happens

[philippemnoel]

Yeah I saw this warning too and it's from the AWS SDK not our code. I'll look into what it means and why it happens

according to: aws/aws-sdk-go-v2#1606 it's just an issue in v 1.25 which we are using, which has apparently been fixed. If we upgrade our go-sdk this should be fixed?

[yanchenm]

according to: aws/aws-sdk-go-v2#1606 it's just an issue in v 1.25 which we are using, which has apparently been fixed. If we upgrade our go-sdk this should be fixed?

Good catch, I'll try that