impr: Reduce memory usage of storing envelopes

[philipphofmann]

📜 Description

Reduce the memory footprint when storing envelopes to disk by writing
the envelopes in chunks by using an NSFileHandle. This addresses one
part of #3630.

💡 Motivation and Context

This addresses one
part of #3630.

💚 How did you test it?

I tested the functionality with unit tests and for the improved memory footprint by storing an envelope with two attachments, each around 10MB, and checked the physical memory peak with XCTest.

self.measure(metrics: [XCTMemoryMetric()]) {
    sut.store(envelope)
}

The improved version shows a peak of 42794.662 kB and the existing version 67832.691 kB. The improved version has around 20 MB smaller physical memory peak, which is roughly the size of the two added attachments. This is because the improved version doesn't duplicate the NSData of the attachment before writing it to disk, but instead directly writes it to disk.

You can check the full test case here, which I didn't commit because CI can't validate the memory peaks as it requires a baseline profile which doesn't work in CI.

Full Test Case

func testStoreLargeEnvelope() throws {
    let event = Event()
    let header = SentryEnvelopeHeader(id: event.eventId)
    let eventItem = SentryEnvelopeItem(event: event)
    
    let maxAttachmentSize : UInt = 20 * 1024 * 1024
    
    let path = try XCTUnwrap(Bundle(for: type(of: self)).path(forResource: "Resources/image", ofType: "png"))
    
    let attachment1 = Attachment(path: path)
    let attachment2 = Attachment(path: path)
    
    let attachmentItem1 = try XCTUnwrap(SentryEnvelopeItem(attachment: attachment1, maxAttachmentSize: maxAttachmentSize))
    let attachmentItem2 = try XCTUnwrap(SentryEnvelopeItem(attachment: attachment2, maxAttachmentSize: maxAttachmentSize))
    
    let envelope = SentryEnvelope(header: header, items: [eventItem, attachmentItem1, attachmentItem2])
    
    self.measure(metrics: [XCTMemoryMetric()]) {
        sut.store(envelope)
    }
}

📝 Checklist

You have to check all boxes before merging:

• I reviewed the submitted code.
• I added tests to verify the changes.
• No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
• I updated the docs if needed.
• Review from the native team if needed.
• No breaking change or entry added to the changelog.
• No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

[codecov]

Codecov Report

Attention: Patch coverage is 82.92683% with 7 lines in your changes missing coverage. Please review.

Project coverage is 91.376%. Comparing base (6d00d1b) to head (54fa456).

Files	Patch %	Lines
Sources/Sentry/SentryFileManager.m	76.470%	4 Missing ⚠️
Sources/Sentry/SentrySerialization.m	81.250%	3 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##              main     #4219       +/-   ##
=============================================
- Coverage   91.434%   91.376%   -0.058%     
=============================================
  Files          611       610        -1     
  Lines        49103     49029       -74     
  Branches     17732     17669       -63     
=============================================
- Hits         44897     44801       -96     
- Misses        4113      4135       +22     
  Partials        93        93               

Files	Coverage Δ
...ts/SentryTests/Helper/SentryFileManagerTests.swift	99.303% <100.000%> (+0.003%)	⬆️
...tryTests/Networking/SentryHttpTransportTests.swift	99.702% <100.000%> (+<0.001%)	⬆️
Sources/Sentry/SentrySerialization.m	87.837% <81.250%> (-0.070%)	⬇️
Sources/Sentry/SentryFileManager.m	92.750% <76.470%> (-0.532%)	⬇️

... and 20 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d00d1b...54fa456. Read the comment docs.

[brustolin]

Thanks for this.
This will help a little bit.

For a next step we should serialize the items, headers and envelope direct to file instead of data and then write the data to file.

[github-actions]

Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1223.55 ms	1245.76 ms	22.21 ms
Size	21.58 KiB	695.04 KiB	673.46 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
a2af9fa	1236.62 ms	1253.12 ms	16.50 ms
7cd187e	1239.02 ms	1261.42 ms	22.40 ms
3297d6e	1231.56 ms	1254.12 ms	22.56 ms
d0deebd	1235.17 ms	1255.54 ms	20.37 ms
3cb68af	1221.15 ms	1238.40 ms	17.25 ms
0589699	1226.90 ms	1243.96 ms	17.06 ms
efd8ddb	1208.88 ms	1234.12 ms	25.24 ms
8b39743	1258.51 ms	1263.26 ms	4.75 ms
9fa5d27	1219.86 ms	1221.71 ms	1.85 ms
c8dbe73	1230.98 ms	1253.37 ms	22.39 ms

App size

Revision	Plain	With Sentry	Diff
a2af9fa	20.76 KiB	432.87 KiB	412.11 KiB
7cd187e	20.76 KiB	401.65 KiB	380.89 KiB
3297d6e	21.58 KiB	418.44 KiB	396.86 KiB
d0deebd	21.58 KiB	542.28 KiB	520.69 KiB
3cb68af	20.76 KiB	401.60 KiB	380.84 KiB
0589699	21.58 KiB	656.60 KiB	635.02 KiB
efd8ddb	21.58 KiB	678.19 KiB	656.61 KiB
8b39743	22.85 KiB	413.98 KiB	391.13 KiB
9fa5d27	20.76 KiB	393.37 KiB	372.61 KiB
c8dbe73	21.58 KiB	615.91 KiB	594.33 KiB

Previous results on branch: impr/reduce-memory-footprint-of-storing-envelope

Startup times

Revision	Plain	With Sentry	Diff
871e525	1221.31 ms	1237.57 ms	16.26 ms
f825c91	1210.26 ms	1227.81 ms	17.55 ms

App size

Revision	Plain	With Sentry	Diff
871e525	21.58 KiB	694.83 KiB	673.25 KiB
f825c91	21.58 KiB	694.83 KiB	673.25 KiB

[philipphofmann]

For a next step we should serialize the items, headers and envelope direct to file instead of data and then write the data to file.

Yes, exactly.

[armcknight]

Great change! One question.