fix: dont buffer on azurerehydration receiver #2105
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dpaasman00
reviewed
Jan 17, 2025
There was a problem hiding this comment.
Should it still check if the output is ok when it pulls from the channel? Doesn't matter to me just wondering
There was a problem hiding this comment.
I think its probably safer in this case if ever the blob_client doesn't close the channel properly in the future as to not panic on closed read. Just think in principle it's a "better safe than sorry" kind of check imo
dpaasman00
approved these changes
Jan 17, 2025
mrsillydog
added a commit
that referenced
this pull request
Jan 21, 2025
* chore: Update modules to v1.69.0 * feat: Make SSAPI receiver storage optional (#2099) * make storage extn optional * update storage test * fix: update readme * fix: Localize `Users` group in Windows installer (BPOP-1396) (#2102) user group var * feat: Add `cgroupruntime` extension (#2104) * add cgroupruntime extension to collector facotries list * add doc * fix: refactor azureeventhubrehydrationreceiver to stream blobs as to not lock up on larger environments (BPOP-831) (#2098) * pre tests working; refactor to stream blobs as to not lock up * fix tests * remove polling parameters and fix gosec error * some more tests * add license * remove extra debug line * address PR feedback * only log messages rather than submit validation errors * spin up goroutine per blob in the batch, change default batch size to 30 to keep a moderate default * add buffered chan size of 5 to start * remove 3 empty request limit * remove testing log line * harden shutdown logic to close channel with a timeout * fix lint * dakota PR feedback * more feedback; sans checkpointing after every blob * minor updates; pr review * fix: dont buffer on azurerehydration receiver (#2105) dont buffer on azurerehydration receiver * fix(awss3rehydrationreceiver): Stream objects from S3 to improve performance (#2103) * refactor to use pagination & concurrency * save checkpoint after processing a batch * fix lint * dont lock before early return * deps(otel): Update OTel dep to v0.117.0 (#2109) * update otel dep to v0.117.0 * update opencontainers/runc to v1.1.15 * revert opencontainers update * replace cilium pkg import * deps: v1.69.0 (#2112) * deps(deps): bump github.com/vektra/mockery/v2 from 2.50.0 to 2.51.1 in /internal/tools * bump google.golang.org/grpc from 1.69.2 to 1.69.4 and google.golang.org/protobuf from 1.36.2 to 1.36.3 in /exporter/chronicleexporter * bump github.com/open-telemetry/opamp-go from 0.17.0 to 0.18.0 in /extension/bindplaneextension * deps(deps): bump google.golang.org/api from 0.214.0 to 0.217.0 in /exporter/googlecloudexporter * bump google.golang.org/api from 0.214.0 to 0.217.0 in /exporter/googlemanagedprometheusexporter --------- Co-authored-by: Dakota Paasman <[email protected]> Co-authored-by: Caleb Hurshman <[email protected]> Co-authored-by: Keith Schmitt <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I believe we want to block on the channel read as the doneChan gets closed when the stream is over.
Thanks to @dpaasman00 for catching this.
Proposed Change
Checklist