Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix re-uploading segment when the previous upload failed #9631

Merged
merged 1 commit into from
Oct 20, 2022

Conversation

Jackie-Jiang
Copy link
Contributor

When the segment upload fails after segment ZK metadata is created but before segment is assigned to the ideal state, and the following cleanup didn't run properly (e.g. hard failure, disconnect from ZK), the next upload should be able to recover the previous failure and re-assign the segment (treat the uploaded segment as a new segment)

@Jackie-Jiang Jackie-Jiang changed the title Fix reloading segment when the previous upload failed Fix re-uploading segment when the previous upload failed Oct 20, 2022
@codecov-commenter
Copy link

codecov-commenter commented Oct 20, 2022

Codecov Report

Merging #9631 (c665140) into master (3e1aba3) will decrease coverage by 1.23%.
The diff coverage is 48.00%.

@@             Coverage Diff              @@
##             master    #9631      +/-   ##
============================================
- Coverage     69.92%   68.68%   -1.24%     
+ Complexity     4930     4924       -6     
============================================
  Files          1943     1943              
  Lines        103907   103926      +19     
  Branches      15745    15750       +5     
============================================
- Hits          72653    71381    -1272     
- Misses        26153    27456    +1303     
+ Partials       5101     5089      -12     
Flag Coverage Δ
integration1 25.84% <16.00%> (+0.06%) ⬆️
integration2 ?
unittests1 67.34% <0.00%> (+0.04%) ⬆️
unittests2 15.70% <48.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ache/pinot/common/metadata/ZKMetadataProvider.java 65.74% <0.00%> (-2.04%) ⬇️
.../controller/helix/core/SegmentDeletionManager.java 73.75% <ø> (ø)
...apache/pinot/controller/api/upload/ZKOperator.java 68.67% <47.82%> (-3.14%) ⬇️
...ntroller/helix/core/PinotHelixResourceManager.java 69.85% <100.00%> (-2.18%) ⬇️
...t/core/plan/StreamingInstanceResponsePlanNode.java 0.00% <0.00%> (-100.00%) ⬇️
...ore/operator/streaming/StreamingResponseUtils.java 0.00% <0.00%> (-100.00%) ⬇️
...server/starter/helix/SegmentReloadStatusValue.java 0.00% <0.00%> (-100.00%) ⬇️
...ager/realtime/PeerSchemeSplitSegmentCommitter.java 0.00% <0.00%> (-100.00%) ⬇️
...urces/ServerReloadControllerJobStatusResponse.java 0.00% <0.00%> (-100.00%) ⬇️
...he/pinot/common/utils/grpc/GrpcRequestBuilder.java 0.00% <0.00%> (-90.91%) ⬇️
... and 141 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Contributor

@vvivekiyer vvivekiyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants