Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle in-memory segment metadata for index checking #10017

Merged

Conversation

Jackie-Jiang
Copy link
Contributor

When checking if pre-processing is needed during segment load, the segment metadata might not be loaded from the local file, and segment properties might not be available. This PR fixes the logic by not using the segment properties when checking if pre-processing is needed.

Copy link
Contributor

@klsince klsince left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@codecov-commenter
Copy link

codecov-commenter commented Dec 21, 2022

Codecov Report

Merging #10017 (9cc1363) into master (f2972c9) will increase coverage by 42.34%.
The diff coverage is 100.00%.

@@              Coverage Diff              @@
##             master   #10017       +/-   ##
=============================================
+ Coverage     28.05%   70.40%   +42.34%     
- Complexity       53     5683     +5630     
=============================================
  Files          1979     1991       +12     
  Lines        106930   107283      +353     
  Branches      16277    16315       +38     
=============================================
+ Hits          30001    75530    +45529     
+ Misses        73993    26476    -47517     
- Partials       2936     5277     +2341     
Flag Coverage Δ
integration1 24.99% <0.00%> (-0.25%) ⬇️
integration2 24.47% <0.00%> (+0.01%) ⬆️
unittests1 67.89% <100.00%> (?)
unittests2 13.63% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../columnminmaxvalue/ColumnMinMaxValueGenerator.java 72.60% <100.00%> (+72.60%) ⬆️
...loader/defaultcolumn/BaseDefaultColumnHandler.java 58.49% <100.00%> (+58.49%) ⬆️
...ntroller/helix/core/minion/TaskMetricsEmitter.java 76.74% <0.00%> (-9.31%) ⬇️
...tream/kafka20/server/KafkaDataServerStartable.java 72.91% <0.00%> (-6.25%) ⬇️
...nction/DistinctCountBitmapAggregationFunction.java 52.33% <0.00%> (-5.70%) ⬇️
...re/accounting/PerQueryCPUMemAccountantFactory.java 66.77% <0.00%> (-1.02%) ⬇️
...core/startree/operator/StarTreeFilterOperator.java 88.07% <0.00%> (-0.67%) ⬇️
...pache/pinot/core/query/utils/idset/EmptyIdSet.java 25.00% <0.00%> (ø)
...ot/common/function/scalar/ComparisonFunctions.java 42.85% <0.00%> (ø)
...anager/realtime/SegmentBuildTimeLeaseExtender.java 63.23% <0.00%> (ø)
... and 1362 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@KKcorps KKcorps merged commit 5dc60be into apache:master Dec 21, 2022
@Jackie-Jiang Jackie-Jiang deleted the handle_in_memory_segment_metadata branch December 21, 2022 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants