Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not sort the instances in InstancePartitions #9866

Merged

Conversation

Jackie-Jiang
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang commented Nov 28, 2022

We should not sort the instances in the generated InstancePartitions because:

  1. It will lead to different assignment result comparing to default consuming segment assignment, which makes migration hard
  2. It will lead to skewed partition assignment within replica group when numInstancesPerReplicaGroup doesn't match stream partitions - the instances with smaller names tend to get more partitions across multiple tables

One thing to call out is that after this fix the mirror'ed servers for different replica can change, thus certain segment movement can be expected.

@codecov-commenter
Copy link

codecov-commenter commented Nov 28, 2022

Codecov Report

Merging #9866 (d070c09) into master (aa839c9) will decrease coverage by 42.20%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #9866       +/-   ##
=============================================
- Coverage     70.38%   28.17%   -42.21%     
+ Complexity     5013       53     -4960     
=============================================
  Files          1972     1960       -12     
  Lines        105687   105332      -355     
  Branches      15988    15950       -38     
=============================================
- Hits          74383    29673    -44710     
- Misses        26109    72724    +46615     
+ Partials       5195     2935     -2260     
Flag Coverage Δ
integration1 25.14% <ø> (-0.01%) ⬇️
integration2 24.71% <ø> (-0.11%) ⬇️
unittests1 ?
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...nstance/InstanceReplicaGroupPartitionSelector.java 0.00% <ø> (-91.72%) ⬇️
...ain/java/org/apache/pinot/spi/utils/LoopUtils.java 0.00% <0.00%> (-100.00%) ⬇️
...in/java/org/apache/pinot/spi/utils/BytesUtils.java 0.00% <0.00%> (-100.00%) ⬇️
...java/org/apache/pinot/spi/trace/BaseRecording.java 0.00% <0.00%> (-100.00%) ⬇️
...java/org/apache/pinot/spi/trace/NoOpRecording.java 0.00% <0.00%> (-100.00%) ⬇️
...ava/org/apache/pinot/spi/config/table/FSTType.java 0.00% <0.00%> (-100.00%) ⬇️
...ava/org/apache/pinot/spi/config/user/RoleType.java 0.00% <0.00%> (-100.00%) ⬇️
...ava/org/apache/pinot/spi/data/MetricFieldSpec.java 0.00% <0.00%> (-100.00%) ⬇️
...ava/org/apache/pinot/spi/stream/StreamMessage.java 0.00% <0.00%> (-100.00%) ⬇️
...java/org/apache/pinot/common/tier/TierFactory.java 0.00% <0.00%> (-100.00%) ⬇️
... and 1345 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Jackie-Jiang Jackie-Jiang force-pushed the fix_instance_partitions_sort branch from d070c09 to 72f6483 Compare November 28, 2022 21:16
Copy link
Member

@jackjlli jackjlli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have the context on why the sort was added at the first place? I'm good with removing the sort.

@Jackie-Jiang
Copy link
Contributor Author

Do we have the context on why the sort was added at the first place? I'm good with removing the sort.

It is there in the first version when the feature is added. I think it is sorted probably because it is easier for human read. The minimize movement feature doesn't return sorted instances, so I believe we don't rely on the list to be sorted.

@Jackie-Jiang Jackie-Jiang merged commit d458668 into apache:master Nov 29, 2022
@Jackie-Jiang Jackie-Jiang deleted the fix_instance_partitions_sort branch November 29, 2022 01:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants