Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically update broker resource on broker changes #8249

Merged
merged 1 commit into from
Feb 28, 2022

Conversation

Jackie-Jiang
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang commented Feb 25, 2022

Description

Fix #7578

Currently, when a new broker joins the cluster, or the tags for a broker is changed, broker resource won't be updated automatically. Users need to either rebuild broker resource for each table in that broker tenant, or wait for the periodic task to fix the broker resource. Before that, the changes for the broker won't be reflected.

This PR makes the following enhancements:

  • Add a boolean flag updateBrokerResource to create/update instance rest APIs to automatically update the broker resource when enabled (disabled by default to keep the current behavior because updating broker resource can be costly for large cluster)
  • Add a rest API to update the broker resource for a specified broker
  • For single-tenant cluster, when broker joins the cluster for the first time as DefaultTenant, automatically update the broker resource so that it can build the routing tables properly.

Release Notes

Add an optional boolean query parameter updateBrokerResource to the following rest APIs:

  • POST /instances: Add an instance
  • PUT /instances/{instanceName}: Update an instance
  • PUT /instances/{instanceName}/updateTags: Update the tags for an instance

Add a rest API to update the broker resource for a specified broker:

  • POST /instances/{instanceName}/updateBrokerResource

For single-tenant cluster, when broker joins the cluster for the first time as DefaultTenant, automatically update the broker resource so that it can build the routing tables properly

@Jackie-Jiang Jackie-Jiang added the release-notes Referenced by PRs that need attention when compiling the next release notes label Feb 25, 2022
@codecov-commenter
Copy link

codecov-commenter commented Feb 25, 2022

Codecov Report

Merging #8249 (914b8ea) into master (27d690d) will increase coverage by 0.02%.
The diff coverage is 74.44%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #8249      +/-   ##
============================================
+ Coverage     70.74%   70.77%   +0.02%     
  Complexity     4238     4238              
============================================
  Files          1629     1629              
  Lines         85213    85364     +151     
  Branches      12830    12857      +27     
============================================
+ Hits          60287    60419     +132     
- Misses        20767    20774       +7     
- Partials       4159     4171      +12     
Flag Coverage Δ
integration1 28.87% <43.17%> (+0.03%) ⬆️
integration2 27.47% <23.34%> (-0.14%) ⬇️
unittests1 66.88% <7.22%> (-0.07%) ⬇️
unittests2 14.16% <47.57%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...er/api/resources/PinotInstanceRestletResource.java 58.41% <50.00%> (-5.79%) ⬇️
...g/apache/pinot/common/utils/helix/HelixHelper.java 51.42% <64.58%> (+6.28%) ⬆️
...ache/pinot/common/metadata/ZKMetadataProvider.java 79.86% <80.00%> (+0.72%) ⬆️
...ntroller/helix/core/PinotHelixResourceManager.java 67.01% <82.55%> (+0.95%) ⬆️
...e/pinot/broker/broker/helix/BaseBrokerStarter.java 78.17% <88.88%> (+1.95%) ⬆️
.../controller/helix/ControllerRequestURLBuilder.java 83.72% <100.00%> (+0.25%) ⬆️
...ller/helix/core/minion/TaskTypeMetricsUpdater.java 80.00% <0.00%> (-20.00%) ⬇️
...rg/apache/pinot/server/starter/ServerInstance.java 88.78% <0.00%> (-2.81%) ⬇️
...form/function/BinaryOperatorTransformFunction.java 41.77% <0.00%> (-2.67%) ⬇️
...core/query/pruner/SelectionQuerySegmentPruner.java 86.36% <0.00%> (-2.28%) ⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 27d690d...914b8ea. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-notes Referenced by PRs that need attention when compiling the next release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ensure broker rebuild is done during initialization
3 participants