Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support large payload in zk put API #7364

Merged
merged 3 commits into from
Dec 9, 2021

Conversation

galalen
Copy link
Contributor

@galalen galalen commented Aug 25, 2021

Description

support large payload in ZK put API, solve issue 7305

Upgrade Notes

Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

  • Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

  • Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

  • New configuration options
  • Deprecation of configurations
  • Signature changes to public methods/interfaces
  • New plugins added or old plugins removed
  • Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

@xiangfu0
Copy link
Contributor

Thanks for adding this!

Can you add a test for this?

@@ -139,7 +140,17 @@ public SuccessResponse putData(
@ApiParam(value = "expectedVersion", required = true, defaultValue = "-1") @QueryParam("expectedVersion")
@DefaultValue("-1") String expectedVersion,
@ApiParam(value = "accessOption", required = true, defaultValue = "1") @QueryParam("accessOption")
@DefaultValue("1") String accessOption) {
@DefaultValue("1") String accessOption,
@ApiParam(value = "payload") @Nullable Payload payload
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want it to be an api param, but the actual payload for the PUT http call. The payload should be the same as the content, but not as an api param. Also we need to change content to be optional

@galalen galalen force-pushed the zk-large-payload-support branch from ffb519c to 38207ba Compare October 17, 2021 15:37
@codecov-commenter
Copy link

codecov-commenter commented Oct 17, 2021

Codecov Report

Merging #7364 (24183b3) into master (fcccf1b) will increase coverage by 0.08%.
The diff coverage is 35.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #7364      +/-   ##
============================================
+ Coverage     71.38%   71.46%   +0.08%     
- Complexity     4081     4083       +2     
============================================
  Files          1583     1583              
  Lines         81863    81868       +5     
  Branches      12238    12239       +1     
============================================
+ Hits          58434    58507      +73     
+ Misses        19466    19403      -63     
+ Partials       3963     3958       -5     
Flag Coverage Δ
integration1 29.31% <0.00%> (+0.10%) ⬆️
integration2 27.68% <0.00%> (+0.02%) ⬆️
unittests1 68.49% <ø> (+0.02%) ⬆️
unittests2 14.44% <35.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ot/controller/api/resources/ZookeeperResource.java 20.33% <31.57%> (+20.33%) ⬆️
.../controller/helix/ControllerRequestURLBuilder.java 83.33% <100.00%> (+0.13%) ⬆️
.../predicate/NotEqualsPredicateEvaluatorFactory.java 66.17% <0.00%> (-11.77%) ⬇️
...ore/query/scheduler/resources/ResourceManager.java 85.71% <0.00%> (-10.72%) ⬇️
.../pinot/core/query/scheduler/PriorityScheduler.java 78.08% <0.00%> (-5.48%) ⬇️
.../helix/core/minion/MinionInstancesCleanupTask.java 77.27% <0.00%> (-4.55%) ⬇️
.../startree/v2/builder/OffHeapSingleTreeBuilder.java 87.42% <0.00%> (-4.20%) ⬇️
...core/query/pruner/SelectionQuerySegmentPruner.java 86.36% <0.00%> (-2.28%) ⬇️
...controller/helix/core/minion/PinotTaskManager.java 65.51% <0.00%> (-1.92%) ⬇️
.../controller/helix/core/SegmentDeletionManager.java 75.60% <0.00%> (-0.82%) ⬇️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcccf1b...24183b3. Read the comment docs.

@galalen galalen force-pushed the zk-large-payload-support branch from 38207ba to 354080d Compare November 3, 2021 07:25
@galalen
Copy link
Contributor Author

galalen commented Nov 21, 2021

@Jackie-Jiang can you check the PR now, I'll be updating tests coverage

Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM otherwise

@@ -0,0 +1,101 @@
package org.apache.pinot.controller.api;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing license header

) {

if (payload != null) {
path = payload.getPath();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should only override the value when it is provided in the payload

Copy link
Contributor Author

@galalen galalen Dec 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jackie-Jiang I've updated the requested changes

@galalen galalen force-pushed the zk-large-payload-support branch from 354080d to 25b6431 Compare November 24, 2021 06:31
draft

remove query_param from payload

rename payload content to data, remove required field in query params

add test case for zk put method
@Jackie-Jiang Jackie-Jiang force-pushed the zk-large-payload-support branch from 25b6431 to 24183b3 Compare December 8, 2021 23:44
@Jackie-Jiang Jackie-Jiang force-pushed the zk-large-payload-support branch from 24183b3 to 36c2385 Compare December 9, 2021 00:59
@Jackie-Jiang Jackie-Jiang merged commit 8627819 into apache:master Dec 9, 2021
kriti-sc pushed a commit to kriti-sc/incubator-pinot that referenced this pull request Dec 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants