[Multi-stage] Optimize query plan serialization #12370
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jackie-Jiang
added
enhancement
release-notes
xiangfu0
reviewed
Feb 5, 2024
| @@ -92,3 +72,7 @@ message MailboxMetadata { | |||
| repeated string virtualAddress = 2; | |||
| map<string, string> customProperty = 3; | |||
| } | |||
|
|
|||
| message Properties { | |||
There was a problem hiding this comment.
This is used to ser/de the map
xiangfu0
approved these changes
Feb 5, 2024
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #12370 +/- ##
============================================
- Coverage 61.71% 61.69% -0.03%
Complexity 207 207
============================================
Files 2426 2425 -1
Lines 132776 132707 -69
Branches 20541 20529 -12
============================================
- Hits 81946 81871 -75
- Misses 44817 44823 +6
Partials 6013 6013
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
walterddr
approved these changes
Feb 5, 2024
| @@ -17,24 +17,6 @@ | |||
| // under the License. | |||
| // | |||
|
|
|||
| /** | |||
There was a problem hiding this comment.
lol we should do license check better
| @@ -92,8 +92,9 @@ public static OpChain compileLeafStage(OpChainExecutionContext executionContext, | |||
| DistributedStagePlan distributedStagePlan, HelixManager helixManager, ServerMetrics serverMetrics, | |||
| QueryExecutor leafQueryExecutor, ExecutorService executorService) { | |||
| long queryArrivalTimeMs = System.currentTimeMillis(); | |||
| ServerPlanRequestContext serverContext = new ServerPlanRequestContext(distributedStagePlan, leafQueryExecutor, | |||
| executorService, executionContext.getPipelineBreakerResult()); | |||
| ServerPlanRequestContext serverContext = | |||
There was a problem hiding this comment.
seems like all auto-formatting? (skipping most of these kinds)
| @@ -204,6 +213,18 @@ void submit(long requestId, DispatchableSubPlan dispatchableSubPlan, long timeou | |||
| } | |||
| } | |||
|
|
|||
| private static class StageInfo { | |||
There was a problem hiding this comment.
This is a simple helper class to help organize the info for each stage.
|
|
||
| @BeforeMethod | ||
| @BeforeClass |
There was a problem hiding this comment.
was this change related to Ser/De or just test optimization?
There was a problem hiding this comment.
Moving the per method setup to per class is test optimization
Jackie-Jiang
force-pushed
the
optimize_ser_de
branch
from
bb47fb1 to
8b13ea0
Compare
Jackie-Jiang
force-pushed
the
optimize_ser_de
branch
from
8b13ea0 to
9de696f
Compare
Jackie-Jiang
force-pushed
the
optimize_ser_de
branch
from
9de696f to
af2c506
Compare
suyashpatel98
pushed a commit
to suyashpatel98/pinot
that referenced
this pull request
Feb 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Optimize the query plan ser-de by:
Backward Incompatible
This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version. Since we are not maintaining backward compatibility for multi-stage engine as of now, user should upgrade both brokers and servers, then start querying.