-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update minion task metadata ZNode path #8959
Update minion task metadata ZNode path #8959
Conversation
Codecov Report
@@ Coverage Diff @@
## master #8959 +/- ##
=============================================
- Coverage 69.70% 26.66% -43.04%
+ Complexity 4709 1 -4708
=============================================
Files 1817 1806 -11
Lines 94791 94534 -257
Branches 14178 14149 -29
=============================================
- Hits 66073 25207 -40866
- Misses 24090 66926 +42836
+ Partials 4628 2401 -2227
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally, we want to automatically move the ZNRecord from the old path to the new path. When reading the ZK metadata, if we find the old path exist:
- If the new one also exist, remove the old one
- If the new one does not exist, copy the old record to the new path, then remove the old one
pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
Outdated
Show resolved
Hide resolved
pinot-common/src/main/java/org/apache/pinot/common/minion/MinionTaskMetadataUtils.java
Outdated
Show resolved
Hide resolved
pinot-common/src/main/java/org/apache/pinot/common/minion/MinionTaskMetadataUtils.java
Outdated
Show resolved
Hide resolved
We can do 1 safely. I'd prefer we keep the current logic and discuss how to implement the proposal correctly before going that direction. |
...mmon/src/main/java/org/apache/pinot/common/minion/RealtimeToOfflineSegmentsTaskMetadata.java
Outdated
Show resolved
Hide resolved
...ot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor comment. lgtm otherwise
pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
Outdated
Show resolved
Hide resolved
664cc08
to
e800c2a
Compare
What does this PR do?
This PR updates minion task metadata ZNode path from
/MINION_TASK_METADATA/${taskType}/${tableNameWithType}
to/MINION_TASK_METADATA/${tableNameWithType}/${taskType}
.The logic is backward compatible: if there exists minion task metadata in ZNode path
/MINION_TASK_METADATA/${tableNameWithType}/${taskType}
, it keeps using it; otherwise, it uses the new ZNode path.How was the code tested?
/MINION_TASK_METADATA/${taskType}/${tableNameWithType}
, and (2) there does not exists minion task metadata.