Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support zone based virtual topology assignment algorithm #2986

Merged

Conversation

MarkGaox
Copy link
Contributor

@MarkGaox MarkGaox commented Jan 3, 2025

Issues

Description

  • Here are some details about my PR, including screenshots of any UI changes:

This PR support zone based virtual topology assignment algorithm. In the rest service, introduced a configurable parameter enabling users to select the desired virtual topology assignment algorithm (e.g.,instance_based or zone_based). Additionally, implemented a zone-based assignment algorithm that directly maps physical zones to virtual zones.

Tests

  • The following tests are written for this issue:
 mvn clean install -Dmaven.test.skip.exec=true
mvn test -Dtest=TestVirtualTopologyGroupService,TestFaultZoneBasedVirtualGroupAssignment,TestClusterService,TestVirtualTopologyGroupAssignment,TestClusterAccessor -pl helix-rest

Changes that Break Backward Compatibility (Optional)

  • This PR doesn't have backward compatibility issue.

Documentation (Optional)

  • In case of new functionality, my PR adds documentation in the following wiki page:

(Link the GitHub wiki you added)

Commits

  • My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Code Quality

  • My diff has been formatted using helix-style.xml
    (helix-style-intellij.xml if IntelliJ IDE is used)

@MarkGaox MarkGaox changed the title Markgaox/support zone based virtual topology Support zone based virtual topology assignment algorithm Jan 3, 2025
Copy link
Contributor

@xyuanlu xyuanlu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had an offline discussion.
When a new host joining the cluster, the current behavior of virtual group assignment will recompute the whole virtual topology assignment, causing the whole partition assignment shuffle.
I feel like the ideal behavior should be ore sticky. When new hosts are added, the existing hosts' mapping should not change.

Copy link
Contributor

@xyuanlu xyuanlu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic looks good.
I think there are many transient in memory data structure created. We could improve the efficiency and readability. Let's talk offline.

}

// Build a deep copy of the current assignment to avoid mutating it directly.
Map<String, Set<String>> updatedAssignment = deepCopy(virtualZoneMapping);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we also build a deep copy in distributeUnassignedZones. I think either this or the one in distributeUnassignedZones can be skipped. (Need to rearrange the code though... for example, moving this in line 75 and remove the one in distributeUnassignedZones)

@xyuanlu
Copy link
Contributor

xyuanlu commented Jan 23, 2025

Generally LGTM. nit comments.

@MarkGaox
Copy link
Contributor Author

Thanks @xyuanlu for reviewing this PR. It's ready to merge.
Final commit message: Support zone based virtual topology assignment algorithm

@junkaixue junkaixue merged commit 03ca2c2 into apache:master Feb 3, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants