Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIFI-14236 Implement consistent hash for Attribute Partitioner #9709

Merged
merged 2 commits into from
Feb 10, 2025

Conversation

exceptionfactory
Copy link
Contributor

Summary

NIFI-14236 Implements consistent hashing for FlowFile Attribute-based partitioning in the Correlation Attribute Partitioner. The changes restore the original behavior prior to NiFi 1.16.0 based on the Google Guava Hashing.consistentHash method.

The implementation copies the LinearCongruentialGenerator class from Google Guava and updates applicable NOTICE files to avoid bringing in the Google Guava dependency.

Changes include a new unit test to validate expected results for attribute values.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using mvn clean install -P contrib-check
    • JDK 21

Licensing

  • New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

- Used implementation from com.google.common.hash.Hashing.consistentHash method
@joewitt
Copy link
Contributor

joewitt commented Feb 10, 2025

if i had a dollar for everytime i've ever said 'LinearCongruentialGenerator' in my life....

import static org.mockito.Mockito.when;

@ExtendWith(MockitoExtension.class)
class CorrelationAttributeParserTest {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was supposed to be CorrelationAttributePartitionerTest - i.e., Partitioner, not Parser :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks!

@sfc-gh-mpayne
Copy link

Thanks @exceptionfactory changes look good to me. As soon as we get a green build I'm happy to merge!

@markap14
Copy link
Contributor

There was a system test failure but it appears unrelated. Otherwise things look good to me. +1 will merge.

@markap14 markap14 merged commit 9b84dd4 into apache:main Feb 10, 2025
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants