Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][test] flaky AdminApi2Test.cleanup #17861

Merged

Conversation

poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Sep 27, 2022

Fixes: #16980
Fixes: #17530

Motivation

When runs AdminApi2Test.testTopicPoliciesWithMultiBroker or BrokerServiceLookupTest.testModularLoadManagerSplitBundle, you can see some logs like this:

2022-09-27T22:21:13,414 - WARN  - [metadata-store-12286-1:LeaderElectionImpl@310] - Leadership released for /loadbalance/leader
2022-09-27T22:21:13,418 - INFO  - [metadata-store-12319-1:LeaderElectionImpl@183] - Acquired leadership on /loadbalance/leader
2022-09-27T22:21:13,418 - INFO  - [metadata-store-12319-1:PulsarService@1067] - This broker was elected leader
2022-09-27T22:21:13,418 - WARN  - [metadata-store-12319-1:LeaderElectionImpl@310] - Leadership released for /loadbalance/leader
2022-09-27T22:21:13,422 - INFO  - [metadata-store-12286-1:LeaderElectionImpl@183] - Acquired leadership on /loadbalance/leader
2022-09-27T22:21:13,422 - INFO  - [metadata-store-12286-1:PulsarService@1067] - This broker was elected leader
2022-09-27T22:21:13,423 - WARN  - [metadata-store-12286-1:LeaderElectionImpl@310] - Leadership released for /loadbalance/leader
2022-09-27T22:21:13,428 - INFO  - [metadata-store-12319-1:LeaderElectionImpl@183] - Acquired leadership on /loadbalance/leader
2022-09-27T22:21:13,428 - INFO  - [metadata-store-12319-1:PulsarService@1067] - This broker was elected leader
2022-09-27T22:21:13,428 - WARN  - [metadata-store-12319-1:LeaderElectionImpl@310] - Leadership released for /loadbalance/leader
2022-09-27T22:21:13,433 - INFO  - [metadata-store-12286-1:LeaderElectionImpl@183] - Acquired leadership on /loadbalance/leader
2022-09-27T22:21:13,433 - INFO  - [metadata-store-12286-1:PulsarService@1067] - This broker was elected leader
2022-09-27T22:21:13,433 - WARN  - [metadata-store-12286-1:LeaderElectionImpl@310] - Leadership released for /loadbalance/leader
2022-09-27T22:21:13,434 - INFO  - [TestNG-method=testModularLoadManagerSplitBundle-1:OwnershipCache@193] - Trying to acquire ownership of pulsar/test/localhost:62802/0x00000000_0xffffffff
2022-09-27T22:21:13,438 - INFO  - [metadata-store-12319-1:LeaderElectionImpl@183] - Acquired leadership on /loadbalance/leader
2022-09-27T22:21:13,438 - INFO  - [metadata-store-12319-1:PulsarService@1067] - This broker was elected leader
2022-09-27T22:21:13,438 - WARN  - [metadata-store-12319-1:LeaderElectionImpl@310] - Leadership released for /loadbalance/leader

....

Location problem

These test uses setup() and PulsarService pulsar_new = startBroker(conf_new) to create two brokers, but the both brokers use the same zkc, so both brokers will share the session of ZK, which will lead to the possibility that the two brokers will constantly elect for the leader. see code:

} else if (res.getStat().isCreatedBySelf()) {
// The existing value is different but was created from the same session
return store.delete(path, Optional.of(res.getStat().getVersion()))
.thenCompose(__ -> tryToBecomeLeader());
}

Impact

  • BrokerServiceLookupTest
constantly elect for the leader makes instability of bundle assign
error occurs
  • AdminApi2Test.cleanup
close PulsarService
close LeaderElectionService
delete zk-node `/rebalance/leader`, but this node has concurrently deleted by elect-thread
error occurs

Modifications

  • Make two brokers use different session

Documentation

  • doc-not-needed

Matching PR in forked repository

PR in forked repository:

@heesung-sn
Copy link
Contributor

LGTM.

@poorbarcode poorbarcode force-pushed the flaky/testModularLoadManagerSplitBundle branch from b9c8317 to 814ba41 Compare September 27, 2022 18:14
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Sep 27, 2022
@congbobo184 congbobo184 added this to the 2.11.0 milestone Sep 28, 2022
@poorbarcode poorbarcode changed the title [fix][flaky-test]BrokerServiceLookupTest.testModularLoadManagerSplitBundle [fix][flaky-test]AdminApi2Test.cleanup Sep 28, 2022
@poorbarcode
Copy link
Contributor Author

Hi @lhotari @nicoloboschi

Could you take a look? Thanks

Copy link
Member

@tisonkun tisonkun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

Copy link
Contributor

@liangyepianzhou liangyepianzhou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work!

@poorbarcode poorbarcode closed this Oct 9, 2022
@poorbarcode poorbarcode reopened this Oct 9, 2022
@poorbarcode poorbarcode closed this Oct 9, 2022
@poorbarcode poorbarcode reopened this Oct 9, 2022
@poorbarcode poorbarcode force-pushed the flaky/testModularLoadManagerSplitBundle branch from cc4351d to a5c5dd8 Compare October 10, 2022 02:38
@poorbarcode
Copy link
Contributor Author

rebase master

@poorbarcode poorbarcode changed the title [fix][flaky-test]AdminApi2Test.cleanup fix [flaky-test]AdminApi2Test.cleanup Oct 11, 2022
@tisonkun
Copy link
Member

@poorbarcode you may change the title to [fix][test] flaky AdminApi2Test.cleanup

@poorbarcode poorbarcode changed the title fix [flaky-test]AdminApi2Test.cleanup [fix][test] AdminApi2Test.cleanup Oct 11, 2022
@poorbarcode
Copy link
Contributor Author

Hi @tisonkun

Got it! Thanks

@poorbarcode poorbarcode changed the title [fix][test] AdminApi2Test.cleanup [fix][test] flaky AdminApi2Test.cleanup Oct 11, 2022
@poorbarcode
Copy link
Contributor Author

Can this PR merge? (^_^)

@Override
protected MetadataStoreExtended createConfigurationMetadataStore() throws MetadataStoreException {
// use MockZooKeeperSession to provide a unique session id for each instance
return new ZKMetadataStore(MockZooKeeperSession.newInstance(mockZooKeeperGlobal), MetadataStoreConfig.builder()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it shouldn't be better to make this the default behavior in MockedPulsarServiceBaseTest ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, already changed

@poorbarcode poorbarcode force-pushed the flaky/testModularLoadManagerSplitBundle branch from 3869e7a to f5c5ced Compare October 12, 2022 02:32
@codecov-commenter
Copy link

codecov-commenter commented Oct 12, 2022

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.85%. Comparing base (6c65ca0) to head (246aac1).
Report is 2367 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #17861       +/-   ##
=============================================
+ Coverage     34.91%   46.85%   +11.93%     
- Complexity     5707    17912    +12205     
=============================================
  Files           607     1573      +966     
  Lines         53396   128213    +74817     
  Branches       5712    14100     +8388     
=============================================
+ Hits          18644    60072    +41428     
- Misses        32119    61953    +29834     
- Partials       2633     6188     +3555     
Flag Coverage Δ
unittests 46.85% <ø> (+11.93%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1148 files with indirect coverage changes

@poorbarcode
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

1 similar comment
@poorbarcode
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@poorbarcode poorbarcode force-pushed the flaky/testModularLoadManagerSplitBundle branch from 5697306 to 4479b1a Compare October 13, 2022 10:09
@poorbarcode poorbarcode reopened this Oct 13, 2022
@poorbarcode
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@Technoboy- Technoboy- merged commit e365afe into apache:master Oct 14, 2022
@Technoboy- Technoboy- modified the milestones: 2.11.0, 2.12.0 Oct 14, 2022
@poorbarcode poorbarcode deleted the flaky/testModularLoadManagerSplitBundle branch October 14, 2022 01:30
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-not-needed Your PR changes do not impact docs ready-to-test type/flaky-tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky-test: AdminApi2Test.cleanup Flaky-test: BrokerServiceLookupTest.testModularLoadManagerSplitBundle