Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[draft] [fix] [client] Could not consume messages after one partition deleted #21649

Closed
wants to merge 1 commit into from

Conversation

poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Nov 30, 2023

Motivation

Background:

  • For the client side, there is a regular task that monitors topic creation and deletion when registering a regex consumer
    • Stop subscribing after a topic has been deleted.
    • Start a new subscriber after a new topic has been created.
  • Allow for topic deletions with regex consumers #5230 stopped to recreate partitions under the partitioned topic to avoid a partition being created after it was deleted. This led to a part of the partitions being deleted.

Issue:

  • Regexp Topic & Partitioned Topics: The Pulsar client stopped subscriptions of all partitions when one partition has been deleted, even if other partitions are still there. It causes the messages in other partitions not to be fully consumed.
  • Regexp Topic: The Pulsar client tries to create subscriptions for all partitions when the consumer starts, even if some partitions have been deleted. It causes the consumer not to start and throws an error "topic not found"( the issue of the current ticket )

You can reproduce the issue by the test testConsumerAfterOnePartDeleted.

Modifications

  • Instead of stopping all partitions when a partition has been deleted, just stop the partitions which have been deleted.
  • Split GetTopicsResult into two classes:
    • GetTopicsResult: the field topics are the collection of the topic names that contains partition name. It is only used by ClientCnx.GET_TOPICS_OF_NAMESPACE.
    • CollatedGetTopicsResult: merge the topic names that are under the same Partitioned Topic, It is only used in the client internal.

TODO: Since this PR has changed a lot, the following things will be done in the subsequent PR:

  • The same test for the Reader API.
  • Collate methods for the class MultiTopicsConsumerImpl and PatternMultiTopicsConsumerImpl
  • Recheck the behavior of TopicListWatcher.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode self-assigned this Nov 30, 2023
@poorbarcode poorbarcode added this to the 3.2.0 milestone Nov 30, 2023
@poorbarcode poorbarcode added type/bug The PR fixed a bug or issue reported a bug category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost labels Nov 30, 2023
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Nov 30, 2023
@poorbarcode poorbarcode added area/client and removed doc-not-needed Your PR changes do not impact docs labels Nov 30, 2023
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Nov 30, 2023
@poorbarcode poorbarcode reopened this Nov 30, 2023
@poorbarcode poorbarcode changed the title [fix] [client] Could not consume messages after one partition deleted [draft] [fix] [client] Could not consume messages after one partition deleted Nov 30, 2023
@poorbarcode poorbarcode changed the title [draft] [fix] [client] Could not consume messages after one partition deleted [fix] [client] Could not consume messages after one partition deleted Nov 30, 2023
@poorbarcode poorbarcode changed the title [fix] [client] Could not consume messages after one partition deleted [draft] [fix] [client] Could not consume messages after one partition deleted Nov 30, 2023
@coderzc coderzc self-requested a review December 1, 2023 03:38
@Technoboy- Technoboy- modified the milestones: 3.2.0, 3.3.0 Dec 22, 2023
@coderzc coderzc modified the milestones: 3.3.0, 3.4.0 May 8, 2024
@poorbarcode
Copy link
Contributor Author

Split to 3 PRs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/client category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost doc-not-needed Your PR changes do not impact docs type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants