Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DBZ-PGYB] Decouple connector to not share the same PostgresPartition to sourcePartition mapping across tasks #24555

Closed
yugabyte-ci opened this issue Oct 22, 2024 · 1 comment · Fixed by yugabyte/debezium#163
Assignees
Labels
area/cdcsdk CDC SDK jira-originated kind/new-feature This is a request for a completely new feature priority/medium Medium priority issue

Comments

@yugabyte-ci
Copy link
Contributor

yugabyte-ci commented Oct 22, 2024

Jira Link: DB-13591

With the introduction of the parallel snapshot model, we can have multiple tasks when the snapshot mode is set to parallel. This introduces a problem at the underlying layer when the connector stores the sourceInfo for its partitions i.e. PostgresPartition objects in Kafka.

The PostgresPartition is identified by a map which has a structure {"server", topicPrefix} - currently this is the same for all the PostgresPartition objects which are created by the tasks when snapshot.mode is parallel and hence they all end up referring to the same source partition in the Kafka topic. Subsequently, what happens is that (assume that we have 2 tasks i.e. 0 and 1):

One task (task_0) completes the snapshot while the other is yet to start.
a. After completion, task_0 updates the sourceInfo saying that its snapshot is completed.
b. When task_1 starts up, it reads the same sourceInfo object and concludes that the snapshot is completed so it skips its snapshot.

The above situation will cause a data loss since task_1 will never actually take a snapshot.

@vaibhav-yb
Copy link
Contributor

vaibhav-yb commented Oct 22, 2024

This PR adds a short term fix by adding the task ID to the PostgresPartition which helps in uniquely identifying a Kafka source partition. However, the longer term fix would be to add the hash ranges to the partition as they will not change while the task ID can change and point to a different hash range if there’s any change to the distribution logic in the future.

yugabyte/debezium#163

@yugabyte-ci yugabyte-ci changed the title [DBZ-PGYB] Connector shares the same PostgresPartition to sourcePartition mapping across tasks [DBZ-PGYB] Decouple connector to not share the same PostgresPartition to sourcePartition mapping across tasks Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cdcsdk CDC SDK jira-originated kind/new-feature This is a request for a completely new feature priority/medium Medium priority issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants