[DBZ-PGYB] Decouple connector to not share the same PostgresPartition
to sourcePartition
mapping across tasks
#24555
Labels
area/cdcsdk
CDC SDK
jira-originated
kind/new-feature
This is a request for a completely new feature
priority/medium
Medium priority issue
Jira Link: DB-13591
With the introduction of the parallel snapshot model, we can have multiple tasks when the snapshot mode is set to parallel. This introduces a problem at the underlying layer when the connector stores the sourceInfo for its partitions i.e.
PostgresPartition
objects in Kafka.The
PostgresPartition
is identified by a map which has a structure{"server", topicPrefix}
- currently this is the same for all thePostgresPartition
objects which are created by the tasks whensnapshot.mode
isparallel
and hence they all end up referring to the same source partition in the Kafka topic. Subsequently, what happens is that (assume that we have 2 tasks i.e. 0 and 1):One task (task_0) completes the snapshot while the other is yet to start.
a. After completion, task_0 updates the sourceInfo saying that its snapshot is completed.
b. When task_1 starts up, it reads the same sourceInfo object and concludes that the snapshot is completed so it skips its snapshot.
The above situation will cause a data loss since task_1 will never actually take a snapshot.
The text was updated successfully, but these errors were encountered: