bulk-cdk-toolkit-extract-cdc: API changes #52040
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
area/connectors
postamar
changed the base branch from
master
to
postamar/source-mysql-pin-cdk-version
| @@ -113,8 +138,4 @@ class CdcPartitionsCreator<T : Comparable<T>>( | |||
| log.info { "Current position '$lowerBound' does not exceed target position '$upperBound'." } | |||
| return listOf(partitionReader) | |||
| } | |||
|
|
|||
| companion object { | |||
| var CDCNeedsRestart: Boolean = false | |||
There was a problem hiding this comment.
This has been replaced by an AtomicBoolean in the cdc partitions creator factory class.
| @@ -298,7 +302,6 @@ class CdcPartitionReader<T : Comparable<T>>( | |||
| } | |||
|
|
|||
| enum class CloseReason(val message: String) { | |||
| TIMEOUT("timed out"), | |||
There was a problem hiding this comment.
this wasn't being used
| startingOffset = syntheticOffset | ||
| startingSchemaHistory = null | ||
| } | ||
| } |
There was a problem hiding this comment.
The diff here is noisy, as it should be, but the changes really aren't that substantial.
| ) | ||
|
|
||
| /** Debezium Engine output, other than records of course. */ | ||
| data class DebeziumState( |
There was a problem hiding this comment.
These two types just weren't used that much and it seems that getting rid of them does improve things.
|
|
||
| data class AbortDebeziumWarmStartState(val reason: String) : InvalidDebeziumWarmStartState | ||
|
|
||
| data class ResetDebeziumWarmStartState(val reason: String) : InvalidDebeziumWarmStartState |
There was a problem hiding this comment.
This abort vs reset failure handling is going to be present in every connector with CDC if it isn't the case already, so it makes sense to recognize it in the CDK. Previously OffsetInvalidNeedsResyncIllegalStateException was used as a stopgap but using exceptions for control flow can be confusing.
postamar
force-pushed
the
postamar/bulk-cdk-extract-cdc-improvements
branch
from
37e4428 to
1565204
Compare
What
This PR makes substantial changes to the API of the extract-cdc toolkit of the Bulk CDK. Thanks @stephane-airbyte for the actionable feedback.
This PR also adds richer data types to better handle invalid debezium states, which can trigger resetting the sync.
How
The
DebeziumInputandDebeziumStatetypes are gone,DebeziumOperationsmethods are adjusted accordingly and broken down into smaller pieces. Changes then propagate toCdcPartitionsCreatorandCdcPartitionReader, and then further down to the tests.Review guide
Follow the order outlined in the previous paragraph.
User Impact
None.
Can this PR be safely reverted and rolled back?