extract-jdbc: add and adopt JdbcPartition and JdbcPartitionFactory

[postamar]

Replaces #44398

Fixes airbytehq/airbyte-internal-issues#9093

This PR makes the Bulk CDK's JDBC toolkit useful for JDBC source connectors which aren't Oracle.

The current JDBC toolkit's greatest shortcoming (and blocker for porting source-mysql) is how tightly it's coupled to the stream state value model defined by

data class DefaultJdbcStreamStateValue(
    @JsonProperty("primary_key") val primaryKey: Map<String, JsonNode> = mapOf(),
    @JsonProperty("cursors") val cursors: Map<String, JsonNode> = mapOf(),
)

This model is going to make implementing CTID or XMIN states really annoying at best! Also forget about re-using the existing state values serialized by the existing source-mysql connector...

Therefore, it's necessary to make the connector fully own the state value model and translating stream state values into SQL queries. For this purpose this PR introduces the JdbcPartition and JdbcPartitionFactory interfaces:

• A JdbcPartition object represents a specific subset of data within a table and provides a query to access it as well as the state value to use as a checkpoint for having emitted said data.
• A JdbcPartitionFactory object will map the input state value into a JdbcPartition object representing the remaining data to read in the table; the factory can optionally subdivide that into smaller partitions.
• This PR also provides default implementations of these two interfaces which makes use of DefaultJdbcStreamStateValue which can be used by naive JDBC source connectors.

A pleasant consequence of this design is that the PartitionsCreatorFactory, PartitionsCreator and PartitionReader implementations for JDBC connectors are much more database-agnostic now:

• JdbcPartitionReader is the PartitionReader implementation for JDBC source connectors and it focuses on running an arbitrary query and packaging its results; it comes in a resumable- and in a non-resumable flavour.
• JdbcPartitionsCreator is the PartitionsCreator implementation for JDBC source connectors and it focuses on computing cursor column upper bounds and fetchSize values, it comes in a sequential- and in a concurrent (i.e. splitting) flavour.
• JdbcPartitionsCreatorFactory is the PartitionsCreatorFactory implementation for JDBC source connectors and merely glues JdbcPartitionFactory to the read operation machinery.

What this means for porting legacy JDBC source connectors like source-mysql to the bulk CDK is this:

• The existing stream state value object definition from source-mysql/src/resources/internal_models.yaml needs to be kept if we are to maintain compatibility (and we really should).
• We need to implement class MySqlPartitionFactory : JdbcPartitionFactory<MySqlSharedState,MySqlStreamState,MySqlPartition> to deserialize the above state values into MySqlPartition objects
• A MySqlPartition subclass may or may not be thin wrapper around a DefaultJdbcPartition subclass, with mysql-specific fairy-dust sprinkled around it, especially to serialize internal_models.yaml objects.
• MySqlSharedState and MySqlStreamState may be even more bare-bones, who knows.

Review guide

I considered breaking down this PR into more commits but the changes mainly consists of deleting and adding whole files so I'm not sure how useful that would be. Here's a suggested file reading order:

1. JdbcPartitionsCreatorFactory.kt which has the PartitionsCreatorFactory implementations for JDBC source connectors.
2. considering that (1) heavily depends on JdbcPartitionFactory, I would look at JdbcPartitionFactory.kt and JdbcPartition.kt next, which should give an idea of what a JdbcPartition is.
3. Either supplement that with Default*.kt files for the implementations, or with JdbcPartitionsCreator.kt and JdbcPartitionReader.kt which have the PartitionsCreator and PartitionReader implementations for source connectors.
4. Look at the tests.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Aug 20, 2024 8:57pm

[postamar]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• extract-jdbc: refactor constants #44482
• bulk-cdk: refactor tests #44459
• extract-jdbc: add and adopt JdbcPartition and JdbcPartitionFactory #44458 👈
• extract-jdbc: rename PartitionsCreator and PartitionReader impls #44457
• extract-jdbc: refactor PartitionReader and PartitionsCreator impls #44456
• extract-jdbc: add and adopt JdbcSharedState and JdbcStreamState #44455
• bulk-cdk-core-extract: replace AirbyteStreamDecorator with AirbyteStreamFactory #44454
• bulk-cdk-core-extract: rewrite commentary for MetadataQuerier #44453
• extract-jdbc: better handling of NULL cursor values #44452
• extract-jdbc: cursor incremental query includes lb #44450 : 1 other dependent PR (#44451 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @postamar and the rest of your teammates on Graphite

[postamar]

Closing because I'm merging the top of the stack #44482 into master using slash-approve-and-merge.