[DBZ-PGYB] Use YugabyteDB smart driver instead of vanilla Postgres JDBC driver #107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
debezium-connector-postgres/src/test/java/io/debezium/connector/postgresql/TestHelper.java
Outdated
Show resolved
Hide resolved
debezium-core/src/main/java/io/debezium/relational/RelationalDatabaseConnectorConfig.java
Outdated
Show resolved
Hide resolved
...nnector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresConnectorConfig.java
Outdated
Show resolved
Hide resolved
...nnector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresConnectorConfig.java
Outdated
Show resolved
Hide resolved
...r-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresConnection.java
Show resolved
Hide resolved
...r-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresConnection.java
Show resolved
Hide resolved
...r-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresConnection.java
Show resolved
Hide resolved
...r-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresConnection.java
Show resolved
Hide resolved
...r-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresConnection.java
Show resolved
Hide resolved
...r-postgres/src/main/java/io/debezium/connector/postgresql/connection/PostgresConnection.java
Outdated
Show resolved
Hide resolved
...m-connector-postgres/src/test/java/io/debezium/connector/postgresql/PostgresConnectorIT.java
Show resolved
Hide resolved
vaibhav-yb
changed the title
| @@ -1140,6 +1140,15 @@ private static int validateFlushLsnSource(Configuration config, Field field, Fie | |||
| return 0; | |||
| } | |||
|
|
|||
| public JdbcConnection.ConnectionFactory getConnectionFactory() { | |||
There was a problem hiding this comment.
Can this be a static method. This can then be called from PostgresConnection as well (instead of duplicating the code there)
There was a problem hiding this comment.
It cannot be a static method since the methods/attributes this is accessing are non-static.
| @@ -85,6 +86,7 @@ public class PostgresConnection extends JdbcConnection { | |||
|
|
|||
| private final TypeRegistry typeRegistry; | |||
| private final PostgresDefaultValueConverter defaultValueConverter; | |||
| private final JdbcConfiguration jdbcConfig; | |||
There was a problem hiding this comment.
why is this new instance variable required ?
There was a problem hiding this comment.
This is required as we access the hostname using jdbcConfig.hostname() in the helper methods of this class.
| @@ -2860,6 +2860,39 @@ public void shouldOutputRecordsInCloudEventsFormat() throws Exception { | |||
| } | |||
| } | |||
|
|
|||
| // YB Note: Running this test also requires a change of values in the method TestHelper#defaultJdbcConfig | |||
There was a problem hiding this comment.
Could add a comment here that this test is for manual testing.
Also instructions on what to do and when
asrinivasanyb
approved these changes
Apr 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The Debezium connector for Postgres uses a single host model where the JDBC driver connects to a PG instance and continues execution. However, when we move to YugabyteDB where we have a multi node deployment, the current model can fail in case the node it has connected to goes down.
Solution
To address that, we have made changes in this PR and replaced the Postgres JDBC driver with YugabyteDB smart driver which allows us to specify multiple hosts in the JDBC url so that the connector does not fail or run into any fatal error while maintaining the High Availability aspect of YugabyteDB.
Changes in this PR include:
pom.xmlfrom2.5.2.Finalto2.5.2.ybpg.20241-SNAPSHOTa. This is done to ensure that upon image compilation, the changed code from Debezium Code is picked up.
org.postgresql.*tocom.yugabyte.*to comply with the new JDBC driver.: (colon)in the configuration propertydatabase.hostname