From 166c226c4b9664d492867c2983a2c82ce97404f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9D=8E=E9=B9=8F=E7=A8=8B?= Date: Thu, 23 May 2024 19:02:04 +0800 Subject: [PATCH 1/5] eanble login with kerberos for hive --- .../connectors/seatunnel/hive/utils/HiveMetaStoreProxy.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/seatunnel-connectors-v2/connector-hive/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreProxy.java b/seatunnel-connectors-v2/connector-hive/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreProxy.java index b3c463d8042..d4d8ca3b7f7 100644 --- a/seatunnel-connectors-v2/connector-hive/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreProxy.java +++ b/seatunnel-connectors-v2/connector-hive/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreProxy.java @@ -56,9 +56,11 @@ private HiveMetaStoreProxy(ReadonlyConfig readonlyConfig) { hiveConf.addResource(new File(hiveSitePath).toURI().toURL()); } if (HiveMetaStoreProxyUtils.enableKerberos(readonlyConfig)) { + Configuration hadoopConfig = new Configuration(); + hadoopConfig.set("hadoop.security.authentication", "kerberos"); this.hiveMetaStoreClient = HadoopLoginFactory.loginWithKerberos( - new Configuration(), + hadoopConfig, readonlyConfig.get(BaseSourceConfigOptions.KRB5_PATH), readonlyConfig.get(BaseSourceConfigOptions.KERBEROS_PRINCIPAL), readonlyConfig.get(BaseSourceConfigOptions.KERBEROS_KEYTAB_PATH), From 12627aceb1dfc0300f9095e8db2227eedb0ae46c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9D=8E=E9=B9=8F=E7=A8=8B?= Date: Thu, 23 May 2024 19:49:14 +0800 Subject: [PATCH 2/5] correct the mysql cdc documents link --- docs/en/connector-v2/source/MySQL-CDC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/connector-v2/source/MySQL-CDC.md b/docs/en/connector-v2/source/MySQL-CDC.md index fd222e84942..ecf2e4dd266 100644 --- a/docs/en/connector-v2/source/MySQL-CDC.md +++ b/docs/en/connector-v2/source/MySQL-CDC.md @@ -146,7 +146,7 @@ When an initial consistent snapshot is made for large databases, your establishe - `interactive_timeout`: The number of seconds the server waits for activity on an interactive connection before closing it. See [MySQL’s documentation](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_interactive_timeout) for more details. - `wait_timeout`: The number of seconds the server waits for activity on a non-interactive connection before closing it. See [MySQL’s documentation](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout) for more details. -*For more database settings see [Debezium MySQL Connector](https://debezium.io/documentation/reference/1.9/connectors/mysql.html#setting-up-mysql)* +*For more database settings see [Debezium MySQL Connector](https://debezium.io/documentation/reference/connectors/mysql.html)* ## Data Type Mapping @@ -196,7 +196,7 @@ When an initial consistent snapshot is made for large databases, your establishe | inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. | | exactly_once | Boolean | No | false | Enable exactly once semantic. | | format | Enum | No | DEFAULT | Optional output format for MySQL CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. | -| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from MySQL server. | +| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/connectors/mysql.html#mysql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from MySQL server. | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | ## Task Example From 11f2e8cdf4923cafb9af0fe9abe7e6330beaefcc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9D=8E=E9=B9=8F=E7=A8=8B?= Date: Thu, 23 May 2024 21:47:19 +0800 Subject: [PATCH 3/5] correct dead links --- docs/en/connector-v2/source/MySQL-CDC.md | 2 +- docs/en/connector-v2/source/Postgre-CDC.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/connector-v2/source/MySQL-CDC.md b/docs/en/connector-v2/source/MySQL-CDC.md index ecf2e4dd266..5d01fc1c8ab 100644 --- a/docs/en/connector-v2/source/MySQL-CDC.md +++ b/docs/en/connector-v2/source/MySQL-CDC.md @@ -196,7 +196,7 @@ When an initial consistent snapshot is made for large databases, your establishe | inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. | | exactly_once | Boolean | No | false | Enable exactly once semantic. | | format | Enum | No | DEFAULT | Optional output format for MySQL CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. | -| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/connectors/mysql.html#mysql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from MySQL server. | +| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/connectors/mysql.html#mysql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from MySQL server. | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | ## Task Example diff --git a/docs/en/connector-v2/source/Postgre-CDC.md b/docs/en/connector-v2/source/Postgre-CDC.md index 0d00d86093c..2b397e7aa4f 100644 --- a/docs/en/connector-v2/source/Postgre-CDC.md +++ b/docs/en/connector-v2/source/Postgre-CDC.md @@ -109,7 +109,7 @@ ALTER TABLE your_table_name REPLICA IDENTITY FULL; | inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. | | exactly_once | Boolean | No | false | Enable exactly once semantic. | | format | Enum | No | DEFAULT | Optional output format for Postgre CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. | -| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/1.9/connectors/postgresql.html#postgresql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from Postgre server. | +| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from Postgre server. | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | ## Task Example From a47d0960fff17ca87996cc9262fe365b750bfb76 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9D=8E=E9=B9=8F=E7=A8=8B?= Date: Thu, 23 May 2024 21:56:12 +0800 Subject: [PATCH 4/5] correct the sqlserver documents link --- docs/en/connector-v2/source/SqlServer-CDC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/connector-v2/source/SqlServer-CDC.md b/docs/en/connector-v2/source/SqlServer-CDC.md index c9b4c3dd7ec..cbfd167682b 100644 --- a/docs/en/connector-v2/source/SqlServer-CDC.md +++ b/docs/en/connector-v2/source/SqlServer-CDC.md @@ -91,7 +91,7 @@ describes how to setup the Sql Server CDC connector to run SQL queries against S | sample-sharding.threshold | int | No | 1000 | This configuration specifies the threshold of estimated shard count to trigger the sample sharding strategy. When the distribution factor is outside the bounds specified by `chunk-key.even-distribution.factor.upper-bound` and `chunk-key.even-distribution.factor.lower-bound`, and the estimated shard count (calculated as approximate row count / chunk size) exceeds this threshold, the sample sharding strategy will be used. This can help to handle large datasets more efficiently. The default value is 1000 shards. | | inverse-sampling.rate | int | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. | | exactly_once | Boolean | No | false | Enable exactly once semantic. | -| debezium.* | config | No | - | Pass-through Debezium's properties to Debezium Embedded Engine which is used to capture data changes from SqlServer server.
See more about
the [Debezium's SqlServer Connector properties](https://debezium.io/documentation/reference/1.9/connectors/sqlserver.html#sqlserver-connector-properties) | +| debezium.* | config | No | - | Pass-through Debezium's properties to Debezium Embedded Engine which is used to capture data changes from SqlServer server.
See more about
the [Debezium's SqlServer Connector properties](https://debezium.io/documentation/reference/connectors/sqlserver.html#sqlserver-connector-properties) | | format | Enum | No | DEFAULT | Optional output format for SqlServer CDC, valid enumerations are "DEFAULT"、"COMPATIBLE_DEBEZIUM_JSON". | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details. | From 2e877868f2ccce8fa13eca844c60ca3c8035863e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9D=8E=E9=B9=8F=E7=A8=8B?= Date: Thu, 23 May 2024 22:11:51 +0800 Subject: [PATCH 5/5] correct dead links --- docs/en/connector-v2/source/Oracle-CDC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/connector-v2/source/Oracle-CDC.md b/docs/en/connector-v2/source/Oracle-CDC.md index 8f1522696ec..bd054c226b3 100644 --- a/docs/en/connector-v2/source/Oracle-CDC.md +++ b/docs/en/connector-v2/source/Oracle-CDC.md @@ -245,7 +245,7 @@ exit; | inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. | | exactly_once | Boolean | No | false | Enable exactly once semantic. | | format | Enum | No | DEFAULT | Optional output format for Oracle CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. | -| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/1.9/connectors/oracle.html#oracle-connector-properties) to Debezium Embedded Engine which is used to capture data changes from Oracle server. | +| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/connectors/oracle.html#oracle-connector-properties) to Debezium Embedded Engine which is used to capture data changes from Oracle server. | | common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details | ## Task Example