-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix code style and Fix e2e bug and Optimize program logic
- Loading branch information
1 parent
ff55078
commit b972922
Showing
28 changed files
with
400 additions
and
679 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -64,7 +64,7 @@ Tips: | |
Connector Options | ||
----------------- | ||
|
||
| Option | Required | Default | Type | Description | | ||
| Option | Required | Default | Type | Description | | ||
|-----------------------|----------|---------|----------|----------------------------------------------------------------------------------------------------------------| | ||
| uri | required | (none) | String | The MongoDB connection uri. | | ||
| database | required | (none) | String | The name of MongoDB database to read or write. | | ||
|
@@ -76,7 +76,6 @@ Connector Options | |
| retry.interval | optional | 1000 | Duration | Specifies the retry time interval if writing records to database failed, the unit is millisecond. | | ||
| upsert-enable | optional | false | Boolean | Whether to write documents via upsert mode. | | ||
| upsert-key | optional | (none) | List | The primary keys for upsert. Only valid in upsert mode. Keys are in `["id","name",...]` format for properties. | | ||
| transaction-enable | optional | false | Boolean | Turn on the Mongodb transaction to ensure the atomicity of the write and update operation through 2pc. | | ||
|
||
How to create a MongoDB Data synchronization jobs | ||
------------------------------------------------- | ||
|
@@ -185,40 +184,19 @@ sink { | |
} | ||
``` | ||
|
||
**Idempotent Writes** | ||
|
||
By specifying a clear primary key and using the upsert method, exactly-once write semantics can be achieved. | ||
|
||
If upsert-key is defined in the configuration, the MongoDB sink will use upsert semantics instead of regular INSERT statements. We combine the primary keys declared in upsert-key as the MongoDB reserved primary key and use upsert mode for writing to ensure idempotent writes. | ||
In the event of a failure, Seatunnel jobs will recover from the last successful checkpoint and reprocess, which may result in duplicate message processing during recovery. It is highly recommended to use upsert mode, as it helps to avoid violating database primary key constraints and generating duplicate data if records need to be reprocessed. | ||
|
||
```bash | ||
sink { | ||
MongoDB { | ||
uri = "mongodb://user:[email protected]:27017" | ||
database = "test_db" | ||
collection = "users" | ||
upsert-enable = true | ||
upsert-key = ["name","status"] | ||
schema = { | ||
fields { | ||
_id = string | ||
name = string | ||
status = string | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
**Why is it not recommended to use transactions for operation?** | ||
|
||
Although MongoDB has fully supported multi-document transactions since version 4.2, it doesn't mean that everyone should use them recklessly. | ||
Transactions are equivalent to locks, node coordination, additional overhead, and performance impact. | ||
Instead, the principle for using transactions should be: avoid using them if possible. | ||
The necessity for using transactions can be greatly avoided by designing systems rationally. | ||
|
||
**Use transactions for insertion and update operations** | ||
**Idempotent Writes** | ||
|
||
By specifying a clear primary key and using the upsert method, exactly-once write semantics can be achieved. | ||
|
||
If upsert-key is defined in the configuration, the MongoDB sink will use upsert semantics instead of regular INSERT statements. We combine the primary keys declared in upsert-key as the MongoDB reserved primary key and use upsert mode for writing to ensure idempotent writes. | ||
In the event of a failure, Seatunnel jobs will recover from the last successful checkpoint and reprocess, which may result in duplicate message processing during recovery. It is highly recommended to use upsert mode, as it helps to avoid violating database primary key constraints and generating duplicate data if records need to be reprocessed. | ||
|
||
```bash | ||
sink { | ||
|
@@ -227,8 +205,7 @@ sink { | |
database = "test_db" | ||
collection = "users" | ||
upsert-enable = true | ||
transaction-enable = true | ||
upsert-key = ["name"] | ||
upsert-key = ["name","status"] | ||
schema = { | ||
fields { | ||
_id = string | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
46 changes: 0 additions & 46 deletions
46
...org/apache/seatunnel/connectors/seatunnel/mongodb/sink/commit/CommittableTransaction.java
This file was deleted.
Oops, something went wrong.
72 changes: 0 additions & 72 deletions
72
...ache/seatunnel/connectors/seatunnel/mongodb/sink/commit/CommittableUpsertTransaction.java
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.