[Feature][Connectors-v2]Paimon version upgrade to 1.0.1 #8074
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Hisoka-X
reviewed
Nov 18, 2024
hawk9821
force-pushed
the
paimon_version_upgrade
branch
2 times, most recently
from
0292734 to
853960f
Compare
Hisoka-X
reviewed
Nov 20, 2024
...ctor-paimon-e2e/src/test/java/org/apache/seatunnel/e2e/connector/paimon/PaimonSinkCDCIT.java
Outdated
Show resolved
Hide resolved
...nel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/fake_cdc_sink_paimon_case9.conf
Outdated
Show resolved
Hide resolved
dailai
reviewed
Nov 22, 2024
...ctor-paimon-e2e/src/test/java/org/apache/seatunnel/e2e/connector/paimon/PaimonSinkCDCIT.java
Outdated
Show resolved
Hide resolved
hawk9821
force-pushed
the
paimon_version_upgrade
branch
3 times, most recently
from
5fa7e4e to
690c856
Compare
|
I think we should keep the default behavior of seatunnel when writing to paimon which is that we still used fixed buckets and the default number of buckets is 1, and we should also mark dynamic buckets as experimental properties. |
hawk9821
force-pushed
the
paimon_version_upgrade
branch
5 times, most recently
from
060fd30 to
9c5377f
Compare
hawk9821
changed the title
Hisoka-X
reviewed
Feb 13, 2025
...on/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/catalog/PaimonCatalog.java
Outdated
Show resolved
Hide resolved
| @@ -216,7 +229,9 @@ private Schema buildPaimonSchema(@NonNull org.apache.paimon.schema.TableSchema s | |||
| Schema.Builder builder = Schema.newBuilder(); | |||
| schema.fields() | |||
| .forEach(field -> builder.column(field.name(), field.type(), field.description())); | |||
| builder.options(schema.options()); | |||
| Map<String, String> options = new HashMap<>(schema.options()); | |||
| options.remove(CoreOptions.PATH.key()); | |||
There was a problem hiding this comment.
Please add a comment of why remove this?
There was a problem hiding this comment.
The paimon interface has changed in the new version. it will throw exception when not remove
java.lang.UnsupportedOperationException: The current catalog FileSystemCatalog does not support specifying the table path when creating a table.
at org.apache.paimon.catalog.AbstractCatalog.validateCustomTablePath(AbstractCatalog.java:582)
at org.apache.paimon.catalog.AbstractCatalog.createTable(AbstractCatalog.java:285)
at org.apache.paimon.catalog.DelegateCatalog.createTable(DelegateCatalog.java:106)
| @@ -128,10 +128,9 @@ public PaimonSinkWriter( | |||
| this.sinkPaimonTableSchema = this.paimonFileStoretable.schema(); | |||
| this.newTableWrite(); | |||
| BucketMode bucketMode = this.paimonFileStoretable.bucketMode(); | |||
| this.dynamicBucket = | |||
| BucketMode.DYNAMIC == bucketMode || BucketMode.GLOBAL_DYNAMIC == bucketMode; | |||
There was a problem hiding this comment.
The paimon interface has changed in the new version
public enum BucketMode {
HASH_FIXED,
HASH_DYNAMIC,
CROSS_PARTITION,
BUCKET_UNAWARE;
...
}
Comment on lines
+73
to
+75
| paimon.table.write-props = { | ||
| bucket = 1 | ||
| } |
Co-authored-by: Jia Fan <[email protected]>
hawk9821
force-pushed
the
paimon_version_upgrade
branch
from
a6c0a80 to
c284d30
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of this pull request
doris has so far supported the Piamon .
doris-paimon, Paimon version upgrade to 1.0.1 .Does this PR introduce any user-facing change?
How was this patch tested?
PaimonSinkDynamicBucketIT#testParallelismWrite
Check list
New License Guide
release-note.