[Feature][Connector-V2][Doris] Add Doris StreamLoad sink connector #3631

TaoZex · 2022-12-02T12:11:44Z

Purpose of this pull request

#3018

Add Doris StreamLoad sink connector

Doris e2e test:

Test json format.Generate 100,000 rows of data using fake source.

Sink 100000 rows data successfully.

Test csv format.Generate 1000 rows of data using fake source.

Sink 1000 rows data successfully.

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist

...ector-doris/src/main/java/org/apache/seatunnel/connectors/doris/client/DorisSinkManager.java

...doris/src/main/java/org/apache/seatunnel/connectors/doris/client/DorisStreamLoadVisitor.java

...s-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/DorisSink.java

...onnector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/DorisSinkWriter.java

TaoZex · 2022-12-04T16:44:55Z

@EricJoy2048 @Hisoka-X @hailin0 PTAL

...nector-doris/src/main/java/org/apache/seatunnel/connectors/doris/client/DorisFlushTuple.java

...ector-doris/src/main/java/org/apache/seatunnel/connectors/doris/client/DorisSinkManager.java

...2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/client/HttpHelper.java

...2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/config/SinkConfig.java

hailin0

LGTM

...-e2e/connector-doris-e2e/src/test/java/org/apache/seatunnel/e2e/connector/doris/DorisIT.java

EricJoy2048 · 2022-12-13T02:23:40Z

@TyrantLucifer PTAL.

...doris/src/main/java/org/apache/seatunnel/connectors/doris/client/DorisStreamLoadVisitor.java

Hisoka-X

LGTM

whutpencil · 2022-12-19T07:54:47Z

@TaoZex hi, I have some question, can you add me wechat: spider360, thanks.

[Feature][Connector-V2][Doris] Add Doris StreamLoad sink connector

7dfea77

TaoZex marked this pull request as draft December 2, 2022 12:11

TaoZex added 5 commits December 2, 2022 20:49

[Feature][Connector-V2][Doris] Update code style

d8fe461

[Feature][Connector-V2][Doris] Use seatunnel format to serilize

4050d53

[Feature][Connector-V2][Doris] Add delimiterParse util

9a08512

[Feature][Connector-V2][Doris] Unify exception

454f1b5

[Feature][Connector-V2][Doris] Update error code

0094168

TaoZex marked this pull request as ready for review December 2, 2022 17:52

TaoZex and others added 2 commits December 3, 2022 01:55

[Feature][Connector-V2][Doris] Update code

45d8760

Merge branch 'apache:dev' into doris_streamload_sink

230ebfc

TyrantLucifer requested changes Dec 3, 2022

View reviewed changes

TaoZex and others added 2 commits December 4, 2022 13:15

Merge branch 'apache:dev' into doris_streamload_sink

cd41985

[Feature][Connector-V2][Doris] Update e2e code

422519f

TaoZex closed this Dec 4, 2022

TaoZex reopened this Dec 4, 2022

TaoZex added 2 commits December 4, 2022 22:58

[Feature][Connector-V2][Doris] Update code

ecf85d7

[Feature][Connector-V2][Doris] Use String format

cccf497

TaoZex requested a review from TyrantLucifer December 4, 2022 16:22

TyrantLucifer requested changes Dec 5, 2022

View reviewed changes

TaoZex and others added 10 commits December 5, 2022 16:06

Merge branch 'apache:dev' into doris_streamload_sink

afee218

[Feature][Connector-V2][Doris] Update code

60a1150

[Feature][Connector-V2][Doris] update code

ed55d36

[Feature][Connector-V2][Doris] Update code

c7f29cd

[Feature][Connector-V2][Doris] Update code

f613bc3

Merge branch 'dev' into doris_streamload_sink

54febcf

Merge branch 'apache:dev' into doris_streamload_sink

9e11ab9

Update README.md

a5dcd93

Update quick-start-flink.md

ef3376b

Update quick-start-seatunnel-engine.md

c3b82d0

hailin0 previously approved these changes Dec 8, 2022

View reviewed changes

...-e2e/connector-doris-e2e/src/test/java/org/apache/seatunnel/e2e/connector/doris/DorisIT.java Outdated Show resolved Hide resolved

[Feature][Connector-V2][Doris] Update e2e test

2be7506

TaoZex dismissed hailin0’s stale review via 2be7506 December 8, 2022 04:22

hailin0 previously approved these changes Dec 8, 2022

View reviewed changes

[Feature][Connector-V2][Doris] Update code

c5046e7

TaoZex dismissed hailin0’s stale review via c5046e7 December 8, 2022 05:45

TaoZex requested a review from hailin0 December 8, 2022 06:09

hailin0 previously approved these changes Dec 8, 2022

View reviewed changes

TaoZex added 2 commits December 8, 2022 22:42

Merge branch 'dev' into doris_streamload_sink

6b7ae70

Merge branch 'dev' into doris_streamload_sink

4894727

TaoZex dismissed hailin0’s stale review via 4894727 December 11, 2022 12:00

TaoZex requested a review from hailin0 December 11, 2022 12:03

Hisoka-X reviewed Dec 13, 2022

View reviewed changes

...doris/src/main/java/org/apache/seatunnel/connectors/doris/client/DorisStreamLoadVisitor.java Show resolved Hide resolved

Merge branch 'apache:dev' into doris_streamload_sink

1b3d598

TaoZex closed this Dec 13, 2022

TaoZex reopened this Dec 13, 2022

Hisoka-X approved these changes Dec 13, 2022

View reviewed changes

TaoZex requested review from TyrantLucifer and hailin0 and removed request for hailin0 and TyrantLucifer December 13, 2022 11:06

EricJoy2048 approved these changes Dec 14, 2022

View reviewed changes

hailin0 approved these changes Dec 14, 2022

View reviewed changes

Hisoka-X requested a review from TyrantLucifer December 14, 2022 02:43

TyrantLucifer approved these changes Dec 14, 2022

View reviewed changes

TyrantLucifer merged commit 72158be into apache:dev Dec 14, 2022

hailin0 mentioned this pull request Dec 15, 2022

[Feature][connector][doris] Support write cdc changelog event in doris sink #3726

Closed

3 tasks

EricJoy2048 mentioned this pull request Dec 18, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2][Doris] Add Doris StreamLoad sink connector #3631

[Feature][Connector-V2][Doris] Add Doris StreamLoad sink connector #3631

TaoZex commented Dec 2, 2022 •

edited

Loading

TaoZex commented Dec 4, 2022 •

edited

Loading

hailin0 left a comment

EricJoy2048 commented Dec 13, 2022

Hisoka-X left a comment

whutpencil commented Dec 19, 2022

[Feature][Connector-V2][Doris] Add Doris StreamLoad sink connector #3631

[Feature][Connector-V2][Doris] Add Doris StreamLoad sink connector #3631

Conversation

TaoZex commented Dec 2, 2022 • edited Loading

Purpose of this pull request

Check list

TaoZex commented Dec 4, 2022 • edited Loading

hailin0 left a comment

Choose a reason for hiding this comment

EricJoy2048 commented Dec 13, 2022

Hisoka-X left a comment

Choose a reason for hiding this comment

whutpencil commented Dec 19, 2022

TaoZex commented Dec 2, 2022 •

edited

Loading

TaoZex commented Dec 4, 2022 •

edited

Loading