-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'dev' into fix/connector-tdengine_source_errors
- Loading branch information
Showing
828 changed files
with
48,863 additions
and
5,636 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,4 +27,4 @@ | |
| -XX:MaxMetaspaceSize=2g | ||
|
|
||
| # G1GC | ||
| -XX:+UseG1GC | ||
| -XX:+UseG1GC | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -76,4 +76,5 @@ connector-tablestore | |
| connector-selectdb-cloud | ||
| connector-hbase | ||
| connector-amazonsqs | ||
| connector-easysearch | ||
| --end-- | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Connector check command usage | ||
|
|
||
| ## Command Entrypoint | ||
|
|
||
| ```shell | ||
| bin/seatunnel-connector.sh | ||
| ``` | ||
|
|
||
| ## Options | ||
|
|
||
| ```text | ||
| Usage: seatunnel-connector.sh [options] | ||
| Options: | ||
| -h, --help Show the usage message | ||
| -l, --list List all supported plugins(sources, sinks, transforms) | ||
| (default: false) | ||
| -o, --option-rule Get option rule of the plugin by the plugin | ||
| identifier(connector name or transform name) | ||
| -pt, --plugin-type SeaTunnel plugin type, support [source, sink, | ||
| transform] | ||
| ``` | ||
|
|
||
| ## Example | ||
|
|
||
| ```shell | ||
| # List all supported connectors(sources and sinks) and transforms | ||
| bin/seatunnel-connector.sh -l | ||
| # List all supported sinks | ||
| bin/seatunnel-connector.sh -l -pt sink | ||
| # Get option rule of the connector or transform by the name | ||
| bin/seatunnel-connector.sh -o Paimon | ||
| # Get option rule of paimon sink | ||
| bin/seatunnel-connector.sh -o Paimon -pt sink | ||
| ``` | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| # Event Listener | ||
|
|
||
| ## Introduction | ||
|
|
||
| The SeaTunnel provides a rich event listening feature that allows you to manage the status at which data is synchronized. | ||
| This functionality is crucial when you need to listen job running status(`org.apache.seatunnel.api.event`). | ||
| This document will guide you through the usage of these parameters and how to leverage them effectively. | ||
|
|
||
| ## Support Those Engines | ||
|
|
||
| > SeaTunnel Zeta<br/> | ||
| > Flink<br/> | ||
| > Spark<br/> | ||
| ## API | ||
|
|
||
| The event API is defined in the `org.apache.seatunnel.api.event` package. | ||
|
|
||
| ### Event Data API | ||
|
|
||
| - `org.apache.seatunnel.api.event.Event` - The interface for event data. | ||
| - `org.apache.seatunnel.api.event.EventType` - The enum for event type. | ||
|
|
||
| ### Event Listener API | ||
|
|
||
| You can customize event handler, such as sending events to external systems | ||
|
|
||
| - `org.apache.seatunnel.api.event.EventHandler` - The interface for event handler, SPI will automatically load subclass from the classpath. | ||
|
|
||
| ### Event Collect API | ||
|
|
||
| - `org.apache.seatunnel.api.source.SourceSplitEnumerator` - Attached event listener API to report events from `SourceSplitEnumerator`. | ||
|
|
||
| ```java | ||
| package org.apache.seatunnel.api.source; | ||
|
|
||
| public interface SourceSplitEnumerator { | ||
|
|
||
| interface Context { | ||
|
|
||
| /** | ||
| * Get the {@link org.apache.seatunnel.api.event.EventListener} of this enumerator. | ||
| * | ||
| * @return | ||
| */ | ||
| EventListener getEventListener(); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| - `org.apache.seatunnel.api.source.SourceReader` - Attached event listener API to report events from `SourceReader`. | ||
|
|
||
| ```java | ||
| package org.apache.seatunnel.api.source; | ||
|
|
||
| public interface SourceReader { | ||
|
|
||
| interface Context { | ||
|
|
||
| /** | ||
| * Get the {@link org.apache.seatunnel.api.event.EventListener} of this reader. | ||
| * | ||
| * @return | ||
| */ | ||
| EventListener getEventListener(); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| - `org.apache.seatunnel.api.sink.SinkWriter` - Attached event listener API to report events from `SinkWriter`. | ||
|
|
||
| ```java | ||
| package org.apache.seatunnel.api.sink; | ||
|
|
||
| public interface SinkWriter { | ||
|
|
||
| interface Context { | ||
|
|
||
| /** | ||
| * Get the {@link org.apache.seatunnel.api.event.EventListener} of this writer. | ||
| * | ||
| * @return | ||
| */ | ||
| EventListener getEventListener(); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Configuration Listener | ||
|
|
||
| To use the event listening feature, you need to configure engine config. | ||
|
|
||
| ### Zeta Engine | ||
|
|
||
| Example config in your config file(seatunnel.yaml): | ||
|
|
||
| ``` | ||
| seatunnel: | ||
| engine: | ||
| event-report-http: | ||
| url: "http://example.com:1024/event/report" | ||
| headers: | ||
| Content-Type: application/json | ||
| ``` | ||
|
|
||
| ### Flink Engine | ||
|
|
||
| You can define the implementation class of `org.apache.seatunnel.api.event.EventHandler` interface and add to the classpath to automatically load it through SPI. | ||
|
|
||
| Support flink version: 1.14.0+ | ||
|
|
||
| Example: `org.apache.seatunnel.api.event.LoggingEventHandler` | ||
|
|
||
| ### Spark Engine | ||
|
|
||
| You can define the implementation class of `org.apache.seatunnel.api.event.EventHandler` interface and add to the classpath to automatically load it through SPI. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.