[Discuss][RoadMap] SeaTunnel RoadMap

[EricJoy2048]

Hi everyone.
Here we can discuss the roadmap of SeaTunnel in Q4.

SeaTunnel API & SeaTunnel Connector

• CDC Connector MySQL [Feature]Support CDC #2394
• Transform API [Feature][API] SeaTunnel Transform API #2678
• Transform V2 Flink Translation [Feature][starter][flink] Support transform-v2 for flink #3267
• Transform V2 Spark Translation [Feature][starter][spark] Support transform-v2 for spark #3268
• Database Synchronization Connector (batch mode)
• More Source & Sink & Transform Connector [Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018
• Support Connector Metrics
• Support TableSourceFactory and can get the Connector Option Rules.
• Support TableSinkFactory and can get the Connector Option Rules.

SeaTunnel Web Server

submodule: https://github.com/apache/incubator-seatunnel-web

• [Feature][Web-Server] Add query SeaTunnel support Source/Sink/Transform List API. #3374
• Release MVP version, and improve the fellings.

SeaTunnel Engine

• Release MVP version
• Support Transform Action
• Support EventType API(For CDC connector we need know Update/Insert/Delete type)
• Run on k8s
• Performance testing & Performance optimization
• Support IMap Storage By JDBC [Discuss][CheckPoint Storage & IMap Storage] Support use JDBC to storage the checkpoint state data and IMap data #3870
• [Discuss][Core]Support Cache between source and sink #3871

Spark/Flink new version support

• [Improve][e2e] Improve end-to-end testing of connector-v2 in a multi-engine environment #2733
• Support Spark3 [Feature][Seatunnel-translation] Support Spark3.3.x #2189
• Support Flink 1.15.X [Bug] [seatunnel-api-flink] Can not find BatchTableEnvironment class when using Flink-1.15.X #2927

CDC

• [Feature][CDC] Support flink running cdc job #4605
• [Feature][CDC] Support spark running cdc job  #4604

[ashulin]

Improve end-to-end testing of connector-v2 in a multi-engine environment;
Now developers need to write a lot of repeated code to test different engines, if we support more version engine, this will generate more repeated test code;

[dijiekstra]

SeaTunnel Web Server:
Release MVP version, and improve the fellings.

[EricJoy2048]

@ashulin @CalvinKirs How about storage Snapshot by JDBC?

[liugddx]

Support Flink 1.15.X #2927

[MonsterChenzhuo]

Run on k8s is anyone working on this, if not, I think I can make it happen

[cason0126]

Support Cache between source and sink
What are the requirements for this feature ？
up to now ， ST Connector is mostly BatchMode ,Is this Feature for in case Sink Speed Less Than Source？
if so， Control Source Speed can solve it with low memory use。
@EricJoy2048

[CalvinKirs]

Run on k8s is anyone working on this, if not, I think I can make it happen

Not yet, welcome you contribute it

[2000liux]

Is there a todolist item in Transform V2 Flink Translation? I can do something related。
@EricJoy2048

[jobmission]

@EricJoy2048 long time no updates?

[somnmos]

how about to support multi-source and complex query sql plugin in zeta engine?