[Feature][st-engine] Support for transform-v2 API #3145
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hailin0
force-pushed
the
st-engine-support-new-transform-api
branch
4 times, most recently
from
9d5254c to
7ad1b54
Compare
hailin0
changed the title
|
|
hailin0
force-pushed
the
st-engine-support-new-transform-api
branch
6 times, most recently
from
0196c32 to
e68e143
Compare
|
@EricJoy2048 @ashulin @Hisoka-X @ic4y @CalvinKirs tx. PTAL |
|
@531651225 @TaoZex @TyrantLucifer @liugddx @laglangyue th. PTAL |
hailin0
commented
Oct 24, 2022
Comment on lines
+55
to
+64
| public static final String PROJECT_ROOT_PATH = getProjectRootPath(); | ||
|
|
||
| private static String getProjectRootPath() { | ||
| String e2eRootModuleDir = "seatunnel-e2e"; | ||
| Path path = Paths.get(System.getProperty("user.dir")); | ||
| while (!path.endsWith(Paths.get(e2eRootModuleDir))) { | ||
| path = path.getParent(); | ||
| } | ||
| return path.getParent().toString(); | ||
| } |
There was a problem hiding this comment.
Update get root path in e2e module
| TestCaseInvocationContextProvider.class | ||
| }) | ||
| @TestInstance(TestInstance.Lifecycle.PER_CLASS) | ||
| @DisabledOnContainer(value = {}, type = {EngineType.FLINK, EngineType.SPARK}, disabledReason = "TODO: Transform v2 translation to spark/flink isn't completed") |
|
@EricJoy2048 Are we should support repartition type transform in this version? |
hailin0
force-pushed
the
st-engine-support-new-transform-api
branch
3 times, most recently
from
d28a173 to
9661564
Compare
TaoZex
reviewed
Oct 24, 2022
TaoZex
reviewed
Oct 24, 2022
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/CopyFieldTransform.java
Outdated
Show resolved
Hide resolved
hailin0
force-pushed
the
st-engine-support-new-transform-api
branch
from
9661564 to
710555d
Compare
8 tasks
hailin0
force-pushed
the
st-engine-support-new-transform-api
branch
5 times, most recently
from
52b3b5b to
cdbf254
Compare
Hisoka-X
reviewed
Oct 31, 2022
seatunnel-common/src/main/java/org/apache/seatunnel/common/config/Common.java
Outdated
Show resolved
Hide resolved
...-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/FlinkExecution.java
Outdated
Show resolved
Hide resolved
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/CopyFieldTransform.java
Outdated
Show resolved
Hide resolved
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/ReplaceTransform.java
Outdated
Show resolved
Hide resolved
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/SplitTransform.java
Outdated
Show resolved
Hide resolved
I suggest to add the support of repartition transform in the subsequent plans. |
hailin0
force-pushed
the
st-engine-support-new-transform-api
branch
3 times, most recently
from
c25cb06 to
f7acd79
Compare
ashulin
previously approved these changes
Nov 1, 2022
| <artifactId>seatunnel-transforms-v2</artifactId> | ||
| <version>${project.version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> |
There was a problem hiding this comment.
It should also need to be added to the release profile.
@CalvinKirs CC
There was a problem hiding this comment.
fixed. added to release profile
@ashulin
Release binary package: - Move engine starter jar from `lib/` to `starter/` dir - Outpout `seatunnel-transforms-v2.jar` to `lib/` dir Maven modules: - Add seatunnel-transforms-v2 module - Add seatunnel-transforms-v2-e2e module Engine Starter: - Add `lib/` to task execute classpath ST-Engine: - Support for Split、Replace、Filter transform E2E: - Copy `seatunnel-transforms-v2.jar` into container `lib/` dir - Improve get project root path for e2e test module
hailin0
force-pushed
the
st-engine-support-new-transform-api
branch
from
f7acd79 to
31deed0
Compare
Hisoka-X
approved these changes
Nov 1, 2022
This was referenced Nov 2, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of this pull request
Reference #2678
Release binary package:
- Move engine starter jar from
lib/tostarter/dir- Outpout
seatunnel-transforms-v2.jartolib/dirMaven modules:
- Add seatunnel-transforms-v2 module
- Add seatunnel-transforms-v2-e2e module
Engine Starter:
- Add
lib/to task execute classpathST-Engine:
- Support for Split、Replace、Filter transform
E2E:
- Copy
seatunnel-transforms-v2.jarinto containerlib/dir- Improve get project root path for e2e test module
Check list
New License Guide