[Feature][transform] transform support explode #7928

CosmosNi · 2024-10-28T04:52:16Z

#7926
support explode transform function

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

Hisoka-X · 2024-10-28T06:18:22Z

cc @corgy-w

Hisoka-X · 2024-10-28T06:19:48Z

cc @YuriyGavrilov as well.

seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/SeaTunnelMultiRowTransform.java

docs/en/transform-v2/explode.md

Hisoka-X · 2024-10-30T02:16:50Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/SeaTunnelMultiRowTransform.java

+     * @param row the data need be transformed.
+     * @return transformed data.
+     */
+    List<T> flatMap(T row);


cc @hailin0

...erver/src/main/java/org/apache/seatunnel/engine/server/task/flow/TransformFlowLifeCycle.java

CosmosNi · 2024-11-01T03:59:51Z

@Hisoka-X @liunaijie @corgy-w please cc

SELECT * FROM fake
LATERAL VIEW EXPLODE (SPILT ( NAME, ',' )) AS NAME
LATERAL VIEW EXPLODE (SPILT ( pk_id, ';' )) AS pk_id
LATERAL VIEW OUTER EXPLODE ( age ) AS age

YuriyGavrilov · 2024-11-07T09:51:32Z

Hi Guys, I can't cover all review, just to inform when we rotate data on 90 degrees we should have possibilities to limit row count due to it can course to problem on next step. For my case we will take 10 or 20 rows per each table and will send this data to llm. Output will be fixed or knowable length size. Columns as rows with description as option (maybe) and llm output (as json perhaps)

Many thanks 🙏 for making all of this happening. It is amazing job.

corgy-w · 2024-11-07T14:42:14Z

Hi Guys, I can't cover all review, just to inform when we rotate data on 90 degrees we should have possibilities to limit row count due to it can course to problem on next step. For my case we will take 10 or 20 rows per each table and will send this data to llm. Output will be fixed or knowable length size. Columns as rows with description as option (maybe) and llm output (as json perhaps)

Many thanks 🙏 for making all of this happening. It is amazing job.

@YuriyGavrilov This pr does not include what you mentioned, your needs are still in progress

Hisoka-X · 2024-11-08T08:32:24Z

docs/en/transform-v2/sql-functions.md

+
+select ARRAY('test1','test2','test3') as arrays
+
+### SPLIT


The split method should be string function. Please move it to https://github.com/apache/seatunnel/blob/dev/docs/en/transform-v2/sql-functions.md#string-functions

Hisoka-X · 2024-11-08T09:38:04Z

...nel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLFunction.java

+                return;
+            }
+            for (Object fieldValue : (Object[]) splitFieldValue) {
+                Object value = keepValueType ? fieldValue : String.valueOf(fieldValue);


if the fieldValue is null, it will be convert to "null" by String.valueOf(fieldValue). The value will be wrong.

Hisoka-X · 2024-11-08T09:38:40Z

...nel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLFunction.java

+        }
+    }
+
+    private SeaTunnelRow copySeaTunnelRow(


Suggested change

private SeaTunnelRow copySeaTunnelRow(

private SeaTunnelRow copySeaTunnelRowWithNewValue(

...nel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLFunction.java

…at config

Hisoka-X

LGTM. Thanks @CosmosNi and @corgy-w @liunaijie @YuriyGavrilov for review!

Hisoka-X · 2024-11-08T12:16:15Z

Hi Guys, I can't cover all review, just to inform when we rotate data on 90 degrees we should have possibilities to limit row count due to it can course to problem on next step. For my case we will take 10 or 20 rows per each table and will send this data to llm. Output will be fixed or knowable length size. Columns as rows with description as option (maybe) and llm output (as json perhaps)

Many thanks 🙏 for making all of this happening. It is amazing job.

We need a sql function named slice to slice array in another PR. cc @corgy-w @CosmosNi

CosmosNi · 2024-11-08T12:42:38Z

like slice(field,start,end)?

Hisoka-X · 2024-11-08T12:46:19Z

like slice(field,start,end)?

yes.

CosmosNi · 2024-11-08T14:50:31Z

...e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/sql_transform/explode_transform.conf

+        row_rules = [
+          {
+            rule_type = MAX_ROW
+            rule_value = 24


name×pk_id×age×array 2×2×2×2+2×1×1×2+1×2×1×2 = 24

[Feature][transform] transform support explode

9b09ad8

github-actions bot added core SeaTunnel core module Zeta transform-v2 e2e api labels Oct 28, 2024

This was linked to issues Oct 28, 2024

[Feature][transform] transform support explode #7926

Closed

[Feature][Transforms-V2] Simple transpose options for LLMs to rotate data for prompts #7848

Closed

[Feature][transform] transform support explode

d83c2aa

liunaijie reviewed Oct 29, 2024

View reviewed changes

seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/SeaTunnelMultiRowTransform.java Outdated Show resolved Hide resolved

[Feature][transform] transform support explode

6f7f52d

github-actions bot added the document label Oct 29, 2024

[Feature][transform] transform support explode

6b80660

CosmosNi requested review from liunaijie and Hisoka-X October 30, 2024 01:51

Hisoka-X reviewed Oct 30, 2024

View reviewed changes

docs/en/transform-v2/explode.md Outdated Show resolved Hide resolved

liunaijie reviewed Oct 30, 2024

View reviewed changes

docs/en/transform-v2/explode.md Outdated Show resolved Hide resolved

Hisoka-X reviewed Oct 30, 2024

View reviewed changes

corgy-w reviewed Oct 30, 2024

View reviewed changes

...erver/src/main/java/org/apache/seatunnel/engine/server/task/flow/TransformFlowLifeCycle.java Show resolved Hide resolved

CosmosNi added 4 commits October 30, 2024 17:42

[Feature][transform] explode transform support spilt list

d30be82

[Feature][transform] explode transform support spilt list

48adff1

[Feature][transform] explode transform support spilt list

3a55991

[Feature][transform] sql transform support lateral view explode

87f3bef

github-actions bot added connectors-v2 paimon labels Nov 1, 2024

CosmosNi requested review from liunaijie and Hisoka-X November 1, 2024 05:34

[Feature][transform] fix error word

fd715b9

github-actions bot removed the paimon label Nov 8, 2024

Hisoka-X reviewed Nov 8, 2024

View reviewed changes

CosmosNi added 3 commits November 8, 2024 17:47

[Feature][transform] transform support explode

013c284

[Feature][RestAPI] Support submit job with seatunnel style hocon form…

619cd2f

…at config

[Feature][transform] transform support explode

46f0857

Hisoka-X previously approved these changes Nov 8, 2024

View reviewed changes

github-actions bot added approved reviewed labels Nov 8, 2024

Hisoka-X added the don't merge There needs to be a specific reason in the PR, and it cannot be merged for the time being. label Nov 8, 2024

[Feature][transform] transform support explode

b337b50

CosmosNi dismissed Hisoka-X’s stale review via b337b50 November 8, 2024 14:39

github-actions bot removed approved reviewed labels Nov 8, 2024

CosmosNi commented Nov 8, 2024

View reviewed changes

Hisoka-X approved these changes Nov 8, 2024

View reviewed changes

Hisoka-X removed the don't merge There needs to be a specific reason in the PR, and it cannot be merged for the time being. label Nov 8, 2024

github-actions bot added approved reviewed labels Nov 8, 2024

hailin0 approved these changes Nov 9, 2024

View reviewed changes

hailin0 merged commit 132278c into apache:dev Nov 9, 2024
8 checks passed

hailin0 mentioned this pull request Nov 24, 2024

SeaTunnel community meeting Topic collect #8133

Closed

3 tasks

hailin0 mentioned this pull request Jan 14, 2025

SeaTunnel community meeting Topic collect #8517

Closed

3 tasks

tzk007 mentioned this pull request Feb 12, 2025

[Bug] [seatunnel-web] run failed , because of jsqlparser.version change 4.5 to 4.9 #8650

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][transform] transform support explode #7928

[Feature][transform] transform support explode #7928

CosmosNi commented Oct 28, 2024

Hisoka-X commented Oct 28, 2024

Hisoka-X commented Oct 28, 2024

Hisoka-X Oct 30, 2024

CosmosNi commented Nov 1, 2024 •

edited

Loading

YuriyGavrilov commented Nov 7, 2024

corgy-w commented Nov 7, 2024

Hisoka-X Nov 8, 2024

Hisoka-X Nov 8, 2024

Hisoka-X Nov 8, 2024

Hisoka-X left a comment

Hisoka-X commented Nov 8, 2024

CosmosNi commented Nov 8, 2024

Hisoka-X commented Nov 8, 2024

CosmosNi Nov 8, 2024 •

edited

Loading

	private SeaTunnelRow copySeaTunnelRow(
	private SeaTunnelRow copySeaTunnelRowWithNewValue(

[Feature][transform] transform support explode #7928

[Feature][transform] transform support explode #7928

Conversation

CosmosNi commented Oct 28, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X commented Oct 28, 2024

Hisoka-X commented Oct 28, 2024

Hisoka-X Oct 30, 2024

Choose a reason for hiding this comment

CosmosNi commented Nov 1, 2024 • edited Loading

YuriyGavrilov commented Nov 7, 2024

corgy-w commented Nov 7, 2024

Hisoka-X Nov 8, 2024

Choose a reason for hiding this comment

Hisoka-X Nov 8, 2024

Choose a reason for hiding this comment

Hisoka-X Nov 8, 2024

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

Hisoka-X commented Nov 8, 2024

CosmosNi commented Nov 8, 2024

Hisoka-X commented Nov 8, 2024

CosmosNi Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

CosmosNi commented Nov 1, 2024 •

edited

Loading

CosmosNi Nov 8, 2024 •

edited

Loading