perf(source): intro native row format

[waruto210]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

• intorcude new NATIVE ROW FORMAT, which is the default row format of nexmark and datagen connecotr, and it is invisible to user.
• nexmark only support NATIVE ROW FORMAT, but datagen can support multiple formats(only native and json now)

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
  - [ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features).
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

• Connector (sources & sinks)

Release note

nexmark: user should not specify row format

create table T (
    schema def...
) with (
    connecotr = 'nexmark'
    ...
);

datagen: If user do not specify a row format, datagen would use NATIVE.

create table T (
    schema def...
) with (
    connecotr = 'datagen'
    ...
) [row foramt json];

Refer to a related PR or issue link (optional)

#6969 ,#4555

[waruto210]

@fuyufjh @lmatz Should we introduce a new native ROW FORMAT(a breaking change now?) or just ignore the ROW FORMAT?

According to issue #6970, we need to introduce a new native ROW FORMAT and need to rewrite the code for this pr.

[tabVersion]

@fuyufjh @lmatz Should we introduce a new native ROW FORMAT(a breaking change now?) or just ignore the ROW FORMAT?

According to issue #6970, we need to introduce a new native ROW FORMAT and need to rewrite the code for this pr.

since datagen connector and nexmark connector both produce stream chunks directly, I prefer rejecting row format in this case. It is hard to tell users what is native row format.

[waruto210]

@fuyufjh @lmatz Should we introduce a new native ROW FORMAT(a breaking change now?) or just ignore the ROW FORMAT?
According to issue #6970, we need to introduce a new native ROW FORMAT and need to rewrite the code for this pr.

since datagen connector and nexmark connector both produce stream chunks directly, I prefer rejecting row format in this case. It is hard to tell users what is native row format.

PTAL #6970, it requires datagen to be able to generate multiple row format.

[codecov]

Codecov Report

Merging #7612 (3a45614) into main (45b5e6b) will decrease coverage by 0.10%.
The diff coverage is 46.96%.

@@            Coverage Diff             @@
##             main    #7612      +/-   ##
==========================================
- Coverage   71.67%   71.58%   -0.10%     
==========================================
  Files        1111     1111              
  Lines      176936   177229     +293     
==========================================
+ Hits       126812   126862      +50     
- Misses      50124    50367     +243     

Flag	Coverage Δ
rust	71.58% <46.96%> (-0.10%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/batch/src/executor/source.rs	0.00% <0.00%> (ø)
src/connector/src/source/nexmark/source/message.rs	0.00% <ø> (-60.98%)	⬇️
src/sqlparser/src/keywords.rs	100.00% <ø> (ø)
src/sqlparser/src/parser.rs	91.57% <29.41%> (-0.37%)	⬇️
...nector/src/source/nexmark/source/combined_event.rs	27.11% <30.04%> (+27.11%)	⬆️
src/connector/src/parser/mod.rs	55.82% <38.46%> (-7.91%)	⬇️
src/sqlparser/src/ast/statement.rs	72.31% <50.00%> (-0.74%)	⬇️
...c/connector/src/source/datagen/source/generator.rs	89.28% <67.74%> (-4.84%)	⬇️
src/common/src/field_generator/mod.rs	75.00% <75.00%> (+0.50%)	⬆️
src/connector/src/source/datagen/source/reader.rs	86.10% <94.73%> (-1.36%)	⬇️
... and 22 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[fuyufjh]

@fuyufjh @lmatz Should we introduce a new native ROW FORMAT(a breaking change now?) or just ignore the ROW FORMAT?
According to issue #6970, we need to introduce a new native ROW FORMAT and need to rewrite the code for this pr.

since datagen connector and nexmark connector both produce stream chunks directly, I prefer rejecting row format in this case. It is hard to tell users what is native row format.

PTAL #6970, it requires datagen to be able to generate multiple row format.

Hmmmm. I think #6970 is meant to generate data with a nested column i.e. STRUCT, ARRAY, JSON. While the ROW FORMAT here is to define the serialization inside Datagen. They are not related. cc. @lmatz @kwannoel

[waruto210]

@fuyufjh @lmatz Should we introduce a new native ROW FORMAT(a breaking change now?) or just ignore the ROW FORMAT?
According to issue #6970, we need to introduce a new native ROW FORMAT and need to rewrite the code for this pr.

since datagen connector and nexmark connector both produce stream chunks directly, I prefer rejecting row format in this case. It is hard to tell users what is native row format.

PTAL #6970, it requires datagen to be able to generate multiple row format.

Hmmmm. I think #6970 is meant to generate data with a nested column i.e. STRUCT, ARRAY, JSON. While the ROW FORMAT here is to define the serialization inside Datagen. They are not related. cc. @lmatz @kwannoel

It seems the main purpose of #6970 is to test source parsing, both nested column and more ROW FORMAT are needed.

For performance, Datagen needs to be able to output native chunk directly, without going through parser; for testing against parsers, Datagen needs to genarate data with more complex columns and serializing data into more formats (e.g. avro, protobuf), then parse the serialized data into chunk.

So I think we need to introduce a native row format to control the Datagen.

[kwannoel]

@fuyufjh @lmatz Should we introduce a new native ROW FORMAT(a breaking change now?) or just ignore the ROW FORMAT?
According to issue #6970, we need to introduce a new native ROW FORMAT and need to rewrite the code for this pr.

since datagen connector and nexmark connector both produce stream chunks directly, I prefer rejecting row format in this case. It is hard to tell users what is native row format.

PTAL #6970, it requires datagen to be able to generate multiple row format.

Hmmmm. I think #6970 is meant to generate data with a nested column i.e. STRUCT, ARRAY, JSON. While the ROW FORMAT here is to define the serialization inside Datagen. They are not related. cc. @lmatz @kwannoel

#6970 is meant to test source parsing of rows, I think @waruto210 has the right idea.
I'm thinking of something like this:

CREATE MATERIALIZED SOURCE IF NOT EXISTS source_abc 
WITH (
   connector='datagen',
   ...
)
ROW FORMAT (AVRO | JSON | PROTOBUF)
...;

Instead of (this is done separately, see #7132):

CREATE TABLE t (v1 struct<...>, v2 int[]);

[tabVersion]

basically LGTM