Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: datagen connector should not be used with ROW FORMAT JSON #3307

Closed
fuyufjh opened this issue Jun 17, 2022 · 4 comments
Closed

bug: datagen connector should not be used with ROW FORMAT JSON #3307

fuyufjh opened this issue Jun 17, 2022 · 4 comments
Assignees
Labels
help wanted Issues that need help from contributors no-issue-activity type/bug Something isn't working

Comments

@fuyufjh
Copy link
Member

fuyufjh commented Jun 17, 2022

Describe the bug

dev=> create source ss (id int) with ('connector' = 'datagen');
ERROR:  sql parser error: Expected ROW, found: ;

But this works:

dev=> create source ss (id int) with ('connector' = 'datagen') ROW FORMAT JSON;
CREATE_SOURCE

To Reproduce

See above.

Expected behavior

datagen connector should not be used with ROW FORMAT JSON

Additional context

@fuyufjh fuyufjh added type/bug Something isn't working help wanted Issues that need help from contributors labels Jun 17, 2022
@fuyufjh
Copy link
Member Author

fuyufjh commented Jun 17, 2022

@shanicky PTAL 🥰

@shanicky
Copy link
Contributor

The connector crate only generates raw bytes messages that need to be converted into chunks by the source parser, so this is a problem for this internal connector.
Let's figure it out.

cc @tabVersion

@tabVersion
Copy link
Contributor

as I suggested in the very beginning, datagen should be at source level rather than connector. Moving to source level can prevent CPU resource waste (serializing generated data to json then deserializing in connector source).

@shanicky
Copy link
Contributor

fixed in #7612

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Issues that need help from contributors no-issue-activity type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants