Upgrade to Flink 1.15 & FLIP-27 Source APIs #37
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
I've checked out this PR and build it locally, together with a locally build Flink 1.15. I've tried some of the SQL recipes but I'm getting some weird results on timestamps.
CREATE TABLE orders (
bidtime TIMESTAMP(3),
price DOUBLE,
item STRING,
supplier STRING,
WATERMARK FOR bidtime AS bidtime - INTERVAL '5' SECONDS
) WITH (
'connector' = 'faker',
'fields.bidtime.expression' = '#{date.past ''30'',''SECONDS''}',
'fields.price.expression' = '#{Number.randomDouble ''2'',''1'',''150''}',
'fields.item.expression' = '#{Commerce.productName}',
'fields.supplier.expression' = '#{regexify ''(Alice|Bob|Carol|Alex|Joe|James|Jane|Jack)''}',
'rows-per-second' = '100'
);I'm getting the same result for every row:
One more question: since this version won't be compatible with Flink 1.14 and lower, will you release this with a major/minor version bump?
|
@MartijnVisser Thank you for testing.
|
|
@knaufk Yes, I was indeed testing with parallelism of 1. It also occurs when parallelism is set to 2 (and probably higher) |
|
@knaufk yes, I'm very sorry for this, but it seems a regression slipped in due to some aggressive caching. Going back to version 1.1.0 should fix this, and it's also fixed in 1.3.0, but that one hasn't been release yet (it's planned to be released in 7 days). My apologies for this! |
|
just for the history, the problem with the same data for every row was fixed at datafaker-net/datafaker#88 |
|
Thanks everyone. I've added a test to guard against this kind of regression in the future and downgraded to 1.1.0 for now. |
knaufk
force-pushed
the
flink115
branch
2 times, most recently
from
d5d760b to
9e37222
Compare
|
I'm having another look. I do find it weird that after pulling your latest commits, I'm now getting a Flink related error. That's probably for the Flink community to look into because I don't expect that they should appear when running a |
|
@MartijnVisser I suspect this somehow related to your local setup. Neither the CI nor me locally can reproduce this. |
|
@knaufk Yes, I needed to remove locally downloaded Maven artifacts |
MartijnVisser
approved these changes
Mar 25, 2022
|
@knaufk @MartijnVisser Datafaker 1.3.0 has been released, this fixes the issue related to the cached data generation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.