Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an end-to-end Kafka source&sink streaming conformance integration test #1307

Closed
yruslan opened this issue Apr 27, 2020 · 2 comments · Fixed by #1366
Closed

Add an end-to-end Kafka source&sink streaming conformance integration test #1307

yruslan opened this issue Apr 27, 2020 · 2 comments · Fixed by #1366
Assignees
Labels
Conformance Conformance Job affected feature New feature priority: medium Important but not urgent

Comments

@yruslan
Copy link
Contributor

yruslan commented Apr 27, 2020

Background

An end-to-end streaming conformance test will help to catch streaming-only issues.
Such as the Catalyst optimizer issue #1306

Feature

Add an end-to-end Kafka source&sink streaming conformance integration test.

  • Data for the test can be usual JSONs. We can even reuse the existing end-to-end conformance test data for that.
  • An instance of Kafka should be brought up.
  • Data from the examples should be sent to raw Kafka topic converting the input data to Avro.
  • A streaming conformance test should be run as Kafka -> Parquet and Kadka -> Kafka

Proposed Solution [Optional]

Use https://www.testcontainers.org/modules/kafka/
It is used by Hyperdrive as well. Thanks Felipe for the suggestion.

@yruslan
Copy link
Contributor Author

yruslan commented Jun 3, 2020

There is one additional thing unit/integration tests of streaming conformance should check. It is the way enceladus_info_date and enceladus_info_date_string are created. It should support a constant value, processing time, and event time column. The strategy for generating these columns are determined by command line parameters.

@AdrianOlosutean
Copy link
Contributor

Release notes
Created Integration test for HyperConformance which checks enceladus_info_date for constant and event_time

AdrianOlosutean added a commit that referenced this issue Jun 17, 2020
…ink-streaming-conformance-integration-test

#1307 Integration test for Streaming Conformance
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Conformance Conformance Job affected feature New feature priority: medium Important but not urgent
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants