move S3Config into destination-s3; update dependencies accordingly #8562

edgao · 2021-12-06T22:57:00Z

What

It doesn't make much sense for destination-jdbc to hold S3Config, or for destination-s3 to depend on destination-jdbc. This is a prereq for #8550.

Also delete the unnecessary JDBC acceptance test - it doesn't even work (b/c destination-jdbc doesn't even declare a main class).

Also update the S3 docs to reflect a past spec.json update.

How

Replace references to S3Config with S3DestinationConfig, then update module dependencies. Most importantly, remove s3's dependency on jdbc.

Note that a few class members got moved around to accommodate this:

S3StreamCopier#DEFAULT_PART_SIZE now lives inside S3DestinationConfig; this was the only place it was referenced anyway.
S3StreamCopier's attemptS3WriteAndDelete and getAmazonS3 methods now live on S3Destination. This necessitated updates to the JDBC, Redshift, and Snowflake destinations, which now list destination-s3 as a dependency.

Also had a few formatting changes for some reason, which intellij insists on making. Intellij is set up according to this, so not quite sure what's happening.

🚨 User Impact 🚨

If anyone had code that depended on S3Config / S3StreamCopier (i.e. outside of this repo) then that will break. Otherwise, this should have no visible impact.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

CLAassistant · 2021-12-06T22:57:07Z

All committers have signed the CLA.

sherifnada

LGTM modulo unit tests

...ctors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/S3Destination.java

sherifnada · 2021-12-07T01:16:17Z

...ctors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/S3Destination.java

+    s3.deleteObject(s3Bucket, outputTableName);
+  }
+
+  public static AmazonS3 getAmazonS3(final S3DestinationConfig s3Config) {


would a unit test make sense here?

added a test, but it's pretty mediocre. AmazonS3 objects don't actually expose any methods to extract their configurations, so all the test accomplishes is to check that this method doesn't throw an exception, and that it returns something non-null :/

actually, I can just kill this method and use S3DestinationConfig#getS3Client instead

edgao · 2021-12-07T17:25:43Z

hm. I'm pretty sure DefaultAirbyteSourceTest and DefaultAirbyteDestinationTest shouldn't be breaking due to these changes; is there a way to rerun just the one specific check?

edgao · 2021-12-07T18:08:23Z

hmmm. between the docs and (I think?) the spec, is spec.json canonical or are we building towards what docs.airbyte describes? Specifically, the docs say that part size is a top-level option for destination-s3, but the spec is putting it inside the lower-level avro/csv/jsonl format objects, with parquet not exposing it at all

(the partSize field on S3DestinationConfig is actually redundant with S3FormatConfig#getPartSize, but not sure which one would be better to remove)

sherifnada

LGTM

...s/destination-s3/src/test/java/io/airbyte/integrations/destination/s3/S3DestinationTest.java

sherifnada · 2021-12-07T22:03:57Z

@edgao is that option relevant for parquet? I'm not seeing a default value being set in the code. @tuliren do you have any context on why page size is not relevant for parquet s3?

tuliren · 2021-12-07T22:47:27Z

@tuliren do you have any context on why page size is not relevant for parquet s3?

Page size is relevant for Parquet. Part size is not.

Here is how we construct the S3 Parquet writer, it requires the page size for compression:

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/parquet/S3ParquetWriter.java#L63

Here is how we construct the routine S3 writer (for JSONL and CSV), it requires the part size for uploading:

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/util/S3StreamTransferManagerHelper.java#L22

tuliren · 2021-12-07T22:53:14Z

hmmm. between the docs and (I think?) the spec, is spec.json canonical or are we building towards what docs.airbyte describes? Specifically, the docs say that part size is a top-level option for destination-s3, but the spec is putting it inside the lower-level avro/csv/jsonl format objects, with parquet not exposing it at all

(the partSize field on S3DestinationConfig is actually redundant with S3FormatConfig#getPartSize, but not sure which one would be better to remove)

@edgao, the doc is outdated. The part size is only needed for Avro, CSV, and JSONL. It is not required for Parquet, at least not for now, because the Parquet writer is constructed differently and does not require this parameter.

So the current spec.json is correct that the part_size_md only exists in the low level format object.

If I remember correctly, when the S3 destination was first rolled out, it only supported CSV. That's why the part size was at the top level initially. But when we added Parquet as the second format, we realized that part size should not be a top level parameter, and moved it inside format. However, the doc did not get updated accordingly.

airbyte-integrations/connectors/destination-jdbc/build.gradle

...integration/java/io/airbyte/integrations/destination/jdbc/JdbcDestinationAcceptanceTest.java

edgao

sounds good, will update the docs to match the spec.

airbyte-integrations/connectors/destination-jdbc/build.gradle

...integration/java/io/airbyte/integrations/destination/jdbc/JdbcDestinationAcceptanceTest.java

edgao · 2021-12-08T21:12:56Z

latest commits just do some final code cleanup (replacing calls to S3StreamCopier.attemptWriteAndDelete with the S3Destination method) and updating the S3 docs to match the spec. Doc update isn't super detailed, it just removes the part_size param and links to spec.json instead of referencing a nonexistent paragraph.

Do I need to run through all the "Airbyter" stuff in the PR template? I.e. /test connector=connectors/<name>, release to dockerhub, etc? Or can I just merge this PR, and the docs update will propagate automatically?

edgao · 2021-12-10T01:09:10Z

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1561440688

edgao · 2021-12-10T21:07:30Z

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1565195945
✅ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1565195945
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 468    287    39%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1160    472    59%

edgao · 2021-12-10T21:07:35Z

/test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1565196085
✅ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1565196085
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 468    287    39%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1160    472    59%

edgao · 2021-12-10T21:07:40Z

/test connector=destination-gcs

🕑 destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1565196940
✅ destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1565196940
No Python unittests run

…8562)

* upgrade gradle * upgrade to Java 17 (and fix a few of the node versioning misses) * oops * try to run a different format version * fix spotless by upgrading / reformatting some files * fix ci settings * upgrade mockito to avoid other errors * undo bad format * fix "incorrect" sql comments * fmt * add debug flag * remove * bump * bump jooq to a version that has a java 17 dist * fix * remove logs * oops * revert jooq upgrade * fix * set up java for connector test * fix yaml * generate std source tests * fail zombie job attempts and add failure reason (#8709) * fail zombie job attempts and add failure reason * remove failure reason * bump gcp dependencies to pick up grpc update (#8713) * Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (#8714) Co-authored-by: jrhizor <[email protected]> * Change CDK "Caching" header to "nested streams & caching" * Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524) Signed-off-by: Sergey Chvalyuk <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> * move S3Config into destination-s3; update dependencies accordingly (#8562) Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: jrhizor <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Iryna Grankova <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Edward Gao <[email protected]>

version bumps for #8562

VitaliiMaltsev · 2021-12-14T18:47:15Z

airbyte-integrations/connectors/destination-snowflake/Dockerfile

@@ -11,6 +11,9 @@ WORKDIR /airbyte

 ENV APPLICATION destination-snowflake

+# Needed for JDK17 (in turn, needed on M1 macs) - see https://github.com/snowflakedb/snowflake-jdbc/issues/589#issuecomment-983944767
+ENV DESTINATION_SNOWFLAKE_OPTS "--add-opens java.base/java.nio=ALL-UNNAMED"


@edgao looks like issue described here with jdk17 still persists in master branch

yeah, I'm not sure what happened - snowflake was definitely passing for me locally, but after the java 17 upgrade it's failing again. Haven't been able to find a fix, either.

@edgao @VitaliiMaltsev Were you guys able to find the fix?

https://github.com/airbytehq/airbyte/pull/8781/files#diff-0fd1ff4a11af6edb89860081578c13598aa6cdb289c3e6ceae2a9d44eb708b80R45-R46 was the fix

(also https://github.com/airbytehq/airbyte/pull/8854/files#diff-a3dd6be9c3819c1fcfd593b0f11c29663912e686e0dd74cae14610ac9c282480R50, but technically we only needed to one or the other)

@edgao is there any side effect using JVM command jvmArgs = ["--add-opens=java.base/java.nio=ALL-UNNAMED"] ?
Any potential security issues?

my understanding is that this isn't a security check, it's more so that the Java devs can modify some Java internal implementation in the future. I.e. they want to discourage reliance on java's internal APIs, so access is now disabled by default.

@edgao thanks for the response. However, based on the JEPS-261:

The --add-exports and --add-opens options must be used with great care. You can use them to gain access to an internal API of a library module, or even of the JDK itself, but you do so at your own risk: If that internal API is changed or removed then your library or application will fail.

I'm not that familiar with how gaining more internal access to java.nio could be concerning. Therefore, asking if you have insight into it.

The risk occurs when upgrading Java versions - there's no guarantee that the underlying implementation will remain the same. So in theory, the Java devs could release a new version of Java, which modifies those internal APIs, and therefore break the snowflake JDBC driver. For Airbyte, this risk is mitigated by the source/destination acceptance tests, which actually build the Docker image and check that it can interact with a real Snowflake instance, i.e. this would be caught before release. But we would be unable to upgrade to that new Java version until an updated snowflake-jdbc is released.

I'm not actually sure which issue is being used to track progress on removing snowflake-jdbc's dependency on those internal APIs, but would recommend asking in https://github.com/snowflakedb/snowflake-jdbc if you're interested in the details / how they're thinking about the problem. Some potentially related issues: snowflakedb/snowflake-jdbc#484, snowflakedb/snowflake-jdbc#589, snowflakedb/snowflake-jdbc#533

…irbytehq#8562)

* upgrade gradle * upgrade to Java 17 (and fix a few of the node versioning misses) * oops * try to run a different format version * fix spotless by upgrading / reformatting some files * fix ci settings * upgrade mockito to avoid other errors * undo bad format * fix "incorrect" sql comments * fmt * add debug flag * remove * bump * bump jooq to a version that has a java 17 dist * fix * remove logs * oops * revert jooq upgrade * fix * set up java for connector test * fix yaml * generate std source tests * fail zombie job attempts and add failure reason (airbytehq#8709) * fail zombie job attempts and add failure reason * remove failure reason * bump gcp dependencies to pick up grpc update (airbytehq#8713) * Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (airbytehq#8714) Co-authored-by: jrhizor <[email protected]> * Change CDK "Caching" header to "nested streams & caching" * Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (airbytehq#8524) Signed-off-by: Sergey Chvalyuk <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> * move S3Config into destination-s3; update dependencies accordingly (airbytehq#8562) Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: jrhizor <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Iryna Grankova <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]> Co-authored-by: Edward Gao <[email protected]>

version bumps for airbytehq#8562

move S3Config into destination-s3; update deps

9ab7fa7

github-actions bot added the area/connectors Connector related issues label Dec 6, 2021

edgao temporarily deployed to more-secrets December 6, 2021 23:25 Inactive

combine S3Config and S3DestinationConfig

34dd5f0

edgao force-pushed the jdbc_s3_dependency_reverse branch from 2e4e078 to 34dd5f0 Compare December 6, 2021 23:27

edgao temporarily deployed to more-secrets December 6, 2021 23:29 Inactive

edgao marked this pull request as ready for review December 7, 2021 00:05

edgao requested a review from sherifnada December 7, 2021 00:05

sherifnada approved these changes Dec 7, 2021

View reviewed changes

add unit tests

b612783

edgao temporarily deployed to more-secrets December 7, 2021 16:15 Inactive

delete unnecessary integration test

7813798

edgao temporarily deployed to more-secrets December 7, 2021 17:37 Inactive

use already-existing method

44d1d0c

edgao temporarily deployed to more-secrets December 7, 2021 18:01 Inactive

sherifnada approved these changes Dec 7, 2021

View reviewed changes

...s/destination-s3/src/test/java/io/airbyte/integrations/destination/s3/S3DestinationTest.java Show resolved Hide resolved

tuliren reviewed Dec 7, 2021

View reviewed changes

airbyte-integrations/connectors/destination-jdbc/build.gradle Show resolved Hide resolved

...integration/java/io/airbyte/integrations/destination/jdbc/JdbcDestinationAcceptanceTest.java Show resolved Hide resolved

edgao commented Dec 8, 2021

View reviewed changes

airbyte-integrations/connectors/destination-jdbc/build.gradle Show resolved Hide resolved

...integration/java/io/airbyte/integrations/destination/jdbc/JdbcDestinationAcceptanceTest.java Show resolved Hide resolved

edgao mentioned this pull request Dec 8, 2021

🎉 Destination Redshift (copy): accept bucket path for staging data #8607

Merged

16 tasks

edgao added 2 commits December 8, 2021 11:42

delete old method

7b4bc5c

update doc

2f08ad1

github-actions bot added the area/documentation Improvements or additions to documentation label Dec 8, 2021

edgao temporarily deployed to more-secrets December 8, 2021 21:08 Inactive

jrhizor temporarily deployed to more-secrets December 10, 2021 01:11 Inactive

fix for jdk17

df68bbe

edgao temporarily deployed to more-secrets December 10, 2021 18:54 Inactive

correct snowflake test instructions

8b48533

edgao temporarily deployed to more-secrets December 10, 2021 21:08 Inactive

jrhizor temporarily deployed to more-secrets December 10, 2021 21:09 Inactive

edgao temporarily deployed to more-secrets December 10, 2021 21:56 Inactive

sherifnada approved these changes Dec 10, 2021

View reviewed changes

edgao merged commit fc91f67 into master Dec 10, 2021

edgao deleted the jdbc_s3_dependency_reverse branch December 10, 2021 23:51

jrhizor pushed a commit that referenced this pull request Dec 10, 2021

move S3Config into destination-s3; update dependencies accordingly (#…

b715f6a

…8562)

edgao mentioned this pull request Dec 11, 2021

Bump connector versions for publishing #8717

Merged

16 tasks

edgao added a commit that referenced this pull request Dec 11, 2021

Bump connector versions for publishing (#8717)

0b8c0f5

version bumps for #8562

jrhizor mentioned this pull request Dec 12, 2021

Bump Airbyte version from 0.33.10-alpha to 0.33.11-alpha #8728

Merged

VitaliiMaltsev reviewed Dec 14, 2021

View reviewed changes

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

move S3Config into destination-s3; update dependencies accordingly (a…

1e282cb

…irbytehq#8562)

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

Bump connector versions for publishing (airbytehq#8717)

b201955

version bumps for airbytehq#8562

karinakuz added connectors/destinations-warehouse connectors/destination/snowflake labels Jan 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

move S3Config into destination-s3; update dependencies accordingly #8562

move S3Config into destination-s3; update dependencies accordingly #8562

edgao commented Dec 6, 2021 •

edited

Loading

CLAassistant commented Dec 6, 2021 •

edited

Loading

sherifnada left a comment

sherifnada Dec 7, 2021

edgao Dec 7, 2021

edgao Dec 7, 2021

edgao commented Dec 7, 2021

edgao commented Dec 7, 2021

sherifnada left a comment

sherifnada commented Dec 7, 2021

tuliren commented Dec 7, 2021 •

edited

Loading

tuliren commented Dec 7, 2021 •

edited

Loading

edgao left a comment

edgao commented Dec 8, 2021

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev Dec 14, 2021 •

edited

Loading

edgao Dec 15, 2021

iamzafar Jan 13, 2022

edgao Jan 14, 2022

iamzafar Jan 15, 2022

edgao Jan 16, 2022

iamzafar Jan 16, 2022

edgao Jan 16, 2022

move S3Config into destination-s3; update dependencies accordingly #8562

move S3Config into destination-s3; update dependencies accordingly #8562

Conversation

edgao commented Dec 6, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

CLAassistant commented Dec 6, 2021 • edited Loading

sherifnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edgao commented Dec 7, 2021

edgao commented Dec 7, 2021

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada commented Dec 7, 2021

tuliren commented Dec 7, 2021 • edited Loading

tuliren commented Dec 7, 2021 • edited Loading

edgao left a comment

Choose a reason for hiding this comment

edgao commented Dec 8, 2021

edgao commented Dec 10, 2021 • edited by github-actions bot Loading

edgao commented Dec 10, 2021 • edited by github-actions bot Loading

edgao commented Dec 10, 2021 • edited by github-actions bot Loading

edgao commented Dec 10, 2021 • edited by github-actions bot Loading

VitaliiMaltsev Dec 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edgao commented Dec 6, 2021 •

edited

Loading

CLAassistant commented Dec 6, 2021 •

edited

Loading

tuliren commented Dec 7, 2021 •

edited

Loading

tuliren commented Dec 7, 2021 •

edited

Loading

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 10, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev Dec 14, 2021 •

edited

Loading