🎉 Source S3: support of Parquet format #5305

antixar · 2021-08-10T15:14:31Z

How

Using same lib 'pyarrow' as for csv parsing

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

antixar · 2021-08-12T22:24:27Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1125652227
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1125652227

antixar · 2021-08-13T07:52:27Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1126887837
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1126887837

…-s3-support-parquet

antixar · 2021-08-13T09:02:47Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1127075915
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1127075915

antixar · 2021-08-13T09:18:50Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1127128527
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1127128527

bazarnov

Please fix branch conflicts + fix the airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py file conflict on this branch.

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py

…port-parquet

antixar · 2021-08-23T10:39:29Z

/test connector=connectors/source-s3

antixar · 2021-08-23T11:09:24Z

/test connector=connectors/source-s3

…port-parquet

antixar · 2021-08-23T12:45:47Z

/test connector=connectors/source-s3

…es_abstract/formats/parquet_spec.py Co-authored-by: George Claireaux <[email protected]>

…port-parquet

Phlair

Great, lgtm! One small note on buffer_size to make that more clear.

Due to the way we're iterating through individual files at the abstract-level, I anticipate issues with partitioned parquet datasets. I think we should make clear in the documentation that partitioned parquet datasets are unsupported for now.
For more context, it should work however the performance could be very bad + the columns used for partition would be missing from output (I think).

...te-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/parquet_spec.py

antixar · 2021-08-31T13:52:15Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1186397350
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1186397350

antixar · 2021-08-31T14:15:07Z

/publish connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1186480409
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1186480409

…port-parquet

antixar · 2021-09-03T11:10:13Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1197766878
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1197766878

…port-parquet

antixar · 2021-09-04T22:57:30Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1201855623
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1201855623

antixar · 2021-09-04T23:16:15Z

/publish connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1201885536
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1201885536

add parquet parser

9f407ad

antixar linked an issue Aug 10, 2021 that may be closed by this pull request

Support Parquet format in S3 source #5102

Closed

antixar self-assigned this Aug 10, 2021

github-actions bot added the area/connectors Connector related issues label Aug 10, 2021

gl-pix added 2 commits August 11, 2021 16:43

add integration tests for partquet formats

7ce4638

add unit tests for parquet

479bfd3

jrhizor temporarily deployed to more-secrets August 12, 2021 22:26 Inactive

update docs and secrets

b454cc2

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 13, 2021

jrhizor temporarily deployed to more-secrets August 13, 2021 07:54 Inactive

gl-pix added 2 commits August 13, 2021 11:03

Merge remote-tracking branch 'origin/master' into antixar/5102-source…

f4888e7

…-s3-support-parquet

fix incorrect import for tests

96bff17

jrhizor temporarily deployed to more-secrets August 13, 2021 09:05 Inactive

add lib pandas for unit tests

86bdb23

jrhizor temporarily deployed to more-secrets August 13, 2021 09:21 Inactive

antixar requested review from bazarnov and midavadim August 13, 2021 09:47

bazarnov requested changes Aug 23, 2021

View reviewed changes

airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py Outdated Show resolved Hide resolved

gl-pix added 3 commits August 23, 2021 13:28

revert changes of foreign connectors

3d0830f

Merge remote-tracking branch 'origin' into antixar/5102-source-s3-sup…

4cdc9fe

…port-parquet

update secret settings

d311bab

Merge remote-tracking branch 'origin' into antixar/5102-source-s3-sup…

93bc077

…port-parquet

antixar requested a review from bazarnov August 23, 2021 12:46

antixar and others added 2 commits August 30, 2021 15:27

Update airbyte-integrations/connectors/source-s3/source_s3/source_fil…

b874bb6

…es_abstract/formats/parquet_spec.py Co-authored-by: George Claireaux <[email protected]>

Update airbyte-integrations/connectors/source-s3/source_s3/source_fil…

11711f6

…es_abstract/formats/parquet_spec.py Co-authored-by: George Claireaux <[email protected]>

airbytehq deleted a comment from Phlair Aug 30, 2021

gl-pix added 3 commits August 30, 2021 17:56

remove some unused default options

c21a222

Merge remote-tracking branch 'origin' into antixar/5102-source-s3-sup…

ad07e27

…port-parquet

update tests

dcdc0eb

antixar requested a review from Phlair August 30, 2021 22:23

Phlair approved these changes Aug 31, 2021

View reviewed changes

...te-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/parquet_spec.py Outdated Show resolved Hide resolved

update docs

7febc6d

jrhizor temporarily deployed to more-secrets August 31, 2021 13:54 Inactive

bump its version

8670d2c

jrhizor temporarily deployed to more-secrets August 31, 2021 14:17 Inactive

Phlair mentioned this pull request Aug 31, 2021

Improve support for Parquet in abstract files sources (e.g. S3) #5760

Closed

sherifnada removed their request for review September 1, 2021 00:39

Merge remote-tracking branch 'origin' into antixar/5102-source-s3-sup…

7095f56

…port-parquet

jrhizor temporarily deployed to more-secrets September 3, 2021 11:12 Inactive

gl-pix added 3 commits September 4, 2021 01:49

Merge remote-tracking branch 'origin' into antixar/5102-source-s3-sup…

d535170

…port-parquet

Merge remote-tracking branch 'origin' into antixar/5102-source-s3-sup…

6c62c3e

…port-parquet

fix expected test

af286f4

jrhizor temporarily deployed to more-secrets September 4, 2021 22:59 Inactive

jrhizor temporarily deployed to more-secrets September 4, 2021 23:18 Inactive

antixar merged commit e5c44e6 into master Sep 4, 2021

antixar deleted the antixar/5102-source-s3-support-parquet branch September 4, 2021 23:40

jrhizor mentioned this pull request Sep 8, 2021

Bump Airbyte version from 0.29.15-alpha to 0.29.16-alpha #5925

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source S3: support of Parquet format #5305

🎉 Source S3: support of Parquet format #5305

antixar commented Aug 10, 2021 •

edited

Loading

antixar commented Aug 12, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 13, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 13, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 13, 2021 •

edited by github-actions bot

Loading

bazarnov left a comment

antixar commented Aug 23, 2021

antixar commented Aug 23, 2021

antixar commented Aug 23, 2021

Phlair left a comment

antixar commented Aug 31, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 31, 2021 •

edited by github-actions bot

Loading

antixar commented Sep 3, 2021 •

edited by github-actions bot

Loading

antixar commented Sep 4, 2021 •

edited by github-actions bot

Loading

antixar commented Sep 4, 2021 •

edited by github-actions bot

Loading

🎉 Source S3: support of Parquet format #5305

🎉 Source S3: support of Parquet format #5305

Conversation

antixar commented Aug 10, 2021 • edited Loading

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

antixar commented Aug 12, 2021 • edited by github-actions bot Loading

antixar commented Aug 13, 2021 • edited by github-actions bot Loading

antixar commented Aug 13, 2021 • edited by github-actions bot Loading

antixar commented Aug 13, 2021 • edited by github-actions bot Loading

bazarnov left a comment

Choose a reason for hiding this comment

antixar commented Aug 23, 2021

antixar commented Aug 23, 2021

antixar commented Aug 23, 2021

Phlair left a comment

Choose a reason for hiding this comment

antixar commented Aug 31, 2021 • edited by github-actions bot Loading

antixar commented Aug 31, 2021 • edited by github-actions bot Loading

antixar commented Sep 3, 2021 • edited by github-actions bot Loading

antixar commented Sep 4, 2021 • edited by github-actions bot Loading

antixar commented Sep 4, 2021 • edited by github-actions bot Loading

antixar commented Aug 10, 2021 •

edited

Loading

antixar commented Aug 12, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 13, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 13, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 13, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 31, 2021 •

edited by github-actions bot

Loading

antixar commented Aug 31, 2021 •

edited by github-actions bot

Loading

antixar commented Sep 3, 2021 •

edited by github-actions bot

Loading

antixar commented Sep 4, 2021 •

edited by github-actions bot

Loading

antixar commented Sep 4, 2021 •

edited by github-actions bot

Loading