🎉 Source Postgres: support all Postgres 14 types #8726

tuliren · 2021-12-12T06:58:18Z

What

Support all Postgres types.
Except two remaining issues that we need to confirm on the destination side if the solution works:
- Postgres source date includes time, and cannot handle BC dates #8903
- https://github.com/airbytehq/airbyte-internal-issues/issues/358

How

Postgres does not have a public type enum. So the MySQL approach won't work for Postgres.
We use JDBCType for most of the types. For corner cases, we check the column type name returned from JDBC, and treat them differently.

🚨 User Impact 🚨

This PR changes some of the Postgres data types. It is published with a minor version bump.

Pre-merge Checklist

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

tuliren · 2021-12-13T07:42:07Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1571824409
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1571824409
🐛 https://gradle.com/s/ikzuaoro3mups

sherifnada · 2021-12-15T10:19:21Z

...postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceOperations.java

+    } else if (value.equalsIgnoreCase("-infinity")) {
+      node.put(columnName, Double.NEGATIVE_INFINITY);
+    } else if (value.equalsIgnoreCase("nan")) {
+      node.put(columnName, Double.NaN);


does json have a concept of infinity and Nan? how are these represented when output from the connector itself?

Good catch. Actually Json numbers do not support NaN or infinity. To support these three special values, we need to make sure that the destination can handle them. Otherwise, the destination will fail. Too bad, I need to revert this.

like how we currently handle dates, you can output them in a string type with a format hint.

Then have normalization handle the special format hint for these special numbers accordingly for the destination

@ChristopheDuong, nice.

Where the code that handles this? I'd like to see what the format hint looks like.

In the json, it's "string" type.

Then, on normalization, if we find a format hint, we know it's actually a date not a string:

airbyte/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py

Line 33 in c5fc568

and (definition["format"] == "date" or "date" in definition["format"])

Example of catalog.json with date:

airbyte/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/catalog.json

Line 17 in c5fc568

"format": "date"

so you can have a special string-float type instead of using json's number in order to encode special float values:

"db_float_column": { "type": "string", "format": "float" }

and normalization can handle it in SQL: For example in bigquery, based on https://stackoverflow.com/a/53692265:

there is no literal representation of NaN or infinity, but the following case-insensitive strings can be explicitly cast to float:

"NaN"
"inf" or "+inf"
"-inf"

case when db_float_column = '-infinity' then cast("-inf" as float) when db_float_column = '+infinity' then cast("+inf" as float) else cast(db_float_coumn as float) end

or differently for another destination

…ns with JDBCType.ARRAY (#8749) * support array for jdbc sources * fixed PR comments, added test cases * added more elements for test case * Fix test case * add array test case for JdbcSourceOperations Co-authored-by: Liren Tu <[email protected]>

Postgres source cannot handle these special values yet See https://github.com/airbytehq/airbyte/issues/8902

tuliren · 2021-12-20T01:07:53Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1599988085
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1599988085
🐛 https://gradle.com/s/qcegtyclrphw2

This reverts commit 3bee7d1.

tuliren · 2021-12-20T03:53:21Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600389644
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600389644
No Python unittests run

tuliren · 2021-12-20T05:16:30Z

/publish connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600584483
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1600584483

* Add skeleton to support all postgres types * Consolidate type tests * Fix corner cases * Bump postgres version * Add tests for time and timetz * Format code * Revert date to timestamp * Update comment * Fix unit tests * 🐛 Jdbc sources: switch from "string" to "array" schema type for columns with JDBCType.ARRAY (airbytehq#8749) * support array for jdbc sources * fixed PR comments, added test cases * added more elements for test case * Fix test case * add array test case for JdbcSourceOperations Co-authored-by: Liren Tu <[email protected]> * Revert changes to support special number values Postgres source cannot handle these special values yet See https://github.com/airbytehq/airbyte/issues/8902 * Revert infinity and nan assertion in unit tests This reverts commit 3bee7d1. * Update documentation * Bump postgres source version in seed Co-authored-by: Yurii Bidiuk <[email protected]>

github-actions bot added the area/connectors Connector related issues label Dec 12, 2021

tuliren temporarily deployed to more-secrets December 12, 2021 06:58 Inactive

tuliren mentioned this pull request Dec 12, 2021

🎉 Source Postgres: support all Postgres 14 types #8308

Closed

40 tasks

tuliren force-pushed the liren/postgres-types-v2 branch from c64f2b4 to 5266017 Compare December 12, 2021 18:37

tuliren temporarily deployed to more-secrets December 12, 2021 18:39 Inactive

Add skeleton to support all postgres types

2fb2000

tuliren force-pushed the liren/postgres-types-v2 branch from 5266017 to 2fb2000 Compare December 13, 2021 00:31

tuliren temporarily deployed to more-secrets December 13, 2021 00:33 Inactive

Consolidate type tests

ea13259

tuliren linked an issue Dec 13, 2021 that may be closed by this pull request

Support all types in PostgreSQL source #7727

Closed

Fix corner cases

1d6440d

tuliren temporarily deployed to more-secrets December 13, 2021 06:35 Inactive

Bump postgres version

0f46c26

github-actions bot added the area/documentation Improvements or additions to documentation label Dec 13, 2021

This was linked to issues Dec 13, 2021

Source Postgres: DB values '-Infinity', 'Infinity', 'Nan' are parsed as null #7871

Closed

Source Postgres: parsed value for type 'bytea' is encoded by Base64 #7905

Closed

tuliren requested review from yurii-bidiuk and sherifnada December 13, 2021 07:07

tuliren temporarily deployed to more-secrets December 13, 2021 07:07 Inactive

jrhizor temporarily deployed to more-secrets December 13, 2021 07:09 Inactive

Add tests for time and timetz

b119f42

tuliren marked this pull request as ready for review December 13, 2021 07:42

tuliren requested a review from etsybaev December 13, 2021 07:42

tuliren temporarily deployed to more-secrets December 13, 2021 07:43 Inactive

jrhizor temporarily deployed to more-secrets December 13, 2021 07:44 Inactive

tuliren requested a review from andriikorotkov December 13, 2021 07:48

Format code

97569e7

tuliren temporarily deployed to more-secrets December 13, 2021 07:53 Inactive

Revert date to timestamp

e1dac8e

sherifnada approved these changes Dec 15, 2021

View reviewed changes

alexandr-shegeda approved these changes Dec 15, 2021

View reviewed changes

yurii-bidiuk temporarily deployed to more-secrets December 16, 2021 09:47 Inactive

etsybaev mentioned this pull request Dec 16, 2021

Support all types in SQL Server source #7728

Closed

Merge branch 'master' into liren/postgres-types-v2

5ae72e8

tuliren temporarily deployed to more-secrets December 17, 2021 23:42 Inactive

Merge branch 'master' into liren/postgres-types-v2

9b3eb23

tuliren temporarily deployed to more-secrets December 20, 2021 00:26 Inactive

tuliren mentioned this pull request Dec 20, 2021

Postgres source date includes time, and cannot handle BC dates #8903

Closed

Revert changes to support special number values

aebb263

Postgres source cannot handle these special values yet See https://github.com/airbytehq/airbyte/issues/8902

tuliren temporarily deployed to more-secrets December 20, 2021 01:04 Inactive

jrhizor temporarily deployed to more-secrets December 20, 2021 01:10 Inactive

Revert infinity and nan assertion in unit tests

9926723

This reverts commit 3bee7d1.

tuliren temporarily deployed to more-secrets December 20, 2021 03:48 Inactive

Update documentation

9c4ed9f

tuliren temporarily deployed to more-secrets December 20, 2021 03:53 Inactive

jrhizor temporarily deployed to more-secrets December 20, 2021 03:55 Inactive

jrhizor temporarily deployed to more-secrets December 20, 2021 05:18 Inactive

Bump postgres source version in seed

1b7d7a7

tuliren temporarily deployed to more-secrets December 20, 2021 07:36 Inactive

tuliren merged commit ff4b83b into master Dec 20, 2021

tuliren deleted the liren/postgres-types-v2 branch December 20, 2021 07:59

tuliren mentioned this pull request Dec 20, 2021

🐛 Jdbc sources: switch from "string" to "array" schema type for columns with JDBCType.ARRAY #8749

Merged

40 tasks

jrhizor mentioned this pull request Dec 20, 2021

Bump Airbyte version from 0.34.2-alpha to 0.34.3-alpha #8972

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Postgres: support all Postgres 14 types #8726

🎉 Source Postgres: support all Postgres 14 types #8726

tuliren commented Dec 12, 2021 •

edited

Loading

tuliren commented Dec 13, 2021 •

edited by github-actions bot

Loading

sherifnada Dec 15, 2021

tuliren Dec 18, 2021

ChristopheDuong Dec 20, 2021

tuliren Dec 20, 2021

ChristopheDuong Dec 20, 2021

ChristopheDuong Dec 20, 2021 •

edited

Loading

tuliren commented Dec 20, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 20, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 20, 2021 •

edited by github-actions bot

Loading

🎉 Source Postgres: support all Postgres 14 types #8726

🎉 Source Postgres: support all Postgres 14 types #8726

Conversation

tuliren commented Dec 12, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

tuliren commented Dec 13, 2021 • edited by github-actions bot Loading

sherifnada Dec 15, 2021

Choose a reason for hiding this comment

tuliren Dec 18, 2021

Choose a reason for hiding this comment

ChristopheDuong Dec 20, 2021

Choose a reason for hiding this comment

tuliren Dec 20, 2021

Choose a reason for hiding this comment

ChristopheDuong Dec 20, 2021

Choose a reason for hiding this comment

ChristopheDuong Dec 20, 2021 • edited Loading

Choose a reason for hiding this comment

tuliren commented Dec 20, 2021 • edited by github-actions bot Loading

tuliren commented Dec 20, 2021 • edited by github-actions bot Loading

tuliren commented Dec 20, 2021 • edited by github-actions bot Loading

tuliren commented Dec 12, 2021 •

edited

Loading

tuliren commented Dec 13, 2021 •

edited by github-actions bot

Loading

ChristopheDuong Dec 20, 2021 •

edited

Loading

tuliren commented Dec 20, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 20, 2021 •

edited by github-actions bot

Loading

tuliren commented Dec 20, 2021 •

edited by github-actions bot

Loading