🎉 Destination Postgres: Add support for additional JDBC URL Params input #11954

noahkawasaki-airbyte · 2022-04-12T23:31:33Z

What

Looking for something easy to get familiar with the connector workflow. Adds ability to use jdbc_url_params in Postgres Destination spec for arbitrary parameters. Solves Postgres Destination for #9501.

Closes #10720

How

Add jdbc_url_params into spec and some tests to make sure the JDBC url has the params appended.

🚨 User Impact 🚨

None, feature is opt-in

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…_url_params

CLAassistant · 2022-04-12T23:31:41Z

All committers have signed the CLA.

noahkawasaki-airbyte · 2022-04-12T23:33:06Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2158128553
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2158128553
No Python unittests run

noahkawasaki-airbyte · 2022-04-12T23:33:22Z

Oh wait Sonar Scan is doing that for me :D

.../source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresSpecTest.java

Co-authored-by: pmossman <[email protected]>

* #10938 fix docs and spec * #10938 add pr * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]>

* update to our newer Dockerfile format * fixes to source-file spec * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]>

* Upgrade dependencies * Upgrade TypeScript

…took all memory of kube pods (#11692)

* Fix countConnectionsForWorkspace to ignore deprecated connections * Fix test expectations

* fix parens * fix formatting for hints * remove hint from non-docusaurus or gitbook doc

… artifact (#11484) * source-bigquery: escape column names and update google cloud bigquery artifact * source-bigquery: escape column names and update google cloud bigquery artifact * source-bigquery: fix formatting * source-bigquery: fix review comments * bump versions * bump versions source_specs.yaml * revert bump versions * fix publish-command.yml * revert publish-command.yml * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]>

* Update airbyte-specification.md * Update upgrading-airbyte.md * Update namespaces.md

* adding env config JOB_KUBE_SIDECAR_CONTAINER_IMAGE_PULL_POLICY to support image pull policy on sidecar containers * add documentation * docs: adding more description and trying to force cla check. * docs: fix wording

noahkawasaki-airbyte · 2022-04-13T23:52:03Z

/publish auto-bump-version=true connector=connectors/source-postgres
/publish auto-bump-version=true connector=connectors/destination-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2164371548
❌ Failed to publish connectors/source-postgres
❌ Couldn't auto-bump version for connectors/source-postgres

noahkawasaki-airbyte · 2022-04-14T00:06:50Z

/publish auto-bump-version=true connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2164417804
❌ Failed to publish connectors/destination-postgres
❌ Couldn't auto-bump version for connectors/destination-postgres

* set up firebase for local cloud deploy * remove log lines * add empty line

noahkawasaki-airbyte · 2022-04-14T00:52:30Z

Seems like a known issue with GHA personal access tokens behaving weirdly sometimes: fatih/vim-go#3104

/publish auto-bump-version=true connector=connectors/source-postgres

…_url_params

…bytehq/airbyte into noah/postgres-jdbc-additional-params

noahkawasaki-airbyte · 2022-04-20T18:44:33Z

omg, closing and reopening clean PR

Add support for source-postgres and destination-postgres to have jdbc…

797b727

…_url_params

github-actions bot added the area/connectors Connector related issues label Apr 12, 2022

Add CHANGELOG note

1d283fa

noahkawasaki-airbyte commented Apr 12, 2022

View reviewed changes

.../source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresSpecTest.java Outdated Show resolved Hide resolved

noahkawasaki-airbyte changed the title ~~Add support for source-postgres and destination-postgres to have jdbc_url_params~~ 🎉 Source and Destination Postgres: Add support for additional JDBC URL Params input Apr 12, 2022

noahkawasaki-airbyte added 2 commits April 12, 2022 16:46

Add in notes to other parts of docs, better message

5b8cb6d

Add negative and positive tests for JDBC URL Params

dcb9e8d

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 12, 2022

Add white space

140ee0b

noahkawasaki-airbyte requested a review from edgao April 13, 2022 00:04

octavia-squidington-iii and others added 12 commits April 12, 2022 17:04

Bump Airbyte version from 0.35.66-alpha to 0.35.67-alpha (#11952)

395cb7e

Co-authored-by: pmossman <[email protected]>

#10938 fix docs and spec (#11803)

a673a47

* #10938 fix docs and spec * #10938 add pr * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]>

Update on-kubernetes.md (#11962)

6adaf71

Source File: Update Dockerfile to latest Airbyte template (#11969)

26aa43b

* update to our newer Dockerfile format * fixes to source-file spec * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]>

Upgrade webapp dependencies (#11940)

1613904

* Upgrade dependencies * Upgrade TypeScript

🐛 [On call / 171] Source Salesforce: fixed the bug when Bulk fetch …

644ace4

…took all memory of kube pods (#11692)

Fix the count connections method (#11955)

2881d62

* Fix countConnectionsForWorkspace to ignore deprecated connections * Fix test expectations

fix parentheses and info/warning/danger formatting in docs (#11981)

14dfc1d

* fix parens * fix formatting for hints * remove hint from non-docusaurus or gitbook doc

doc: update troubleshooting for discourse forum (#11970)

1cf52f6

Docs: correct minor mistakes (#11848)

7bb4bae

* Update airbyte-specification.md * Update upgrading-airbyte.md * Update namespaces.md

Issue: 11867 add option for sidecar pull policy (#11868)

5e00358

* adding env config JOB_KUBE_SIDECAR_CONTAINER_IMAGE_PULL_POLICY to support image pull policy on sidecar containers * add documentation * docs: adding more description and trying to force cla check. * docs: fix wording

Tr1985 referenced this pull request Apr 13, 2022

remove core label, replace with platform (#7305)

6deae28

Use Firebase Auth Emulator host if present (#11958)

fa9c467

* set up firebase for local cloud deploy * remove log lines * add empty line

noahkawasaki-airbyte temporarily deployed to more-secrets April 20, 2022 18:38 Inactive

noahkawasaki-airbyte marked this pull request as ready for review April 20, 2022 18:40

noahkawasaki-airbyte added 14 commits April 20, 2022 11:43

Add support for source-postgres and destination-postgres to have jdbc…

b85ffe0

…_url_params

Add CHANGELOG note

885a1e7

Add in notes to other parts of docs, better message

a7b0566

Add negative and positive tests for JDBC URL Params

0aa8cc1

Add white space

c67f4e4

Bump versions in Dockerfile, I thought autobumper did this lol

583eff5

Manually run processresources

3db4ae0

Apply AbstractJdbcDestination params to DestinationPostgres

0633640

Revert source-postgres, better to do in separate PR

12595f8

processResources

ef20f67

Revert source-postgres docs too

9468575

Simplify PR, reduce scope of string constants

de3e3f4

Bump to 0.3.19 and set as unpublished tag

0646595

Merge branch 'noah/postgres-jdbc-additional-params' of github.com:air…

006d0b6

…bytehq/airbyte into noah/postgres-jdbc-additional-params

noahkawasaki-airbyte requested a review from a team as a code owner April 20, 2022 18:43

github-actions bot added area/frontend area/platform issues related to the platform area/scheduler area/server area/worker Related to worker CDK Connector Development Kit kubernetes normalization labels Apr 20, 2022

noahkawasaki-airbyte closed this Apr 20, 2022

noahkawasaki-airbyte removed the request for review from a team April 20, 2022 18:52

swyxio deleted the noah/postgres-jdbc-additional-params branch October 12, 2022 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination Postgres: Add support for additional JDBC URL Params input #11954

🎉 Destination Postgres: Add support for additional JDBC URL Params input #11954

noahkawasaki-airbyte commented Apr 12, 2022 •

edited

Loading

CLAassistant commented Apr 12, 2022 •

edited

Loading

noahkawasaki-airbyte commented Apr 12, 2022 •

edited by github-actions bot

Loading

noahkawasaki-airbyte commented Apr 12, 2022

noahkawasaki-airbyte commented Apr 13, 2022 •

edited by github-actions bot

Loading

noahkawasaki-airbyte commented Apr 14, 2022 •

edited by github-actions bot

Loading

noahkawasaki-airbyte commented Apr 14, 2022

noahkawasaki-airbyte commented Apr 20, 2022

🎉 Destination Postgres: Add support for additional JDBC URL Params input #11954

🎉 Destination Postgres: Add support for additional JDBC URL Params input #11954

Conversation

noahkawasaki-airbyte commented Apr 12, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Apr 12, 2022 • edited Loading

noahkawasaki-airbyte commented Apr 12, 2022 • edited by github-actions bot Loading

noahkawasaki-airbyte commented Apr 12, 2022

noahkawasaki-airbyte commented Apr 13, 2022 • edited by github-actions bot Loading

noahkawasaki-airbyte commented Apr 14, 2022 • edited by github-actions bot Loading

noahkawasaki-airbyte commented Apr 14, 2022

noahkawasaki-airbyte commented Apr 20, 2022

noahkawasaki-airbyte commented Apr 12, 2022 •

edited

Loading

CLAassistant commented Apr 12, 2022 •

edited

Loading

noahkawasaki-airbyte commented Apr 12, 2022 •

edited by github-actions bot

Loading

noahkawasaki-airbyte commented Apr 13, 2022 •

edited by github-actions bot

Loading

noahkawasaki-airbyte commented Apr 14, 2022 •

edited by github-actions bot

Loading