Revert back to erroring out on a null cursor value #21825

rodireich · 2023-01-24T21:28:49Z

What

Put back the original change that stopped a sync when a null cursor value is discovered.
This will reverse the temporary fix we put in #oncall/1159.

rodireich · 2023-01-24T21:29:52Z

/test connector=connectors/source-postgres

github-actions · 2023-01-24T21:30:50Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

❌ Sources (25)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.36`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.36`	🔵 (ignored)	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.15`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.15`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.19`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.17`	✅	✅
`source-db2-strict-encrypt`	`0.1.17`	🔵 (ignored)	🔵 (ignored)
`source-dynamodb`	`0.1.0`	✅	✅
`source-jdbc`	`0.3.5`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`0.4.28`	✅	✅
`source-mssql-strict-encrypt`	`0.4.28`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`1.0.19`	✅	✅
`source-mysql-strict-encrypt`	`1.0.19`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.22`	✅	✅
`source-oracle-strict-encrypt`	`0.3.22`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`1.0.40`	✅	❌ (diff seed version)
`source-postgres-strict-encrypt`	`1.0.40`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.16`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-snowflake`	`0.1.29`	✅	✅
`source-tidb`	`0.2.2`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

github-actions · 2023-01-24T21:38:55Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	23.98%	❌

rodireich · 2023-01-24T22:16:48Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4000895681
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4000895681
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:22: `future_state` has a bypass reason, skipping.
================== 54 passed, 6 skipped in 716.80s (0:11:56) ===================

rodireich · 2023-01-24T22:49:59Z

/test connector=connectors/source-alloydb

🕑 connectors/source-alloydb https://github.com/airbytehq/airbyte/actions/runs/4001106153
✅ connectors/source-alloydb https://github.com/airbytehq/airbyte/actions/runs/4001106153
No Python unittests run

Build Passed

Test summary info:

All Passed

ryankfu

LGTM

rodireich · 2023-01-24T23:20:23Z

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4001302403

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

rodireich · 2023-01-24T23:21:31Z

/publish connector=connectors/source-alloydb

🕑 Publishing the following connectors:
connectors/source-alloydb
https://github.com/airbytehq/airbyte/actions/runs/4001309186

Connector	Did it publish?	Were definitions generated?
connectors/source-alloydb	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

rodireich · 2023-01-24T23:33:23Z

/publish connector=connectors/source-alloydb-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-alloydb-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4001378300

Connector	Did it publish?	Were definitions generated?
connectors/source-alloydb-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

rodireich · 2023-01-25T00:59:39Z

/publish connector=connectors/source-postgres-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4001863565

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

revert back to erroring out on a null cursor value

c44235a

rodireich linked an issue Jan 24, 2023 that may be closed by this pull request

Database Sources: Handle when a cursor value is NULL #17150

Closed

octavia-squidington-iv added area/connectors Connector related issues connectors/source/postgres connectors/source/relational-db labels Jan 24, 2023

rodireich temporarily deployed to more-secrets January 24, 2023 21:30 — with GitHub Actions Inactive

rodireich changed the title ~~revert back to erroring out on a null cursor value~~ Revert back to erroring out on a null cursor value Jan 24, 2023

rodireich requested review from akashkulk, sergio-ropero, subodh1810, grishick, VitaliiMaltsev and sashaNeshcheret January 24, 2023 23:06

rodireich marked this pull request as ready for review January 24, 2023 23:08

rodireich requested a review from a team as a code owner January 24, 2023 23:08

ryankfu approved these changes Jan 24, 2023

View reviewed changes

Update dockerfiles and release notes

5d8e91f

octavia-squidington-iv added area/documentation Improvements or additions to documentation connectors/source/postgres-strict-encrypt connectors/source/alloydb connectors/source/alloydb-strict-encrypt labels Jan 24, 2023

rodireich temporarily deployed to more-secrets January 24, 2023 23:21 — with GitHub Actions Inactive

auto-bump connector version

1623cfe

octavia-squidington-iii temporarily deployed to more-secrets January 24, 2023 23:34 — with GitHub Actions Inactive

rodireich closed this Jan 25, 2023

rodireich reopened this Jan 25, 2023

rodireich temporarily deployed to more-secrets January 25, 2023 01:27 — with GitHub Actions Inactive

rodireich enabled auto-merge (squash) January 25, 2023 01:28

rodireich merged commit 6d0177a into master Jan 25, 2023

rodireich deleted the 17150-put-back-null-cursor-val-error branch January 25, 2023 02:01

rodireich mentioned this pull request Jan 25, 2023

Bump postgres version in source_definitions file #21886

Merged

github-actions bot mentioned this pull request Jan 25, 2023

Bump helm chart version reference to 0.43.24 #21887

Closed

This was referenced Jan 27, 2023

Bump Airbyte version from 0.40.29 to 0.40.30 #21954

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #21991

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #22045

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert back to erroring out on a null cursor value #21825

Revert back to erroring out on a null cursor value #21825

rodireich commented Jan 24, 2023 •

edited

Loading

rodireich commented Jan 24, 2023

github-actions bot commented Jan 24, 2023 •

edited

Loading

❌ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

github-actions bot commented Jan 24, 2023 •

edited

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

ryankfu left a comment

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 25, 2023 •

edited by github-actions bot

Loading

Revert back to erroring out on a null cursor value #21825

Revert back to erroring out on a null cursor value #21825

Conversation

rodireich commented Jan 24, 2023 • edited Loading

What

rodireich commented Jan 24, 2023

github-actions bot commented Jan 24, 2023 • edited Loading

Affected Connector Report

❌ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

github-actions bot commented Jan 24, 2023 • edited Loading

Airbyte Code Coverage

rodireich commented Jan 24, 2023 • edited by github-actions bot Loading

Build Passed

rodireich commented Jan 24, 2023 • edited by github-actions bot Loading

Build Passed

ryankfu left a comment

Choose a reason for hiding this comment

rodireich commented Jan 24, 2023 • edited by github-actions bot Loading

rodireich commented Jan 24, 2023 • edited by github-actions bot Loading

rodireich commented Jan 24, 2023 • edited by github-actions bot Loading

rodireich commented Jan 25, 2023 • edited by github-actions bot Loading

rodireich commented Jan 24, 2023 •

edited

Loading

github-actions bot commented Jan 24, 2023 •

edited

Loading

github-actions bot commented Jan 24, 2023 •

edited

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 24, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 25, 2023 •

edited by github-actions bot

Loading