Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Normalization ssh tunnel - replace '\\n' -> '\n' #9793

Merged
merged 1 commit into from
Jan 27, 2022
Merged

Normalization ssh tunnel - replace '\\n' -> '\n' #9793

merged 1 commit into from
Jan 27, 2022

Conversation

grubberr
Copy link
Contributor

@grubberr grubberr commented Jan 25, 2022 •
edited
Loading

Signed-off-by: Sergey Chvalyuk [email protected]

What

Users can provide SSH-TUNNEL key certificate in PEM format where newline characters \n replaced by \\n.
For example, this can happen if the user copy-paste the PEM certificate from the JSON file.
From the parser point of view, such PEM certificates are not correct
but because backslash symbols are not allowed in base64 encoding
we can improve user experience and safely replace back \\n -> \n
Similar PR #8371 for Java connectors.

How

Describe the solution

Recommended reading order

  1. x.java
  2. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

@grubberr
replace '\\n' -> '\n'
5b16d6f 
Signed-off-by: Sergey Chvalyuk <[email protected]>
@github-actions github-actions bot added area/platform issues related to the platform area/worker Related to worker normalization labels Jan 25, 2022
@grubberr grubberr linked an issue Jan 25, 2022 that may be closed by this pull request
Destination PostgreSQL Database Connection Error on normalization step using SSH_KEY_AUTH #9569
Closed
@grubberr grubberr changed the title replace '\\n' -> '\n' Normalization ssh tunnel - replace '\\n' -> '\n' Jan 25, 2022
@grubberr grubberr temporarily deployed to more-secrets January 25, 2022 18:11 Inactive
@grubberr grubberr self-assigned this Jan 25, 2022
@grubberr grubberr requested review from midavadim and antixar January 25, 2022 18:21
antixar
antixar approved these changes Jan 26, 2022
View reviewed changes
Copy link
Contributor

@antixar antixar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@grubberr
Copy link
Contributor Author

grubberr commented Jan 26, 2022 •
edited by github-actions bot
Loading

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1752001161
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1752001161
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     155      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520     39    92%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     46    69%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    188    85%

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 26, 2022 17:00 Inactive
@grubberr
Copy link
Contributor Author

grubberr commented Jan 26, 2022 •
edited by github-actions bot
Loading

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1752732456
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1752732456

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 26, 2022 19:53 Inactive
@grubberr
Copy link
Contributor Author

grubberr commented Jan 26, 2022 •
edited by github-actions bot
Loading

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1753008422
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1753008422

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 26, 2022 21:04 Inactive
@grubberr grubberr merged commit e53e290 into master Jan 27, 2022
@grubberr grubberr deleted the grubberr/9569-base-normalization_ssj_key_nl_fix branch January 27, 2022 07:22
This was referenced Jan 28, 2022
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@antixar antixar antixar approved these changes

@midavadim midavadim Awaiting requested review from midavadim

Assignees

@grubberr grubberr

Labels
area/platform issues related to the platform area/worker Related to worker normalization
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Destination PostgreSQL Database Connection Error on normalization step using SSH_KEY_AUTH
3 participants
@grubberr @antixar @octavia-squidington-iii