🎉 Source Freshdesk: migrate to CDK #8682

vitaliizazmic · 2021-12-09T22:46:17Z

What

Refactoring Source Freshdesk to use the CDK

How

This task is part of issue #6164. The Source significantly relies on dead CDK code. So import BaseClient and BaseSource from base_python were replaced by import from deprecated package of the CDK. Gradle file, setup.py, Dockerfile, requirements.txt were updated. main_dev.py was renamed to main.py, updated, and added to .dockerignore. Also SAT was enabled to Source.

Community member or Airbyter

…_cdk

vitaliizazmic · 2021-12-09T22:46:50Z

/test connector=source-freshdesk

🕑 source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1561023685
✅ source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1561023685
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_freshdesk/__init__.py       2      0   100%
	 source_freshdesk/api.py          190     86    55%
	 source_freshdesk/client.py        38     10    74%
	 source_freshdesk/errors.py         8      0   100%
	 source_freshdesk/source.py         4      0   100%
	 source_freshdesk/utils.py         42     23    45%
	 --------------------------------------------------
	 TOTAL                            284    119    58%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_freshdesk/__init__.py       2      0   100%
	 source_freshdesk/api.py          190     75    61%
	 source_freshdesk/client.py        38     18    53%
	 source_freshdesk/errors.py         8      0   100%
	 source_freshdesk/source.py         4      0   100%
	 source_freshdesk/utils.py         42      2    95%
	 --------------------------------------------------
	 TOTAL                            284     95    67%

yevhenii-ldv · 2021-12-10T12:44:54Z

airbyte-integrations/connectors/source-freshdesk/integration_tests/catalog.json

+{
+  "streams": [
+    {
+      "name": "agents",


Do we need this file? Haven't seen it used anywhere

Looks like someone someone by mistake added it here instead of at secrets

yevhenii-ldv · 2021-12-10T12:48:36Z

airbyte-integrations/connectors/source-freshdesk/integration_tests/configured_catalog.json

+      "stream": {
+        "name": "agents",
+        "json_schema": {},
+        "supported_sync_modes": ["incremental"],


A very strange catalog. Why is supported_sync_modes a incremental, can't it be the same as full_refresh?

yevhenii-ldv · 2021-12-10T12:49:08Z

airbyte-integrations/connectors/source-freshdesk/integration_tests/configured_catalog.json

+        "json_schema": {},
+        "supported_sync_modes": ["incremental"],
+        "source_defined_cursor": true,
+        "default_cursor_field": null


Is this stream can be Incremental, why default_cursor_field is null?

yevhenii-ldv · 2021-12-10T12:49:15Z

airbyte-integrations/connectors/source-freshdesk/integration_tests/configured_catalog.json

+        "default_cursor_field": null
+      },
+      "sync_mode": "incremental",
+      "cursor_field": null,


keu · 2021-12-10T14:54:00Z

airbyte-integrations/connectors/source-freshdesk/acceptance-test-config.yml

+    - config_path: "secrets/config.json"
+  basic_read:
+    - config_path: "secrets/config.json"
+      configured_catalog_path: "integration_tests/configured_catalog.json"


we don't need configured catalog if test all streams

Suggested change

configured_catalog_path: "integration_tests/configured_catalog.json"

keu · 2021-12-10T14:54:10Z

airbyte-integrations/connectors/source-freshdesk/acceptance-test-config.yml

+      empty_streams: ["satisfaction_ratings", "tickets", "time_entries", "conversations"]
+  incremental:
+    - config_path: "secrets/config.json"
+      configured_catalog_path: "integration_tests/configured_catalog.json"


Suggested change

configured_catalog_path: "integration_tests/configured_catalog.json"

keu · 2021-12-10T14:54:18Z

airbyte-integrations/connectors/source-freshdesk/acceptance-test-config.yml

+      future_state_path: "integration_tests/abnormal_state.json"
+  full_refresh:
+    - config_path: "secrets/config.json"
+      configured_catalog_path: "integration_tests/configured_catalog.json"


Suggested change

configured_catalog_path: "integration_tests/configured_catalog.json"

keu · 2021-12-10T14:55:22Z

airbyte-integrations/connectors/source-freshdesk/build.gradle

@@ -2,28 +2,10 @@ plugins {
    id 'airbyte-python'
    id 'airbyte-docker'
    id 'airbyte-integration-test-java'


Suggested change

id 'airbyte-integration-test-java'

keu · 2021-12-10T14:57:41Z

airbyte-integrations/connectors/source-freshdesk/source_freshdesk/client.py

@@ -5,8 +5,8 @@

 from typing import Any, Iterable, Mapping, Tuple

-from airbyte_protocol import AirbyteStream
-from base_python import BaseClient
+from airbyte_cdk.models.airbyte_protocol import AirbyteStream


Suggested change

from airbyte_cdk.models.airbyte_protocol import AirbyteStream

from airbyte_cdk.models import AirbyteStream

keu

see comments

vitaliizazmic · 2021-12-13T11:12:00Z

/test connector=source-freshdesk

🕑 source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1572584430
✅ source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1572584430
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_freshdesk/__init__.py       2      0   100%
	 source_freshdesk/api.py          190     86    55%
	 source_freshdesk/client.py        38     10    74%
	 source_freshdesk/errors.py         8      0   100%
	 source_freshdesk/source.py         4      0   100%
	 source_freshdesk/utils.py         42     23    45%
	 --------------------------------------------------
	 TOTAL                            284    119    58%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_freshdesk/__init__.py       2      0   100%
	 source_freshdesk/api.py          190     75    61%
	 source_freshdesk/client.py        38     18    53%
	 source_freshdesk/errors.py         8      0   100%
	 source_freshdesk/source.py         4      0   100%
	 source_freshdesk/utils.py         42      2    95%
	 --------------------------------------------------
	 TOTAL                            284     95    67%

keu

lgtm

…_cdk

vitaliizazmic · 2021-12-14T14:16:00Z

/publish connector=connectors/source-freshdesk

🕑 connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1578247444
✅ connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1578247444

vitaliizazmic added 5 commits November 28, 2021 23:07

Source Freshdesk #7764 - migrate to the CDK

3669262

Merge branch 'master' into vitalii/7764_freshdesk_refactor_to_use_the…

35e8b21

…_cdk

Source Freshdesk #7764 - acceptance tests

e13689d

Source Freshdesk #7764 - update state

13c89e5

Merge branch 'master' into vitalii/7764_freshdesk_refactor_to_use_the…

2e42213

…_cdk

vitaliizazmic added the area/connectors Connector related issues label Dec 9, 2021

vitaliizazmic added this to the Connectors Dec 10 2021 milestone Dec 9, 2021

vitaliizazmic self-assigned this Dec 9, 2021

jrhizor temporarily deployed to more-secrets December 9, 2021 22:48 Inactive

vitaliizazmic requested review from augan-rymkhan, eliziario, keu and sergei-solonitcyn December 9, 2021 23:05

yevhenii-ldv reviewed Dec 10, 2021

View reviewed changes

keu reviewed Dec 10, 2021

View reviewed changes

keu suggested changes Dec 10, 2021

View reviewed changes

VasylLazebnyk removed this from the Connectors Dec 10 2021 milestone Dec 13, 2021

Source Freshdesk #7764 - fixing according to PR

78b63bf

vitaliizazmic temporarily deployed to more-secrets December 13, 2021 11:15 Inactive

jrhizor temporarily deployed to more-secrets December 13, 2021 11:16 Inactive

vitaliizazmic requested a review from keu December 13, 2021 11:44

Source Freshdesk #7764 - fixing client according to PR

c90b0cb

vitaliizazmic temporarily deployed to more-secrets December 13, 2021 11:49 Inactive

keu approved these changes Dec 13, 2021

View reviewed changes

keu changed the title ~~🎉 Source Freshdesk: migrate to the CDK~~ 🎉 Source Freshdesk: migrate to CDK Dec 13, 2021

Merge branch 'master' into vitalii/7764_freshdesk_refactor_to_use_the…

856482f

…_cdk

vitaliizazmic temporarily deployed to more-secrets December 14, 2021 13:38 Inactive

Source Freshdesk #7764 - bump version and update docs

c1e113c

github-actions bot added the area/documentation Improvements or additions to documentation label Dec 14, 2021

vitaliizazmic temporarily deployed to more-secrets December 14, 2021 14:16 Inactive

jrhizor temporarily deployed to more-secrets December 14, 2021 14:17 Inactive

vitaliizazmic temporarily deployed to more-secrets December 14, 2021 14:36 Inactive

vitaliizazmic merged commit ad44895 into master Dec 14, 2021

vitaliizazmic deleted the vitalii/7764_freshdesk_refactor_to_use_the_cdk branch December 14, 2021 15:05

ycherniaiev added connectors/sources-api connectors/source/freshdesk labels Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Freshdesk: migrate to CDK #8682

🎉 Source Freshdesk: migrate to CDK #8682

vitaliizazmic commented Dec 9, 2021

vitaliizazmic commented Dec 9, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv Dec 10, 2021

eliziario Dec 10, 2021

yevhenii-ldv Dec 10, 2021

yevhenii-ldv Dec 10, 2021

yevhenii-ldv Dec 10, 2021

keu Dec 10, 2021

keu Dec 10, 2021

keu Dec 10, 2021

keu Dec 10, 2021

keu Dec 10, 2021

keu left a comment

vitaliizazmic commented Dec 13, 2021 •

edited by github-actions bot

Loading

keu left a comment

vitaliizazmic commented Dec 14, 2021 •

edited by github-actions bot

Loading

	from airbyte_cdk.models.airbyte_protocol import AirbyteStream
	from airbyte_cdk.models import AirbyteStream

🎉 Source Freshdesk: migrate to CDK #8682

🎉 Source Freshdesk: migrate to CDK #8682

Conversation

vitaliizazmic commented Dec 9, 2021

What

How

Community member or Airbyter

vitaliizazmic commented Dec 9, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keu left a comment

Choose a reason for hiding this comment

vitaliizazmic commented Dec 13, 2021 • edited by github-actions bot Loading

keu left a comment

Choose a reason for hiding this comment

vitaliizazmic commented Dec 14, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Dec 9, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Dec 13, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Dec 14, 2021 •

edited by github-actions bot

Loading