Add Teradata Provider #36953

sc250072 · 2024-01-22T16:26:58Z

Initial Implementation of Airflow Teradata Provider for Teradata - complete cloud analytics and data platform. As part of initial release, Added below functionalities.

Teradata Hook
Teradata Operator
Teradata to Teradata Transfer Operator.

Added required documentation, unit and system tests for Teradata Provider changes.

With this PR:

New Airflow Provider for Teradata will be released.

Teradata Provider System Test Health Board:

https://teradata.github.io/airflow/

Teradata Provider documentation build status: https://github.com/Teradata/airflow/actions/runs/7613734774

Teradata Provider static checks status: https://github.com/Teradata/airflow/actions/runs/7613748694/job/20734385279

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…lytics and data platform Implemented new airflow provider with Hook, Operator and Transfer Operator

sunilkmallam · 2024-01-22T16:35:07Z

Hi Airflow Community,

We've sent a proposal for Teradata being Airflow's new community provider. After 72 hours of voting, we have 9 positive votes (4 binding and 5 non-binding).
Here's a link to the voting thread.

Please review our PR and let us know if there's anything else you'd need from us.

Thanks,
Sunil Mallam (Teradata)

hussein-awala

It looks clean! I will run the tests, let's wait for them

.github/CODEOWNERS

tests/providers/teradata/hooks/test_teradata.py

eladkal

Partial review for now

airflow/providers/teradata/transfers/teradata_to_teradata.py

airflow/providers/teradata/example_dags/example_teradata_to_teradata_transfer_operator.py

airflow/providers/teradata/example_dags/example_teradata_operator.py

…_to_teradata operator

sc250072 · 2024-01-24T16:25:17Z

Partial review for now

Addressed the review comments.

flolas · 2024-01-24T22:00:05Z

Hi @SatishChGit,
I think that many long-time Teradata users rely on BTEQ queries. How about in the future integrating a TtuHook (Teradata tools and utilities) for fastexport/fastload/bteq in our provider? I have a working version with tests here: https://github.com/flolas/apache-airflow-providers-teradata.

sc250072 · 2024-01-25T02:45:24Z

integrating

Thank you for this. We will check the possibility and will add in our roadmap.

sunilkmallam · 2024-01-27T02:48:13Z

Partial review for now

Addressed the review comments.

Hi @eladkal, have you had a chance to review the changes we did?

josh-fell

Solid work here! A few changes needed notwithstanding the example DAG question.

airflow/providers/teradata/example_dags/example_teradata_operator.py

airflow/providers/teradata/hooks/teradata.py

airflow/providers/teradata/operators/teradata.py

docs/apache-airflow-providers-teradata/index.rst

pyproject.toml

tests/providers/teradata/hooks/test_teradata.py

tests/providers/teradata/operators/test_teradata.py

airflow/providers/teradata/hooks/teradata.py

Co-authored-by: Josh Fell <[email protected]>

commit ade9ce3 Author: SatishChGit <[email protected]> Date: Wed Feb 14 01:04:46 2024 -0800 Redesigned teradatatoteradataoperator by removing private method

sunilkmallam · 2024-02-15T03:59:48Z

@airflow team, just following up on the changes we made. Could someone please review them?

airflow/providers/teradata/CHANGELOG.rst

eladkal

LGTM

Co-authored-by: Elad Kalif <[email protected]>

sc250072 · 2024-02-15T09:14:17Z

Solid work here! A few changes needed notwithstanding the example DAG question.

Addressed all review comments.

josh-fell

LGTM! A very minor comment for code snippet formatting in the docs, but that definitely doesn't deter from the work here.

docs/apache-airflow-providers-teradata/operators/teradata.rst

Co-authored-by: Josh Fell <[email protected]>

sc250072 · 2024-02-15T15:00:33Z

LGTM! A very minor comment for code snippet formatting in the docs, but that definitely doesn't deter from the work here.

Thank you for the review. Committed suggested change.

boring-cyborg · 2024-02-15T18:07:37Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

potiuk · 2024-02-15T18:07:46Z

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

josh-fell · 2024-02-15T18:09:30Z

Nice work @SatishChGit, congrats!

sunilkmallam · 2024-02-15T18:17:13Z

Thank you, everyone, for your support! We truly appreciate it!

eladkal · 2024-02-15T18:42:41Z

🎉 🎉

Great work!

sc250072 · 2024-02-16T01:11:42Z

Thank you every one for your support!. When is the airflow next release. After next airflow release, hope teradata provider will be listed at https://airflow.apache.org/docs/#active-providers.

kaxil · 2024-02-21T08:20:40Z

Thank you every one for your support!. When is the airflow next release. After next airflow release, hope teradata provider will be listed at https://airflow.apache.org/docs/#active-providers.

It will be in the next wave of providers for the release, or we can cut an ad-hoc one cc @eladkal

satish-chinthanippu added 2 commits January 22, 2024 21:02

Initial Implementation of new Airflow Provider for Teradata cloud ana…

3bc1c50

…lytics and data platform Implemented new airflow provider with Hook, Operator and Transfer Operator

Static check issues fixed

dbb14e3

sc250072 requested review from potiuk, ashb and jedcunningham as code owners January 22, 2024 16:26

boring-cyborg bot added area:dev-tools area:providers labels Jan 22, 2024

Merge branch 'main' into teradata_release_1.0.0

dcc4808

hussein-awala reviewed Jan 23, 2024

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

tests/providers/teradata/hooks/test_teradata.py Outdated Show resolved Hide resolved

satish-chinthanippu added 2 commits January 23, 2024 10:12

Removed code owner and commented line

98f0d73

Merge branch 'main' into teradata_release_1.0.0

2ad6075

sc250072 requested a review from hussein-awala January 23, 2024 05:43

Merge branch 'main' into teradata_release_1.0.0

0302d1b

eladkal requested changes Jan 23, 2024

View reviewed changes

satish-chinthanippu added 3 commits January 24, 2024 20:09

Updated names as per review comments

e8c4353

Merge branch 'main' into teradata_release_1.0.0

589a819

Renamed parameters in Example DAGs, unit and System tests of teradata…

7accb6a

…_to_teradata operator

sc250072 requested a review from eladkal January 24, 2024 16:21

satish-chinthanippu added 2 commits January 25, 2024 13:42

Merge branch 'main' into teradata_release_1.0.0

09d5214

Merge branch 'main' into teradata_release_1.0.0

fd8c529

sc250072 requested a review from sunilkmallam January 29, 2024 05:47

josh-fell requested changes Jan 30, 2024

View reviewed changes

eladkal changed the title ~~Airflow Teradata Provider: Initial implementation of Airflow Teradata Provider~~ Add Teradata Provider Feb 2, 2024

satish-chinthanippu and others added 2 commits February 2, 2024 16:56

Update airflow/providers/teradata/operators/teradata.py

d02d16f

Co-authored-by: Josh Fell <[email protected]>

Merge remote-tracking branch 'upstream/main' into teradata_release_1.0.0

92c1542

satish-chinthanippu added 2 commits February 14, 2024 01:18

Squashed commit of the following:

e11d169

commit ade9ce3 Author: SatishChGit <[email protected]> Date: Wed Feb 14 01:04:46 2024 -0800 Redesigned teradatatoteradataoperator by removing private method

Merge branch 'main' into teradata_release_1.0.0

bcb014e

sc250072 requested a review from eladkal February 14, 2024 11:35

eladkal added the provider:teradata label Feb 15, 2024

eladkal reviewed Feb 15, 2024

View reviewed changes

airflow/providers/teradata/CHANGELOG.rst Outdated Show resolved Hide resolved

eladkal approved these changes Feb 15, 2024

View reviewed changes

satish-chinthanippu and others added 2 commits February 15, 2024 13:12

Update airflow/providers/teradata/CHANGELOG.rst

dfe9967

Co-authored-by: Elad Kalif <[email protected]>

Merge branch 'main' into teradata_release_1.0.0

a80f746

potiuk approved these changes Feb 15, 2024

View reviewed changes

josh-fell approved these changes Feb 15, 2024

View reviewed changes

docs/apache-airflow-providers-teradata/operators/teradata.rst Show resolved Hide resolved

satish-chinthanippu and others added 2 commits February 15, 2024 20:11

Merge branch 'main' into teradata_release_1.0.0

996b0d6

Update docs/apache-airflow-providers-teradata/operators/teradata.rst

18475fd

Co-authored-by: Josh Fell <[email protected]>

Merge branch 'main' into teradata_release_1.0.0

3b79807

potiuk merged commit c41794d into apache:main Feb 15, 2024
84 checks passed

ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Feb 19, 2024

mkincaid mentioned this pull request Feb 21, 2024

Possible package name conflict between TTU provider and new Teradata provider flolas/apache-airflow-providers-teradata#5

Open

eladkal mentioned this pull request Feb 23, 2024

Status of testing Providers that were prepared on February 23, 2024 #37648

Closed

2 tasks

sc250072 mentioned this pull request Mar 7, 2024

Sync v2-8-stable with v2-8-test to release 2.8.3 #37941

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Teradata Provider #36953

Add Teradata Provider #36953

sc250072 commented Jan 22, 2024

sunilkmallam commented Jan 22, 2024

hussein-awala left a comment

eladkal left a comment

sc250072 commented Jan 24, 2024

flolas commented Jan 24, 2024

sc250072 commented Jan 25, 2024

sunilkmallam commented Jan 27, 2024

josh-fell left a comment

sunilkmallam commented Feb 15, 2024

eladkal left a comment

sc250072 commented Feb 15, 2024

josh-fell left a comment

sc250072 commented Feb 15, 2024

boring-cyborg bot commented Feb 15, 2024

potiuk commented Feb 15, 2024

josh-fell commented Feb 15, 2024

sunilkmallam commented Feb 15, 2024

eladkal commented Feb 15, 2024

sc250072 commented Feb 16, 2024 •

edited

Loading

kaxil commented Feb 21, 2024

Add Teradata Provider #36953

Add Teradata Provider #36953

Conversation

sc250072 commented Jan 22, 2024

sunilkmallam commented Jan 22, 2024

hussein-awala left a comment

Choose a reason for hiding this comment

eladkal left a comment

Choose a reason for hiding this comment

sc250072 commented Jan 24, 2024

flolas commented Jan 24, 2024

sc250072 commented Jan 25, 2024

sunilkmallam commented Jan 27, 2024

josh-fell left a comment

Choose a reason for hiding this comment

sunilkmallam commented Feb 15, 2024

eladkal left a comment

Choose a reason for hiding this comment

sc250072 commented Feb 15, 2024

josh-fell left a comment

Choose a reason for hiding this comment

sc250072 commented Feb 15, 2024

boring-cyborg bot commented Feb 15, 2024

potiuk commented Feb 15, 2024

josh-fell commented Feb 15, 2024

sunilkmallam commented Feb 15, 2024

eladkal commented Feb 15, 2024

sc250072 commented Feb 16, 2024 • edited Loading

kaxil commented Feb 21, 2024

sc250072 commented Feb 16, 2024 •

edited

Loading