Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move standard, alibaba and common.sql provider to the new structure #45964

Merged
merged 1 commit into from
Jan 25, 2025

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Jan 23, 2025

This one also adds some missing steps in provider migration script. improves the output and instructions and make it more robust for various variations of provider features and usage of imports.

With those changes standard provider was 100% moved automatically.

Also common.sql needs to be added in order to migrate standard in
order to build production image.

Alibaba was added accidentally during demo at dev call so we are
committing it as well since it is already migrated.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Copy link
Member

@gopidesupavan gopidesupavan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool :)

@potiuk potiuk changed the title Move standad provider to the new structure Move standard provider to the new structure Jan 23, 2025
@potiuk potiuk force-pushed the move-standard-provider branch from e964e91 to 2b1d299 Compare January 23, 2025 14:45
@potiuk
Copy link
Member Author

potiuk commented Jan 23, 2025

Got some fixes in 🤞 . also found a few small things - now IDE integration works nicely with the providers in separate folders (it was broken for a few days for the new providers).

@potiuk
Copy link
Member Author

potiuk commented Jan 23, 2025

Also found out that decorators/test_python.py have not been moved to standards provider when we moved stuff :).

This is actually nice thing about this splitting - we will likely find out all the stuff where providers are still imported from core tests :)

@potiuk
Copy link
Member Author

potiuk commented Jan 23, 2025

And there are still two things to fix I see :)

@dimberman
Copy link
Contributor

@potiuk I've created a PR here with a minimal k8s fix to ensure that the fix provided will fix our tests too. Thank you for responding quickly!

#45974

@dimberman
Copy link
Contributor

@potiuk
Copy link
Member Author

potiuk commented Jan 23, 2025

@potiuk potiuk force-pushed the move-standard-provider branch from 2b1d299 to cff724e Compare January 23, 2025 19:14
@potiuk
Copy link
Member Author

potiuk commented Jan 23, 2025

OK. looks like I have to convert common.sql first - to make that one succeed

@potiuk potiuk changed the title Move standard and common.sql provider to the new structure Move standard, alibaba and common.sql provider to the new structure Jan 24, 2025
@potiuk
Copy link
Member Author

potiuk commented Jan 24, 2025

I also accidentally added alibaba yesterday during demo 🤦 ... But since it's already migrated, I will leave it in - updated description and title.

@potiuk
Copy link
Member Author

potiuk commented Jan 24, 2025

And added few more replacements (logo/ .pre-commit-config.yaml) that could also be automated easliy (standard provider did not have them - but alibaba did).

@potiuk potiuk force-pushed the move-standard-provider branch 2 times, most recently from 5912b00 to 049521c Compare January 24, 2025 21:26
@gopidesupavan
Copy link
Member

Hah... Found a few things missing from the previous moves - for example task_sdk was still present and installed in our compatibility tests - and started to mess around with pre-airflow-3 versions :).

oh good catch, thanks Jarek, this move revealing good stuff. :)

@gopidesupavan
Copy link
Member

Also found out that decorators/test_python.py have not been moved to standards provider when we moved stuff :).

This is actually nice thing about this splitting - we will likely find out all the stuff where providers are still imported from core tests :)

ah i see, yes indeed.

@potiuk potiuk force-pushed the move-standard-provider branch from 049521c to 12b2a6f Compare January 25, 2025 08:31
@potiuk
Copy link
Member Author

potiuk commented Jan 25, 2025

oh good catch, thanks Jarek, this move revealing good stuff. :)

Yeah. I actually love to do these kinds of refactoring, they usually reval some dead bodies in the closet :D

@potiuk potiuk force-pushed the move-standard-provider branch from 12b2a6f to 1dd2891 Compare January 25, 2025 10:27
@potiuk potiuk requested a review from o-nikolas as a code owner January 25, 2025 10:27
@potiuk potiuk force-pushed the move-standard-provider branch from 1dd2891 to 035bf6c Compare January 25, 2025 10:27
@potiuk
Copy link
Member Author

potiuk commented Jan 25, 2025

And we need to add task-sdk at least for now - until we release it in PyPI - in order to run various provider verification tests as those are run with packages built and installed from sources :).

Another step closer to get to the package split - @ashb

@potiuk potiuk force-pushed the move-standard-provider branch 8 times, most recently from 15cadb9 to 796c207 Compare January 25, 2025 20:43
This one also adds some missing steps in provider migration script.
improves the output and instructions and make it more robust for
various variations of provider features and usage of imports.

With those changes `standard` provider was 100% moved automatically.

Also common.sql needs to be added in order to migrate standard in
order to build production image.

Alibaba was added accidentally during demo at dev call so we are
committing it as well since it is already migrated.
@potiuk potiuk force-pushed the move-standard-provider branch from 796c207 to bb45cd7 Compare January 25, 2025 21:35
@potiuk
Copy link
Member Author

potiuk commented Jan 25, 2025

Finally got it green... This single PR allowed me to trace and fix quite a number of small issues we had with moving to the "providers" and now to the "newer providers" strucuture... Next step will be to launch full-blown migration process :).

@potiuk potiuk merged commit 29b9e8e into apache:main Jan 25, 2025
91 checks passed
@potiuk potiuk deleted the move-standard-provider branch January 25, 2025 22:27
@utkarsharma2 utkarsharma2 added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jan 27, 2025
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
…pache#45964)

This one also adds some missing steps in provider migration script.
improves the output and instructions and make it more robust for
various variations of provider features and usage of imports.

With those changes `standard` provider was 100% moved automatically.

Also common.sql needs to be added in order to migrate standard in
order to build production image.

Alibaba was added accidentally during demo at dev call so we are
committing it as well since it is already migrated.
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
…pache#45964)

This one also adds some missing steps in provider migration script.
improves the output and instructions and make it more robust for
various variations of provider features and usage of imports.

With those changes `standard` provider was 100% moved automatically.

Also common.sql needs to be added in order to migrate standard in
order to build production image.

Alibaba was added accidentally during demo at dev call so we are
committing it as well since it is already migrated.
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
…pache#45964)

This one also adds some missing steps in provider migration script.
improves the output and instructions and make it more robust for
various variations of provider features and usage of imports.

With those changes `standard` provider was 100% moved automatically.

Also common.sql needs to be added in order to migrate standard in
order to build production image.

Alibaba was added accidentally during demo at dev call so we are
committing it as well since it is already migrated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools area:providers changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) kind:documentation provider:standard
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants