-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Removed the unused provider's package #46608
Removed the unused provider's package #46608
Conversation
ce8d8c2
to
c0fda0b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The existence of this directory for the amazon provider was one of the sneaky ways in which tests were breaking when I was migrating that provider. Amazon is used as an example in a lot of tests and some code was detecting it as not migrated because of that.
Any who, I made a note to come back this morning and delete the others but you've already got it, nice!
c0fda0b
to
0085837
Compare
ac0c9b5
to
6fcd868
Compare
e6d99a0
to
3e16bf0
Compare
3e16bf0
to
4ae9b39
Compare
There are still few errors due to cross-provider dependencies with fab provider / azure paths not fixed - but generally it should be ready for review (I will fix the errors tomorrow). |
For those who will be reviewing this one - I recommend |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few sanity check comments, compared to Niko I did not find any glitches. Good to be merged after nit's applied.
👀 👀 👀 👀 👀 👀 👀 👀 |
This is a set of cleanup steps (first stage) that allow us to remove the "intermediate" provider's distribution from Airlfow code and replace it fully with individual provider's distributions - already with own `pyproject.toml` files and basically being (when we complete) a completely separate distributions from Airflow and without implicit dependencies between unrelated distributions. There are a number of other changes needed but that one is only focusing on removing all references to the "umbrella" `providers` distribution and consequences of removing it. Those are the changes implemented in this PR: * There are no separate "providers" system tests - each provider has own system tests and there are no common "generic" providers empty system test * Integration tests are moved to respective providers under the `integration` package inside `tests` directory * (nearly) empty __init__.py files are added in `tests` directories of providers - this way "tests" becomes just a directory and root for all tests per provider, rather than a Python package on its own. That allows to use "from integration.PROVIDER import" and "from system.PROVIDER" rather than importing them from the root of the whole airflow project. The (nearly) is because we need to handle multiple "system", "system.apache" and other import locations. * Removed references to "providers/" generic package which were scheduled for removal after all providers are moved to the new structure * Few remaining references / links referring to old "providers/src" and "providers/test" have been fixed. * The "conftest.py" files in all providers are trimmed down - the code to store ignored deprecation warnings have been moved to the test_common pytest_plugin. That allows to remove 90+ duplicated snippets of deprecation_warnings retrieval while keeping the warnings per-provider in the provider's distribution. * The "moving_providers" scripts are removed. They've done their job and are not needed any more - we keep them in history * The __init__.py files are automatically checked and properly updated in provider folders - in order to properly handle path extension mechanisms * The www tests that were using FAB permisssion model are moved to the FAB provider tests.
df4f601
to
4b8e4a2
Compare
Rebased and applied the nits. |
BTW.I plan to have a separate discussion on the "final" layout of the provider test code so I plan to merge it rather quickly and then write up a discussion thread where I would want people to chime in and make some final decisions. I am afraid if I won't merge it quickly some test conflicts will take more than they are worth. So @o-nikolas @Lee-W -> I think, if the tests pass after rebase, I will merge it, but it is really up to a follow up discussion how the folder and structure and imports will look like eventually (this is all test code so we can remove it again it will be far simpler to do follow-up renames. |
Merged ! |
This is a set of cleanup steps (first stage) that allow us to remove the "intermediate" provider's distribution from Airlfow code and replace it fully with individual provider's distributions - already with own `pyproject.toml` files and basically being (when we complete) a completely separate distributions from Airflow and without implicit dependencies between unrelated distributions. There are a number of other changes needed but that one is only focusing on removing all references to the "umbrella" `providers` distribution and consequences of removing it. Those are the changes implemented in this PR: * There are no separate "providers" system tests - each provider has own system tests and there are no common "generic" providers empty system test * Integration tests are moved to respective providers under the `integration` package inside `tests` directory * (nearly) empty __init__.py files are added in `tests` directories of providers - this way "tests" becomes just a directory and root for all tests per provider, rather than a Python package on its own. That allows to use "from integration.PROVIDER import" and "from system.PROVIDER" rather than importing them from the root of the whole airflow project. The (nearly) is because we need to handle multiple "system", "system.apache" and other import locations. * Removed references to "providers/" generic package which were scheduled for removal after all providers are moved to the new structure * Few remaining references / links referring to old "providers/src" and "providers/test" have been fixed. * The "conftest.py" files in all providers are trimmed down - the code to store ignored deprecation warnings have been moved to the test_common pytest_plugin. That allows to remove 90+ duplicated snippets of deprecation_warnings retrieval while keeping the warnings per-provider in the provider's distribution. * The "moving_providers" scripts are removed. They've done their job and are not needed any more - we keep them in history * The __init__.py files are automatically checked and properly updated in provider folders - in order to properly handle path extension mechanisms * The www tests that were using FAB permisssion model are moved to the FAB provider tests.
This is a set of cleanup steps (first stage) that allow us to remove
the "intermediate" provider's distribution from Airlfow code and replace
it fully with individual provider's distributions - already with own
pyproject.toml
files and basically being (when we complete) acompletely separate distributions from Airflow and without implicit
dependencies between unrelated distributions.
There are a number of other changes needed but that one is only focusing
on removing all references to the "umbrella"
providers
distributionand consequences of removing it.
Those are the changes implemented in this PR:
There are no separate "providers" system tests - each provider has
own system tests and there are no common "generic" providers empty
system test
Integration tests are moved to respective providers under the
integration
package insidetests
directory(nearly) empty init.py files are added in
tests
directoriesof providers - this way "tests" becomes just a directory and root
for all tests per provider, rather than a Python package on its own.
That allows to use "from integration.PROVIDER import" and
"from system.PROVIDER" rather than importing them from the root of
the whole airflow project. The (nearly) is because we need to
handle multiple "system", "system.apache" and other import locations.
Removed references to "providers/" generic package which were
scheduled for removal after all providers are moved to the new
structure
Few remaining references / links referring to old "providers/src" and
"providers/test" have been fixed.
The "conftest.py" files in all providers are trimmed down - the code
to store ignored deprecation warnings have been moved to the
test_common pytest_plugin. That allows to remove 90+ duplicated
snippets of deprecation_warnings retrieval while keeping the warnings
per-provider in the provider's distribution.
The "moving_providers" scripts are removed. They've done their job and
are not needed any more - we keep them in history
The www tests that were using FAB permisssion model are moved to the
FAB provider tests.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.