Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed the unused provider's package #46608

Merged

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Feb 10, 2025

This is a set of cleanup steps (first stage) that allow us to remove
the "intermediate" provider's distribution from Airlfow code and replace
it fully with individual provider's distributions - already with own
pyproject.toml files and basically being (when we complete) a
completely separate distributions from Airflow and without implicit
dependencies between unrelated distributions.

There are a number of other changes needed but that one is only focusing
on removing all references to the "umbrella" providers distribution
and consequences of removing it.

Those are the changes implemented in this PR:

  • There are no separate "providers" system tests - each provider has
    own system tests and there are no common "generic" providers empty
    system test

  • Integration tests are moved to respective providers under the
    integration package inside tests directory

  • (nearly) empty init.py files are added in tests directories
    of providers - this way "tests" becomes just a directory and root
    for all tests per provider, rather than a Python package on its own.
    That allows to use "from integration.PROVIDER import" and
    "from system.PROVIDER" rather than importing them from the root of
    the whole airflow project. The (nearly) is because we need to
    handle multiple "system", "system.apache" and other import locations.

  • Removed references to "providers/" generic package which were
    scheduled for removal after all providers are moved to the new
    structure

  • Few remaining references / links referring to old "providers/src" and
    "providers/test" have been fixed.

  • The "conftest.py" files in all providers are trimmed down - the code
    to store ignored deprecation warnings have been moved to the
    test_common pytest_plugin. That allows to remove 90+ duplicated
    snippets of deprecation_warnings retrieval while keeping the warnings
    per-provider in the provider's distribution.

  • The "moving_providers" scripts are removed. They've done their job and
    are not needed any more - we keep them in history

  • The www tests that were using FAB permisssion model are moved to the
    FAB provider tests.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk force-pushed the remove-unused-providers-package-sources-and-tests branch 4 times, most recently from ce8d8c2 to c0fda0b Compare February 10, 2025 08:17
Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existence of this directory for the amazon provider was one of the sneaky ways in which tests were breaking when I was migrating that provider. Amazon is used as an example in a lot of tests and some code was detecting it as not migrated because of that.

Any who, I made a note to come back this morning and delete the others but you've already got it, nice!

@potiuk potiuk force-pushed the remove-unused-providers-package-sources-and-tests branch from c0fda0b to 0085837 Compare February 11, 2025 22:57
@potiuk potiuk force-pushed the remove-unused-providers-package-sources-and-tests branch 2 times, most recently from ac0c9b5 to 6fcd868 Compare February 12, 2025 00:04
@potiuk potiuk force-pushed the remove-unused-providers-package-sources-and-tests branch 2 times, most recently from e6d99a0 to 3e16bf0 Compare February 14, 2025 01:03
@potiuk potiuk marked this pull request as ready for review February 14, 2025 01:03
@potiuk potiuk force-pushed the remove-unused-providers-package-sources-and-tests branch from 3e16bf0 to 4ae9b39 Compare February 14, 2025 01:49
@potiuk
Copy link
Member Author

potiuk commented Feb 14, 2025

There are still few errors due to cross-provider dependencies with fab provider / azure paths not fixed - but generally it should be ready for review (I will fix the errors tomorrow).

@potiuk
Copy link
Member Author

potiuk commented Feb 15, 2025

For those who will be reviewing this one - I recommend git diff -> UI of Github is very, very slow with that huge of a change.

Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few sanity check comments, compared to Niko I did not find any glitches. Good to be merged after nit's applied.

@potiuk
Copy link
Member Author

potiuk commented Feb 15, 2025

Just a few sanity check comments, compared to Niko I did not find any glitches. Good to be merged after nit's applied.

👀 👀 👀 👀 👀 👀 👀 👀

This is a set of cleanup steps (first stage) that allow us to remove
the "intermediate" provider's distribution from Airlfow code and replace
it fully with individual provider's distributions - already with own
`pyproject.toml` files and basically being (when we complete) a
completely separate distributions from Airflow and without implicit
dependencies between unrelated distributions.

There are a number of other changes needed but that one is only focusing
on removing all references to the "umbrella" `providers` distribution
and consequences of removing it.

Those are the changes implemented in this PR:

* There are no separate "providers" system tests - each provider has
  own system tests and there are no common "generic" providers empty
  system test

* Integration tests are moved to respective providers under the
  `integration` package inside `tests` directory

* (nearly) empty __init__.py files are added in `tests` directories
  of providers - this way "tests" becomes just a directory and root
  for all tests per provider, rather than a Python package on its own.
  That allows to use "from integration.PROVIDER import" and
  "from system.PROVIDER" rather than importing them from the root of
  the whole airflow project. The (nearly) is because we need to
  handle multiple "system", "system.apache" and other import locations.

* Removed references to "providers/" generic package which were
  scheduled for removal after all providers are moved to the new
  structure

* Few remaining references / links referring to old "providers/src" and
  "providers/test" have been fixed.

* The "conftest.py" files in all providers are trimmed down - the code
  to store ignored deprecation warnings have been moved to the
  test_common pytest_plugin. That allows to remove 90+ duplicated
  snippets of deprecation_warnings retrieval while keeping the warnings
  per-provider in the provider's distribution.

* The "moving_providers" scripts are removed. They've done their job and
  are not needed any more - we keep them in history

* The __init__.py files are automatically checked and properly updated
  in provider folders - in order to properly handle path extension
  mechanisms

* The www tests that were using FAB permisssion model are moved to the
  FAB provider tests.
@potiuk potiuk force-pushed the remove-unused-providers-package-sources-and-tests branch from df4f601 to 4b8e4a2 Compare February 15, 2025 15:37
@potiuk
Copy link
Member Author

potiuk commented Feb 15, 2025

Rebased and applied the nits.

@potiuk
Copy link
Member Author

potiuk commented Feb 15, 2025

BTW.I plan to have a separate discussion on the "final" layout of the provider test code so I plan to merge it rather quickly and then write up a discussion thread where I would want people to chime in and make some final decisions. I am afraid if I won't merge it quickly some test conflicts will take more than they are worth. So @o-nikolas @Lee-W -> I think, if the tests pass after rebase, I will merge it, but it is really up to a follow up discussion how the folder and structure and imports will look like eventually (this is all test code so we can remove it again it will be far simpler to do follow-up renames.

@potiuk potiuk merged commit e027457 into apache:main Feb 15, 2025
90 checks passed
@potiuk potiuk deleted the remove-unused-providers-package-sources-and-tests branch February 15, 2025 16:44
@potiuk
Copy link
Member Author

potiuk commented Feb 15, 2025

Merged !

dantonbertuol pushed a commit to dantonbertuol/airflow that referenced this pull request Feb 17, 2025
This is a set of cleanup steps (first stage) that allow us to remove
the "intermediate" provider's distribution from Airlfow code and replace
it fully with individual provider's distributions - already with own
`pyproject.toml` files and basically being (when we complete) a
completely separate distributions from Airflow and without implicit
dependencies between unrelated distributions.

There are a number of other changes needed but that one is only focusing
on removing all references to the "umbrella" `providers` distribution
and consequences of removing it.

Those are the changes implemented in this PR:

* There are no separate "providers" system tests - each provider has
  own system tests and there are no common "generic" providers empty
  system test

* Integration tests are moved to respective providers under the
  `integration` package inside `tests` directory

* (nearly) empty __init__.py files are added in `tests` directories
  of providers - this way "tests" becomes just a directory and root
  for all tests per provider, rather than a Python package on its own.
  That allows to use "from integration.PROVIDER import" and
  "from system.PROVIDER" rather than importing them from the root of
  the whole airflow project. The (nearly) is because we need to
  handle multiple "system", "system.apache" and other import locations.

* Removed references to "providers/" generic package which were
  scheduled for removal after all providers are moved to the new
  structure

* Few remaining references / links referring to old "providers/src" and
  "providers/test" have been fixed.

* The "conftest.py" files in all providers are trimmed down - the code
  to store ignored deprecation warnings have been moved to the
  test_common pytest_plugin. That allows to remove 90+ duplicated
  snippets of deprecation_warnings retrieval while keeping the warnings
  per-provider in the provider's distribution.

* The "moving_providers" scripts are removed. They've done their job and
  are not needed any more - we keep them in history

* The __init__.py files are automatically checked and properly updated
  in provider folders - in order to properly handle path extension
  mechanisms

* The www tests that were using FAB permisssion model are moved to the
  FAB provider tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants