Move new provider tests to "provider_tests" submodule #45955
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
While testing "standard" provider move to the new structure it turned out, that "providers" package used now to keep tests in the new providers is problematic - while it solves the ambiguity of "from celery import", it introduces another ambiguity when trying to import internal utility classes in tests. For example when trying to import BasePythonTest in standard operatori currently we refer to the import from root directory of Airflow:: ``` from providers.tests.standard.operators.test_python ``` However, if we change the package structure to have `providers` as subpackage of `tests` and trying to import it from `tests` as root folder to tests, we have: ``` from providers.standard.operators.test_python ``` And it can ambiguously attempt to import ``` from airflow.providers.standard.operators.test_python ``` This confuses IDEs and attempts to run multiple tests together, because pytest modifies PYTHONPATH while running tests - and PYTHONPATH modification can be different, depending on which tests are selected to run. Changing the sub-package to provider_tests makes it unambiguous and we can now import: ``` from provider_tests.standard.operators.test_python ``` This is unambiguous and allows to import the tests inside the provider package, without importing from root of Airlfow package - making provider package tests "standalone" inside provider.
boring-cyborg
bot
added
provider:airbyte
provider:apache-iceberg
provider:celery
provider:edge
potiuk
requested review from
gopidesupavan,
eladkal,
hussein-awala,
kaxil,
romsharon98 and
shahar1
eladkal
approved these changes
Jan 23, 2025
dauinh
pushed a commit
to dauinh/airflow
that referenced
this pull request
Jan 24, 2025
While testing "standard" provider move to the new structure it turned out, that "providers" package used now to keep tests in the new providers is problematic - while it solves the ambiguity of "from celery import", it introduces another ambiguity when trying to import internal utility classes in tests. For example when trying to import BasePythonTest in standard operatori currently we refer to the import from root directory of Airflow:: ``` from providers.tests.standard.operators.test_python ``` However, if we change the package structure to have `providers` as subpackage of `tests` and trying to import it from `tests` as root folder to tests, we have: ``` from providers.standard.operators.test_python ``` And it can ambiguously attempt to import ``` from airflow.providers.standard.operators.test_python ``` This confuses IDEs and attempts to run multiple tests together, because pytest modifies PYTHONPATH while running tests - and PYTHONPATH modification can be different, depending on which tests are selected to run. Changing the sub-package to provider_tests makes it unambiguous and we can now import: ``` from provider_tests.standard.operators.test_python ``` This is unambiguous and allows to import the tests inside the provider package, without importing from root of Airlfow package - making provider package tests "standalone" inside provider.
utkarsharma2
added
the
changelog:skip
gpathak128
pushed a commit
to gpathak128/airflow
that referenced
this pull request
Jan 29, 2025
While testing "standard" provider move to the new structure it turned out, that "providers" package used now to keep tests in the new providers is problematic - while it solves the ambiguity of "from celery import", it introduces another ambiguity when trying to import internal utility classes in tests. For example when trying to import BasePythonTest in standard operatori currently we refer to the import from root directory of Airflow:: ``` from providers.tests.standard.operators.test_python ``` However, if we change the package structure to have `providers` as subpackage of `tests` and trying to import it from `tests` as root folder to tests, we have: ``` from providers.standard.operators.test_python ``` And it can ambiguously attempt to import ``` from airflow.providers.standard.operators.test_python ``` This confuses IDEs and attempts to run multiple tests together, because pytest modifies PYTHONPATH while running tests - and PYTHONPATH modification can be different, depending on which tests are selected to run. Changing the sub-package to provider_tests makes it unambiguous and we can now import: ``` from provider_tests.standard.operators.test_python ``` This is unambiguous and allows to import the tests inside the provider package, without importing from root of Airlfow package - making provider package tests "standalone" inside provider.
got686-yandex
pushed a commit
to got686-yandex/airflow
that referenced
this pull request
Jan 30, 2025
While testing "standard" provider move to the new structure it turned out, that "providers" package used now to keep tests in the new providers is problematic - while it solves the ambiguity of "from celery import", it introduces another ambiguity when trying to import internal utility classes in tests. For example when trying to import BasePythonTest in standard operatori currently we refer to the import from root directory of Airflow:: ``` from providers.tests.standard.operators.test_python ``` However, if we change the package structure to have `providers` as subpackage of `tests` and trying to import it from `tests` as root folder to tests, we have: ``` from providers.standard.operators.test_python ``` And it can ambiguously attempt to import ``` from airflow.providers.standard.operators.test_python ``` This confuses IDEs and attempts to run multiple tests together, because pytest modifies PYTHONPATH while running tests - and PYTHONPATH modification can be different, depending on which tests are selected to run. Changing the sub-package to provider_tests makes it unambiguous and we can now import: ``` from provider_tests.standard.operators.test_python ``` This is unambiguous and allows to import the tests inside the provider package, without importing from root of Airlfow package - making provider package tests "standalone" inside provider.
niklasr22
pushed a commit
to niklasr22/airflow
that referenced
this pull request
Feb 8, 2025
While testing "standard" provider move to the new structure it turned out, that "providers" package used now to keep tests in the new providers is problematic - while it solves the ambiguity of "from celery import", it introduces another ambiguity when trying to import internal utility classes in tests. For example when trying to import BasePythonTest in standard operatori currently we refer to the import from root directory of Airflow:: ``` from providers.tests.standard.operators.test_python ``` However, if we change the package structure to have `providers` as subpackage of `tests` and trying to import it from `tests` as root folder to tests, we have: ``` from providers.standard.operators.test_python ``` And it can ambiguously attempt to import ``` from airflow.providers.standard.operators.test_python ``` This confuses IDEs and attempts to run multiple tests together, because pytest modifies PYTHONPATH while running tests - and PYTHONPATH modification can be different, depending on which tests are selected to run. Changing the sub-package to provider_tests makes it unambiguous and we can now import: ``` from provider_tests.standard.operators.test_python ``` This is unambiguous and allows to import the tests inside the provider package, without importing from root of Airlfow package - making provider package tests "standalone" inside provider.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While testing "standard" provider move to the new structure it turned out, that "providers" package used now to keep tests in the new providers is problematic - while it solves the ambiguity of "from celery import", it introduces another ambiguity when trying to import internal utility classes in tests. For example when trying to import BasePythonTest in standard operatori currently we refer to the import from root directory of Airflow::
However, if we change the package structure to have
providersas subpackage oftestsand trying to import it fromtestsas root folder to tests, we have:And it can ambiguously attempt to import
This confuses IDEs and attempts to run multiple tests together, because pytest modifies PYTHONPATH while running tests - and PYTHONPATH modification can be different, depending on which tests are selected to run.
Changing the sub-package to provider_tests makes it unambiguous and we can now import:
This is unambiguous and allows to import the tests inside the provider package, without importing from root of Airlfow package - making provider package tests "standalone" inside provider.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.