Move standad provider to the new structure

This one also adds some missing steps in provider migration script.
improves the output and instructions and make it more robust for
various variations of provider features and usage of imports.

With those changes `standard` provider was 100% moved automatically.

.github/boring-cyborg.yml

@@ -22,10 +22,7 @@ labelPRBasedOnFilePath:
     - providers/airbyte/**
 
   provider:alibaba:
-    - providers/src/airflow/providers/alibaba/**/*
-    - docs/apache-airflow-providers-alibaba/**/*
-    - providers/tests/alibaba/**/*
-    - providers/tests/system/alibaba/**/*
+    - providers/alibaba/**
 
   provider:amazon:
     - providers/src/airflow/providers/amazon/**/*
@@ -179,9 +176,7 @@ labelPRBasedOnFilePath:
     - providers/tests/system/common/sql/**/*
 
   provider:standard:
-    - providers/src/airflow/providers/standard/**/*
-    - docs/apache-airflow-providers-standard/**/*
-    - providers/tests/standard/**/*
+    - providers/standard/**
 
   provider:databricks:
     - providers/src/airflow/providers/databricks/**/*
@@ -647,12 +642,12 @@ labelPRBasedOnFilePath:
     - airflow/cli/commands/local_commands/triggerer_command.py
     - airflow/jobs/triggerer_job_runner.py
     - airflow/models/trigger.py
-    - providers/src/airflow/providers/standard/triggers/**/*
+    - providers/standard/src/airflow/providers/standard/triggers/**/*
     - tests/cli/commands/local_commands/test_triggerer_command.py
     - tests/jobs/test_triggerer_job.py
     - tests/models/test_trigger.py
     - tests/jobs/test_triggerer_job_logging.py
-    - providers/tests/standard/triggers/**/*
+    - providers/standard/tests/provider_tests/standard/triggers/**/*
 
   area:Serialization:
     - airflow/serialization/**/*

.pre-commit-config.yaml

@@ -724,6 +724,8 @@ repos:
           ^airflow/operators/.*$|
           ^providers/src/airflow/providers/.*$|
           ^providers/src/airflow/providers/standard/sensors/.*$|
+          ^providers/.*/src/airflow/providers/.*$|
+          ^providers/.*/src/airflow/providers/standard/sensors/.*$|
           ^dev/provider_packages/.*$
       - id: check-base-operator-usage
         language: pygrep
@@ -733,8 +735,9 @@ repos:
         pass_filenames: true
         files: >
           (?x)
-          ^providers/src/airflow/providers/.*\.py$
-        exclude: providers/src/airflow/providers/standard/operators/bash.py|providers/src/airflow/providers/standard/operators/python.py|providers/src/airflow/providers/standard/sensors/external_task.py
+          ^providers/src/airflow/providers/.*\.py$|
+          ^providers/.*/src/airflow/providers/.*\.py$
+        exclude: providers/standard/.*/.*\.py$
       - id: check-get-lineage-collector-providers
         language: python
         name: Check providers import hook lineage code from compat
@@ -1206,9 +1209,11 @@ repos:
           ^airflow/utils/helpers.py$ |
           ^airflow/utils/log/secrets_masker.py$ |
           ^providers/src/airflow/providers/ |
-          ^(providers/)?tests/ |
-          task_sdk/src/airflow/sdk/definitions/dag.py$ |
-          task_sdk/src/airflow/sdk/definitions/_internal/node.py$ |
+          ^tests/ |
+          ^providers/tests/ |
+          ^providers/.*/tests/ |
+          ^task_sdk/src/airflow/sdk/definitions/dag.py$ |
+          ^task_sdk/src/airflow/sdk/definitions/_internal/node.py$ |
           ^dev/.*\.py$ |
           ^scripts/.*\.py$ |
           ^docker_tests/.*$ |

dev/breeze/src/airflow_breeze/templates/pyproject_TEMPLATE.toml.jinja2

@@ -63,9 +63,9 @@ classifiers = [
     "Framework :: Apache Airflow",
     "Framework :: Apache Airflow :: Provider",
     "License :: OSI Approved :: Apache Software License",
-    {%- for python_version in SUPPORTED_PYTHON_VERSIONS %}
+    {% for python_version in SUPPORTED_PYTHON_VERSIONS %}
     "Programming Language :: Python :: {{ python_version }}",
-    {%- endfor %}
+    {% endfor %}
     "Topic :: System :: Monitoring",
 ]
 requires-python = "~=3.9"

dev/breeze/tests/test_selective_checks.py

@@ -253,7 +253,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
         ),
         (
             pytest.param(
-                ("providers/src/airflow/providers/standard/operators/python.py",),
+                ("providers/standard/src/airflow/providers/standard/operators/python.py",),
                 {
                     "selected-providers-list-as-string": None,
                     "all-python-versions": "['3.9']",
@@ -823,7 +823,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
             id="Only Always and common providers tests should run when only common.io and tests/always changed",
         ),
         pytest.param(
-            ("providers/src/airflow/providers/standard/operators/bash.py",),
+            ("providers/standard/src/airflow/providers/standard/operators/bash.py",),
             {
                 "selected-providers-list-as-string": "common.compat standard",
                 "all-python-versions": "['3.9']",
@@ -848,7 +848,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
             id="Providers standard tests and Serialization tests to run when airflow bash.py changed",
         ),
         pytest.param(
-            ("providers/src/airflow/providers/standard/operators/bash.py",),
+            ("providers/standard/src/airflow/providers/standard/operators/bash.py",),
             {
                 "selected-providers-list-as-string": None,
                 "all-python-versions": "['3.9']",

dev/moving_providers/README.md

@@ -24,6 +24,8 @@
 - [Moving providers to new structure](#moving-providers-to-new-structure)
   - [How to use the script](#how-to-use-the-script)
   - [Options](#options)
+- [What happens under the hood](#what-happens-under-the-hood)
+- [What to do next](#what-to-do-next)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
@@ -44,12 +46,21 @@ some manual adjustments needed in more complex cases.
 
 The script follows https://peps.python.org/pep-0723/ and uses inlined dependencies - so it can be run as-is
 by modern tools without creating dedicated virtualenv - the virtualenv with dependencies is
-created on-the-fly by PEP 723 compatible tools. For example one can use uv to run it:
+created on-the-fly by PEP 723 compatible tools.
+
+For example this one will make a dry-run of moving Alibaba provider:
 
 ```shell
 uv run dev/moving_providers/move_providers.py alibaba
 ```
 
+And this one will perform update and move the Alibaba provider:
+
+
+```shell
+uv run dev/moving_providers/move_providers.py alibaba --perform-update
+```
+
 ## Options
 
 
@@ -60,9 +71,8 @@ uv run dev/moving_providers/move_providers.py alibaba
 > uv run dev/moving_providers/move_providers.py --help
 > ```
 
-By default the script runs in `--dry-run` mode, which means it will not make any changes to the file system,
-but will print what it would do. To actually move the files, you need to pass `--no-dry-run` option and you
-will be asked to commit the code and create a PR:
+By default the script runs in dry run mode, which means it will not make any changes to the file system,
+but will print what it would do. To actually move the files, you need to pass `--perform-update` flag.
 
 ```shell
 uv run dev/moving_providers/move_providers.py alibaba --no-dry-run
@@ -85,3 +95,26 @@ You can also specify `--quiet` option to see less output:
 ```shell
 uv run dev/moving_providers/move_providers.py alibaba --quiet
 ```
+
+# What happens under the hood
+
+When you run the script with `--perform-update` flag, you will see the diff of the changes
+that the script made, and you will be able to scroll through it (with your configured editor)
+to verify that it looks good.
+
+The script will:
+
+* move the provider to the new structure and apply fixes
+* build CI image to add the new provider in the image packages
+* run static checks to verify that the moved provider code is good and apply auto-fixes in some cases
+
+# What to do next
+
+After all that you need to fix all potential static check problems, run all the tests for the provider and
+fix any issues that might happen:
+
+1) Fix all the static check errors, add them to git
+2) run `breeze testing providers-tests --test-type 'Providers[LIST_OF_PROVIDER_IDS_MOVED]'` and fix all tests.
+3) Add changes to git, create branch, commit the changes and create a PR!
+
+Good luck!