Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-72: Implement short circuit and branch operators #46584

Merged
merged 2 commits into from
Mar 14, 2025

Conversation

shahar1
Copy link
Contributor

@shahar1 shahar1 commented Feb 8, 2025

closes: #45823
related: #44912

Update: issue #43883 (Short Circuit for mapped operators) will be closed in a separate PR.

TODOS:

  • Implement ShortCircuitOperator (including decorators)
  • Implement BaseBranchOperator, BranchPythonOperator
  • Fix core unit tests
  • Fix tests of of BranchSQLOperator
  • Fix tests of standard branch operators (within the standard provider)
  • Add new core unit tests
  • Relocate relevant modules to the standard provider

Verified by running some of the example DAGs:

image image image image image image

Known Issues:


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@shahar1 shahar1 force-pushed the implement-short-circuit-operator branch from caf0c04 to 34a0644 Compare February 8, 2025 19:51
@ashb ashb linked an issue Feb 10, 2025 that may be closed by this pull request
@shahar1 shahar1 force-pushed the implement-short-circuit-operator branch 2 times, most recently from 00492b1 to 6edd2ef Compare February 11, 2025 19:22
@shahar1 shahar1 force-pushed the implement-short-circuit-operator branch 4 times, most recently from 239670d to b527298 Compare February 11, 2025 20:59
@shahar1 shahar1 force-pushed the implement-short-circuit-operator branch 3 times, most recently from 7fc5e04 to fc4f8a4 Compare February 22, 2025 07:02
@shahar1 shahar1 changed the title [WIP] Implement short circuit operator [WIP] Implement short circuit and branch operator Feb 22, 2025
@kaxil kaxil force-pushed the implement-short-circuit-operator branch 3 times, most recently from b3bf065 to 8930f86 Compare March 13, 2025 22:52
@kaxil
Copy link
Member

kaxil commented Mar 14, 2025

Known Issue:

Branching in Mapped Task Groups does not work. So the issue mentioned in #30883 is back. We will handle that in a separate PR.

@kaxil kaxil force-pushed the implement-short-circuit-operator branch 2 times, most recently from dc487b6 to 5247781 Compare March 14, 2025 14:27
@kaxil
Copy link
Member

kaxil commented Mar 14, 2025

Added one more known issue to the PR description and added some screenshots

@kaxil kaxil force-pushed the implement-short-circuit-operator branch 4 times, most recently from e74f4c9 to 67dac0b Compare March 14, 2025 22:17
@kaxil
Copy link
Member

kaxil commented Mar 14, 2025

We should be finally good to go on this!!

Waiting for the CI 🤞

@kaxil kaxil force-pushed the implement-short-circuit-operator branch from 67dac0b to f6f5805 Compare March 14, 2025 22:50
@kaxil kaxil merged commit a6f6733 into apache:main Mar 14, 2025
12 checks passed
@kaxil
Copy link
Member

kaxil commented Mar 14, 2025

CI passed: https://github.com/apache/airflow/actions/runs/13866617491/job/38807327529?pr=46584

The static checks with providers was fixed in the last commit

@kaxil
Copy link
Member

kaxil commented Mar 14, 2025

@shahar1 and I will fix any teething issues if something fails -- but we wanted to get it in since it is already too big and has been out for a while.

kaxil added a commit to astronomer/airflow that referenced this pull request Mar 16, 2025
These comments have been resolved in the original PR apache#46584 .
@kaxil kaxil mentioned this pull request Mar 16, 2025
hussein-awala pushed a commit that referenced this pull request Mar 16, 2025
These comments have been resolved in the original PR #46584 .
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Mar 21, 2025
closes: apache#45823
related: apache#44912

Co-Authored-By: Kaxil Naik <[email protected]>
---------

Co-authored-by: Kaxil Naik <[email protected]>
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Mar 21, 2025
These comments have been resolved in the original PR apache#46584 .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Get Operators subclassing SkipMixin working with Task SDK
3 participants