Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename execution_date to logical_date across codebase #43902

Merged
merged 15 commits into from
Nov 15, 2024

Conversation

sunank200
Copy link
Collaborator

@sunank200 sunank200 commented Nov 11, 2024

Motivation

This PR renames execution_date to logical_date across the codebase. The shift towards logical_date helps move away from the limitations of execution_date, particularly with dynamic DAG runs and cases where multiple runs occur at the same time.

Key Changes

  • Replaced all instances of execution_date with logical_date including models.
  • Updated references in database models, templates, and functions.
  • Removed unique constraints on execution_date in the database to allow multiple DAG runs with the same logical time.

How execution_date and logical_date Work

  1. Logical date is equivalent to execution date: The two are just different names for the same value.
  2. Timetable controls logical date: The logical date can be set to any value, not necessarily tied to the data interval's start or end.
  3. Schedules dictate behavior: For value-based schedules (like cron), the logical date is set by the timetable class used.

Transitioning from execution_date

  • Airflow 3.0 will fully remove execution_date.
  • For uniquely identifying runs:
    • Use run_id for human-readable identifiers.
    • Use data_interval_start/data_interval_end for data partitioning in future.
    • id (auto-incremented) is recommended for ordering DAG runs.
  • If your existing logic relies on execution_date, switch to using data_interval_start or data_interval_end for identifying time ranges.

Testing

  • Updated unit tests to reflect the changes.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes provider related issues labels Nov 11, 2024
@sunank200 sunank200 added legacy api Whether legacy API changes should be allowed in PR legacy ui Whether legacy UI change should be allowed in PR labels Nov 11, 2024
@sunank200 sunank200 force-pushed the rename-execution-date-to-logical-date branch 9 times, most recently from ea36b6c to 65f9b8e Compare November 12, 2024 02:54
@sunank200 sunank200 requested review from Lee-W and phanikumv November 12, 2024 02:54
@uranusjr
Copy link
Member

This needs a news fragment. Rewording the above summary would work fine. See existing files in newsfragments (the significant ones) for some examples.

@uranusjr
Copy link
Member

uranusjr commented Nov 12, 2024

For templates, replace {{ ds }} with {{ data_interval_start | ds }}.

This should probably use logical_date for max compatibility. Also should mention ts (same rewrite).

Oh wait, we’re not removing those in this PR yet, so this should not be mentioned at all.

@sunank200 sunank200 added the AIP-83 Remove Execution Date Unique Constraint from DAG Run label Nov 25, 2024
LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this pull request Jan 5, 2025
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit to astronomer/airflow that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
kaxil added a commit that referenced this pull request Jan 12, 2025
We removed all the deprecated keys in #43902 so we no longer need this code.

In preparation of #45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Jan 13, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jan 13, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
We removed all the deprecated keys in apache#43902 so we no longer need this code.

In preparation of apache#45583, I want to simplify this code. We can always revert/re-add this later when we need to deprecate a key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AIP-83 Remove Execution Date Unique Constraint from DAG Run airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler legacy api Whether legacy API changes should be allowed in PR legacy ui Whether legacy UI change should be allowed in PR provider:cncf-kubernetes Kubernetes provider related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants