More meaningful name for `@task.kubernetes` pods #46462

insomnes · 2025-02-05T12:11:24Z

Use name based on python callable in @task.kubernetes pod name generation

generate specific pod name base by provided python callable
drop uuid usage with no name (random is controlled by random_name_suffix arg, there is no need for extra UUID usage)
add specific pod naming tests for decorator flow because decorator distinguishes between name=None and no name argument provided

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

RNHTTR

What will happen if two Dag runs are triggered at the same time, and two TaskInstances are created from the same task simultaneously?

airflow/decorators/__init__.pyi

insomnes · 2025-02-05T17:18:06Z

What will happen if two Dag runs are triggered at the same time, and two TaskInstances are created from the same task simultaneously?

@RNHTTR
If random_name_suffix arg is set (as default) they would have two different names by separate random suffixes which are generated as part of pod request object during execute calls in KPO by build_pod_request_obj method.

If user has disabled name randomization by random_name_suffix=False -- that's more on user side to ensure name uniqueness.

I am happy to add a test for this case if you can guide me a bit on how to setup such test.

airflow/decorators/__init__.pyi

RNHTTR · 2025-02-06T00:04:26Z

Looks good imo, but tests are failing now

insomnes · 2025-02-06T10:30:02Z

I think: the problem is on the CI side with k8s cluster tests and not in the code, will wait a bit and retry by rebasing the branch. My last attempt failed to set the environment, and previous failures were about failing jobs.

…ration - generate specific pod name base by provided python callable - drop uuid usage by default (random is controlled by `random_name_suffix` arg) - add specific pod naming tests for decorator flow

insomnes · 2025-02-06T17:52:48Z

Oh, now there is conflict after the provider move to a new structure. Will check it a bit later today

potiuk · 2025-02-06T19:29:35Z

Yeah... Sorry :) . It was finaly green, so I merged it. But I think in this case git will properly fix stuff when you rebase - automatically

chore: apply pre-commit

…che#45864) * Allow passing empty labels in the driver config * Fix formatting * Fix formatting --------- Co-authored-by: Maxim Logvinenko <[email protected]>

…che#46422)

…pache#46219) * BREAKING CHANGE: replace Airflow config by conx extras in SMTP provider * fix static checks

…e#46328)

We can't just check if CeleryExecutor is one of the executors when determining if we can expose the automount param, we need to make sure it's the _only_ executor - the others need it.

* Add XCom update API * Add tests for XCom update API * Fix failures on execution

* Add logical in the tests * Fix compact test

apache#46480) Closes apache#46477 The chart version 1.16.0 would be released before airflow v3, so it's better to keep api server in beta so it's easier to make changes. Signed-off-by: Andrii Korotkov <[email protected]>

* Fix task sdk client dry-run mode The `run_after` field was missing in the fake dag run response. * Fix trailing whitespaces

* refactor: Moved microsoft winrm provider * refactor: fixed static checks

* Update console link to filter on multiple instance ids

* Add state filter to task page. * Refactor task instance and dagrun states to constants util.

* Move CNCF Kubernetes to new provider structure * Fix doc include path and k8s test * Fix taskflow tutorial * Fix test_project_structure * Strip src. prefix instead of replacing all src. Co-authored-by: Kalyan R <[email protected]> * Merge fix get_classes_from_file apache#46454 * Fix TestCncfProviderProjectStructure - rename PROVIDER from "cncf" to "cncf/kubernetes" - remove MISSING_EXAMPLES_FOR_CLASSES * Fix k8s CI requirements * fixup! Fix k8s CI requirements --------- Co-authored-by: Kalyan R <[email protected]> Co-authored-by: Jarek Potiuk <[email protected]>

The big change here (other than just moving code around) is to introduce a conceptual separation between Definition/Execution time and Scheduler time. This means that the expansion of tasks (creating the TaskInstance rows with different map_index values) is now done on the scheduler, and we now deserialize to different classes. For example, when we deserialize the `DictOfListsExpandInput` it gets turned into an instance of SchedulerDictOfListsExpandInput. This is primarily designed so that DB access is kept 100% out of the TaskSDK. Some of the changes here are on the "wat" side of the scale, and this is mostly designed to not break 100% of our tests, and we have apache#45549 to look at that more holistically. To support the "reduce" style task which takes as input a sequence of all the pushed (mapped) XCom values, and to keep the previous behaviour of not loading all values in to memory at once, we have added a new HEAD route to the Task Execution interface that returns the number of mapped XCom values so that it is possible to implement `__len__` on the new LazyXComSequence class. I have deleted a tranche of tests from tests/models that were to do with runtime behavoir and and now tested in the TaskSDK instead.

…he#46485) * Fix bug in query invalidation and remove custom predicate logic * Address PR feedback * Only use key fn when we have props to pass

…Databric…" (apache#45066) This reverts commit 09d8a80. Co-authored-by: Amogh Desai <[email protected]>

Part of apache#46045

* Moving yandex provider to new provider structure * fixup! Moving yandex provider to new provider structure --------- Co-authored-by: Jarek Potiuk <[email protected]>

potiuk · 2025-02-06T20:15:08Z

still conflicts to solve :(

insomnes · 2025-02-06T20:21:42Z

Holy moly I am so sorry for this mess, and I am really sorry for the ping for the whole repo I will close this one and apply my changes to the new structure, that would be easier. I am sorry for bothering you all

potiuk · 2025-02-06T20:34:44Z

No worries :)

insomnes requested review from jedcunningham and hussein-awala as code owners February 5, 2025 12:11

boring-cyborg bot added area:providers provider:cncf-kubernetes Kubernetes provider related issues labels Feb 5, 2025

insomnes force-pushed the task-k8s-callable-name branch from d56a591 to 0b63a45 Compare February 5, 2025 12:12

insomnes mentioned this pull request Feb 5, 2025

More meaningful name for @task.kubernetes pods by default #46464

Closed

2 tasks

RNHTTR reviewed Feb 5, 2025

View reviewed changes

airflow/decorators/__init__.pyi Show resolved Hide resolved

insomnes commented Feb 5, 2025

View reviewed changes

airflow/decorators/__init__.pyi Show resolved Hide resolved

RNHTTR approved these changes Feb 5, 2025

View reviewed changes

insomnes force-pushed the task-k8s-callable-name branch from 0fcc0cc to bf8ea71 Compare February 5, 2025 23:00

insomnes force-pushed the task-k8s-callable-name branch from bf8ea71 to 75b03e8 Compare February 6, 2025 08:02

insomnes added 2 commits February 6, 2025 12:11

Use name based on python callable in @task.kubernetes pod name gene…

50ab03d

…ration - generate specific pod name base by provided python callable - drop uuid usage by default (random is controlled by `random_name_suffix` arg) - add specific pod naming tests for decorator flow

Update decorator doc-string

45f68e4

insomnes force-pushed the task-k8s-callable-name branch from 75b03e8 to 45f68e4 Compare February 6, 2025 11:11

0BVer and others added 11 commits February 6, 2025 21:02

docs: clarify Gunicorn's role in webserver worker refresh (apache#46371)

ad74921

chore: apply pre-commit

Allow passing empty labels in the spark kubernetes driver config (apa…

821e43a

…che#45864) * Allow passing empty labels in the driver config * Fix formatting * Fix formatting --------- Co-authored-by: Maxim Logvinenko <[email protected]>

SnowflakeSqlApiOperator snowflake_conn_id add to template_fields (apa…

0f0a105

…che#46422)

Replace Airflow email config by connection extras in SMTP provider (a…

926544e

…pache#46219) * BREAKING CHANGE: replace Airflow config by conx extras in SMTP provider * fix static checks

Integrate the SimpleAuthManager UI in dev mode (apache#46511)

8566e5f

AIP-84 Remove unecessary datamodels config and from_attributes (apach…

0b70737

…e#46328)

Fix scheduler ServiceAccount automount for multi-executor (apache#46486)

7968955

We can't just check if CeleryExecutor is one of the executors when determining if we can expose the automount param, we need to make sure it's the _only_ executor - the others need it.

Add update XCom endpoint in RestAPI (apache#46457)

5449f39

* Add XCom update API * Add tests for XCom update API * Fix failures on execution

Fix missing logical_date in DagRun for DMS operator tests (apache#46521)

78d96b0

* Add logical in the tests * Fix compact test

Update AWS auth manager to use Fastapi instead of Flask (apache#46381)

fff997d

feluelle and others added 11 commits February 6, 2025 21:02

Fix task sdk client dry-run mode (apache#46524)

380e49d

* Fix task sdk client dry-run mode The `run_after` field was missing in the fake dag run response. * Fix trailing whitespaces

Move microsoft winrm to new provider structure (apache#46469)

e27b7ea

* refactor: Moved microsoft winrm provider * refactor: fixed static checks

Adding extra links for EC2 (apache#46340)

01ca57e

* Update console link to filter on multiple instance ids

Add state filter to task page. (apache#46465)

2d36aef

* Add state filter to task page. * Refactor task instance and dagrun states to constants util.

Fix bug in query invalidation and remove custom predicate logic (apac…

682f9a3

…he#46485) * Fix bug in query invalidation and remove custom predicate logic * Address PR feedback * Only use key fn when we have props to pass

Move Apache Flink to new provider structure (apache#46132)

b57714a

Revert "Revert "Added job_clusters as a templated parameter to Create…

cdf1204

…Databric…" (apache#45066) This reverts commit 09d8a80. Co-authored-by: Amogh Desai <[email protected]>

Move apache.impala provider to a new structure (apache#46532)

da34d80

Part of apache#46045

Moving yandex provider to new provider structure (apache#46525)

78ab629

* Moving yandex provider to new provider structure * fixup! Moving yandex provider to new provider structure --------- Co-authored-by: Jarek Potiuk <[email protected]>

insomnes requested review from ephraimbuddy, dstandish, potiuk, ashb, eladkal, o-nikolas, uranusjr, bbovenzi, pierrejeambrun, ryanahamilton, jscheffl, bolkedebruin and XD-DENG as code owners February 6, 2025 20:06

insomnes closed this Feb 6, 2025

insomnes deleted the task-k8s-callable-name branch February 6, 2025 20:24

insomnes mentioned this pull request Feb 6, 2025

More meaningful @task.kubernetes pod naming #46535

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More meaningful name for `@task.kubernetes` pods #46462

More meaningful name for `@task.kubernetes` pods #46462

insomnes commented Feb 5, 2025 •

edited

Loading

RNHTTR left a comment

insomnes commented Feb 5, 2025 •

edited

Loading

RNHTTR commented Feb 6, 2025

insomnes commented Feb 6, 2025 •

edited

Loading

insomnes commented Feb 6, 2025

potiuk commented Feb 6, 2025

potiuk commented Feb 6, 2025

insomnes commented Feb 6, 2025

potiuk commented Feb 6, 2025

More meaningful name for @task.kubernetes pods #46462

More meaningful name for @task.kubernetes pods #46462

Conversation

insomnes commented Feb 5, 2025 • edited Loading

RNHTTR left a comment

Choose a reason for hiding this comment

insomnes commented Feb 5, 2025 • edited Loading

RNHTTR commented Feb 6, 2025

insomnes commented Feb 6, 2025 • edited Loading

insomnes commented Feb 6, 2025

potiuk commented Feb 6, 2025

potiuk commented Feb 6, 2025

insomnes commented Feb 6, 2025

potiuk commented Feb 6, 2025

More meaningful name for `@task.kubernetes` pods #46462

More meaningful name for `@task.kubernetes` pods #46462

insomnes commented Feb 5, 2025 •

edited

Loading

insomnes commented Feb 5, 2025 •

edited

Loading

insomnes commented Feb 6, 2025 •

edited

Loading