Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry K8s tests on failures #46064

Merged
merged 6 commits into from
Jan 26, 2025
Merged

Retry K8s tests on failures #46064

merged 6 commits into from
Jan 26, 2025

Conversation

shubhamraj-git
Copy link
Contributor

We face a lot of K8s failures on CI, Which usually gets pass after re-run.
This PR allows it to re-run for specific number of times, default being 2 without manual intervention.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Copy link
Member

potiuk commented Jan 26, 2025

Looks good.

@shubhamraj-git shubhamraj-git marked this pull request as ready for review January 26, 2025 13:23
@shubhamraj-git shubhamraj-git changed the title [Draft] Retry K8s tests on failures Retry K8s tests on failures Jan 26, 2025
@potiuk potiuk merged commit cffb3c8 into apache:main Jan 26, 2025
86 checks passed
potiuk added a commit to potiuk/airflow that referenced this pull request Jan 28, 2025
@@ -447,7 +447,7 @@ def check_async_run_results(
from airflow_breeze.utils.docker_command_utils import fix_ownership_using_docker

fix_ownership_using_docker()
sys.exit(1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shubhamraj-git -> yeah here, the problem is that that we usually exited here.

@@ -458,6 +458,8 @@ def check_async_run_results(
for output in outputs:
Path(output.file_name).unlink(missing_ok=True)

return tests_failed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The right solution here will be:

if tests_failed:
   sys.exit(1)

The problem is that we do not check the result of check_async_run_results - the method was supposed to "sys.exit(1)` here rather than return False.

My bad I have not noticed it :(

potiuk added a commit that referenced this pull request Jan 28, 2025
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Jan 30, 2025
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants