[turborepo] `--continue': Tasks are executed even when dependencies have failed

[robaca]

What version of Turborepo are you using?

1.10.6

What package manager are you using / does the bug impact?

Yarn v2/v3 (node_modules linker only)

What operating system are you using?

Linux

Describe the Bug

We are currently running our CI pipeline with the --continue flag.

Since some days, we experience broken pipelines: We have a deploy:dev task that performs a deployment, and test:interface for interface tests of the same project that depends on it. When deploy:dev fails, the test:interface still executes afterwards.

Since we experience this issue, we tried to downgrade to a version that worked before (1.9.3), but here we now encounter the same issue. Could be that it's related to bug in a transient dependency of turborepo.

The task failures are reflected in the summary JSON, where we can see that a task like @myproject/some-service#deploy:dev exits with non-zero and is in the dependencies list of @myproject/some-service#test:interface. That latter task also doesn't start before the deployment task has finished.

Expected Behavior

We expect, that in case of failing tasks, only tasks that do not depend on these tasks are executed.

To Reproduce

Not sure if this is related:

Depending on the module, we have different dependencies and so multiple task configurations of deploy:dev in turbo.json.

        ...
        "deploy:dev": {
            "dependsOn": [
                "dockerize",
                "publish-cdn-assets",
                "@myproject/mock-server#deploy:dev"
            ],
            ...
        },
        "@myproject/mock-server#deploy:dev": {
            "dependsOn": [
                "dockerize",
                "publish-cdn-assets"
            ],
            ...
        },
        "@myproject/some-service#deploy:dev": {
            "dependsOn": [
                "dockerize",
                "publish-cdn-assets",
                "@myproject/mock-server#deploy:dev",
                "@myproject/other-service#deploy:dev"
            ],
            ...
        },
        "test:interface": {
            "dependsOn": [
                "deploy:dev",
                "@myproject/mock-server#deploy:dev"
            ],
            ...
        }

Reproduction Repo

No response

[NicholasLYang]

Hi, thanks for the issue. If I'm understanding this correctly, the --continue flag is working as intended, i.e. the entire turbo execution is continued despite some tasks failing. Is there a version of turbo where the --continue flag works as you detailed?

We can definitely investigate a mode where execution continues only for tasks that do not depend on failed tasks, but we're currently right in the middle of moving to Rust, so until that's done, we probably will not be able to add this as a feature.

[NicholasLYang]

Quick update, it appears that we did previously have this behavior, but it was not intended, so we changed it to the current behavior. We'll investigate adding similar functionality post-port.

[ruettenm]

Thanks for the feedback @NicholasLYang 🙂

I was already very wondering because I was pretty sure that at some point in the past the behavior was like Carsten explained it 😅

We build parts of our pipeline based on this „unofficial“ behavior 😅 In the last days we did a bigger infrastructure migration and because of this our deployments are not so stable like before and we realized that the behavior changed.

We will have a look if we find some workaround until you implement the feature again.

Best regards
Matthias

[robaca]

Hi @NicholasLYang, we are using continue in our pipelines to get as many ("real", not followups) errors as possible and remote caching all other results.

I wonder what would be the value of a task execution where a prerequisite (task dependency) failed.

[robaca]

Can you tell us the last version that behaved the old way? Then we will downgrade. We also think about contributing with a pull request by ourselves.

[mehulkar]

Hey @robaca , we talked about this at our team meeting today and our current guidance is to not use the --continue flag for your case. The original intention of --continue was to ignore all errors, but our implementation of it was to stop execution for a task if a dependency did not succeed. This was a consequence of the DAG library we used. After the Rust port, we are back to the intended behavior of --continue.

However, it does seem like useful behavior to stop execution when a dependency fails while allowing other independent parts of the DAG to continue, so we want to spend some time thinking about the API. One idea is adding a value to the flag. Something like:

--continue=all
--continue=independent-tasks-only

[robaca]

Can we expect some progress here in near future?

[mehulkar]

@robaca the team doesn't have this on our immediate roadmap, but we would accept a proposal or a PR that implements the approach above!

[robaca]

While personally not feeling able to provide a PR, I still think that this is of big value, also for others. In our pipeline we finally run acceptance tests for multiple services in parallel. If one of them fails due to flakiness or local bug, we still want the others to finish and contribute to the remote caching. Also the feedback helps to locate errors by looking at affected dependent tasks

[jenseng]

Implemented in #10023 🤞 🚀