[turborepo] turbo-ignore always builds all projects for new branches on Vercel

[kieranm]

What version of Turborepo are you using?

turbo-ignore@1.10.16

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Mac

Describe the Bug

We have a monorepo with multiple websites, and for each Vercel app we have set the ignored build step to npx turbo-ignore. We have the Git integration enabled for preview builds.

When we open a PR, we expect that only changed apps will be built, to reduce unnecessary builds.

However, in Vercel I see the following output:

Running "npx turbo-ignore"
--
22:51:51.448 | npm WARN exec The following package was not found and will be installed: [email protected]
22:51:51.767 | ≫   Using Turborepo to determine if this project is affected by the commit...
22:51:51.768 |  
22:51:51.772 | ≫   Inferred "@daybridge/daybridge.design" as workspace from "package.json"
22:51:51.772 | ≫   Using "build" as the task from the arguments
22:51:51.773 | ≫   No previous deployments found for "@daybridge/daybridge.design" on branch "day-2084-item-form-does-not-appear-for-items-ending-at-midnight-on"
22:51:51.773 | ✓ Proceeding with deployment

The @daybridge/daybridge.design app did not have any changed files, so it should have been ignored. But the message suggests that all web apps are being built because they haven't been deployed before on this branch.

Expected Behavior

I'm not sure if I'm doing something wrong, or if there is some context I'm missing, but it's important that apps that have not been changed don't deploy unnecessarily, so I would expect turbo-ignore to be skipping these even if they haven't been deployed before for a specific branch.

To Reproduce

• Monorepo with one or more web projects
• Each project added as a separate app to Vercel, with preview builds enabled
• Each web project has its ignored build step set to npx turbo-ignore
• Make a change impacting only one of the web projects and none of the others
• Open PR in Github

Reproduction Repo

No response

[gsoltis]

Can you try setting --fallback to handle scenarios when it's the first deploy of a particular branch?

See Usage

[kieranm]

Thanks! I can definitely try setting this, but the guidance on Vercel's docs says its only necessary to use npx turbo-ignore without any additional flags. This used to work just fine. So I would like to understand why this has started happening, and potentially whether its a bug.

[kieranm]

In fact, in Vercel this field is actually disabled so it's not possible to change it without bailing out and setting a custom command. Despite having this set, all my apps are currently being built even if there are no changes.

[gsoltis]

Is this only for the first commit of a new branch, or is this happening on every subsequent commit as well?

[tknickman]

Hey @kieranm! While strange, this is intended behavior. Here is why we decided to make this change.

Originally, when turbo-ignore was experimental, we would always compare HEAD against HEAD^1 (the parent commit) to determine changes. This works most of the time, and is still the default when not deploying on Vercel. However, there are some corner cases that make this a dangerous default. The most obvious one is when the branch you are deploying has multiple commits.

Say you have a branch - feat-1 with the following commits:
ghi: a web change
def: a web change
abc: a docs change

When deploying feat-1, if using HEAD^1 as the comparison, only web would be identified as having changes, and thus would be the only app deployed, despite docs also having changes in the branch. This is really frustrating and confusing, and requires manual intervention from the end-user (you would have to redeploy).

To fix this, we don't compare against HEAD^1, we compare against the last commit that was deployed for the current branch. This ensures that you're always checking to see if anything from the last deployed commit up to and including your current branch HEAD contains any relevant changes.

Now, the issue you're experiencing comes in when the branch has never been deployed before. In this case, we don't have anything to compare against. So instead of falling back to HEAD^1 which could be frustratingly incorrect (pushing a new branch and having deploys skipped for no reason is not fun) as described above, we stay on the safe side and always deploy.

As @gsoltis pointed out, we do provide an escape hatch for this via the --fallback arg:

-f, --fallback <ref>       On Vercel, if no previously deployed SHA is available to compare against, fallback to comparing against the provided ref

This allows you to provide a comparison to use in the event that the previously deployed commit is not available (first push of branch etc.)

We're happy to entertain better options here if you have some suggestions as this is something that we fully admit is a bit confusing. But our goal is for turbo-ignore to never incorrectly skip a deploy. This does mean that sometimes if we can't come to a conclusive result we opt for the safe approach of deploying just in case.

[kieranm]

Thanks for the explanation! The behaviour makes sense

If there are no previous deploys for a branch, is there a way of checking for changes by comparing with the last deploy of the branch you are merging into? Eg. if feat-1 was being merged into staging, look at the last successfully deployed staging commit and then compare against that?

In any case, I would suggest that the copy in the screenshot on Vercel could change or be clarified, because it's currently a bit misleading if apps are going to be deployed for safety even if they haven't changed vs the base branch. The behaviour is sensible but just needs to be made clearer I think.

[tknickman]

Yes that would be the best solution here, and we're working on something internally to solve this that should be available soon.

Right now, this is a limitation on Vercel due to the way the repo is cloned (to keep clones speedy, we only get details about the current branch, no others) so Vercel doesn't know anything about the base to compare against.

[octref]

Do you have an ETA for this feature? Currently a new PR to update README would build all of our 5 projects for preview.

[octref]

@tknickman I'm trying to use a custom script to figure out SHA of main branch and provide that to --fallback. However, because of Vercel's shallow clone on the build machine, I can't even find the SHA of main.

[octref]

I opened #7724

[valeriangalliat]

I've come up with a workaround in the meantime

• Create a SSH keypair e.g. ssh-keygen -t ed25519 -f vercel
• Add the public key as a read-only deploy key in your GitHub repo
• Add the private key in your project's env vars on Vercel e.g. GITHUB_DEPLOY_KEY

Then run the following script as part of ignoreCommand:

mkdir -p ~/.ssh
echo "$GITHUB_DEPLOY_KEY" > ~/.ssh/id_ed25519
chmod 0600 ~/.ssh/id_ed25519
git remote add origin [email protected]:your-org/your-repo
GIT_SSH_COMMAND='ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_ed25519' git fetch origin main:refs/remotes/origin/main
npx turbo-ignore --fallback origin/main

There's a slight overhead to doing that, but most of the time there's not many extra refs to fetch between the shallow clone and main, and in our case with a monorepo that contains multiple Vercel projects, it's much better than having each new branch triggering a build in all the Vercel projects even if only touches one.

[Ghislain89]

@valeriangalliat how do you deal with main and production deployments? If the above scripts would run on main for a production deployment, it would compare main to origin/main and result in a 0 exit code, right?

[anthonyshew]

We've improved Turborepo on Vercel such that you no longer need to use this turbo-ignore. Instead, we encourage you to use the new setting to automatically ignore projects without changes: https://vercel.com/changelog/automatically-skip-unnecessary-deployments-in-monorepos-6UleK0qIbq8MByAtP6HP5U

[kaisermann]

Hey there 👋 Does using a custom ignore script prevent skip unnecessary deployments from running? My team has a monorepo with 3 apps (each app with a staging and production build). All of them use the turbo-ignore ignore script and have the "skip unnecessary deployments" turned on. We are still seeing at least one build on the first commit of a PR for each app/environment, even if they were unaffected. Am I missing something 😅?