type=gha caching error "failed to solve: failed to compute cache key: blob not found"

[JacobHayes]

I'm not sure this is the right repo, but yesterday a few of my GitHub Actions builds started failing with:

#18 [11/19] COPY requirements.txt .
#18 ERROR: blob not found
------
 > [11/19] COPY requirements.txt .:
------
Dockerfile:80
--------------------
  78 |     # [...]
  79 |     ENV [...]
  80 | >>> COPY requirements.txt .
  81 |     # [...]
  82 |     RUN [...]
--------------------
error: failed to solve: failed to compute cache key: blob not found
Error: buildx failed with: error: failed to solve: failed to compute cache key: blob not found

I see a few other errors around failed to compute cache key, but none for blob not found nor related to GHA caching backend. Our GHA cache may have been pruned a bit, but should buildkit's gha caching be able to compensate?

---

The GitHub workflow is setup like:

    runs-on: ubuntu-latest
    steps:
      - uses: actions/[email protected]
      # Build the docker image
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: v0.6.1 # v0.6.0 is the minimum version supporting the GH Actions cache backend
      - name: Build docker image and push
        uses: docker/build-push-action@v2
        with:
          build-args: GIT_SHA=${{ github.sha }}
          cache-from: type=gha,scope=${{ github.workflow }}
          cache-to: type=gha,mode=max,scope=${{ github.workflow }}
          context: .
          push: true
          tags: "[...]:${{ github.sha }}"

and here's the docker version, etc:

BuildKit version
  moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.9.0 c8bb937807d405d92be91f06ce2629e6202ac7a9

  /usr/bin/docker version
  Client:
   Version:           20.10.7+azure
   API version:       1.41
   Go version:        go1.13.15
   Git commit:        f0df35096d5f5e6b559b42c7fde6c65a2909f7c5
   Built:             Mon May 31 09:40:07 2021
   OS/Arch:           linux/amd64
   Context:           default
   Experimental:      true
  
  Server:
   Engine:
    Version:          20.10.7+azure
    API version:      1.41 (minimum version 1.12)
    Go version:       go1.13.15
    Git commit:       b0f5bc36fea9dfb9672e1e9b1278ebab797b9ee0
    Built:            Tue Jun  1 22:34:42 2021
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          1.4.9+azure
    GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
   runc:
    Version:          1.0.1
    GitCommit:        4144b63817ebcc5b358fc2c8ef95f7cddd709aa7
   docker-init:
    Version:          0.19.0
    GitCommit:        
  /usr/bin/docker info
  Client:
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Build with BuildKit (Docker Inc., v0.6.1)
  
  Server:
   Containers: 1
    Running: 1
    Paused: 0
    Stopped: 0
   Images: 17
   Server Version: 20.10.7+azure
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 1
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: e25210fe30a0a703442421b0f60afac609f950a3
   runc version: 4144b63817ebcc5b358fc2c8ef95f7cddd709aa7
   init version: 
   Security Options:
    apparmor
    seccomp
     Profile: default
   Kernel Version: 5.8.0-1039-azure
   Operating System: Ubuntu 20.04.2 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 6.791GiB
   Name: fv-az177-346
   ID: IBWM:5VKH:D3RK:3XE6:WEQ3:QOM4:MND4:L5CT:MIPV:Y6ZI:VP22:O47N
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Registry: https://index.docker.io/v1/
   Labels:
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false

I spot checked the versions ^ above between a passing and failing run and things appear to be the same.

[patrikholcak]

We have been using the gha cache for about 10 days and started seeing the same issue on all builds, always failing on the step that has changed from build to build. The image with all layers is around 200MB (.NET Core app) with a couple of builds per day.

#30 [frontend  9/10] COPY js ./js
#30 ERROR: blob not found
------
 > [frontend  9/10] COPY js ./js:
------
Dockerfile:21
--------------------
  19 |     COPY css ./css
  20 |     COPY less ./less
  21 | >>> COPY js ./js
  22 |     RUN yarn build
  23 |     
--------------------
error: failed to solve: failed to compute cache key: blob not found
Error: buildx failed with: error: failed to solve: failed to compute cache key: blob not found

runs-on: ubuntu-20.04
steps:
  - uses: actions/checkout@v2
  - uses: docker/setup-buildx-action@v1
  - uses: docker/login-action@v1
      with:
        registry: ghcr.io
  - uses: docker/build-push-action@v2
     with:
       context: .
       push: true
       tags: ghcr.io/…
       cache-from: type=gha
       cache-to: type=gha,mode=max

[BerriJ]

I experience the same issue with my Ubuntu-based development container:
https://github.com/BerriJ/devenv/pull/2/checks?check_run_id=3486547427

This issue is also discussed in the GitHub action: docker/build-push-action#422 and it's tracked in docker/buildx#681