allow exporting to github cache backend

[tonistiigi]

Client-side support for moby/buildkit#1974

--cache-to type=gha[,scope=foo]
--cache-from type=gha[,scope=foo]

If called outside github environment and required env are not present then the configuration will be skipped. Token/URL can also be passed directly in csv but I would expect only a program wrapping buildx to use that method.

Signed-off-by: Tonis Tiigi [email protected]

[tonistiigi]

Added caps detection so that type=gha can be used even if current buildkit does not expose that backend without getting a fatal error. We should be able to merge this PR now and make it easier to test buildkit PR.

depends on #565

[tonistiigi]

I've taken this out of draft. If the caps detection works and we are ok with the ux we can merge this and continue testing/updating the Buildkit PR.

[crazy-max]

Ux is fine to me to start with. Thanks

[JanJakes]

Is there any way to use this with https://github.com/docker/setup-buildx-action yet? Or do we need to wait when Buildx is tagged?

[douglasduteil]

Are we waiting for 0.6.0 version @crazy-max ?
Or...

Is it already working somehow

https://github.com/crazy-max/test-docker-action/blob/46a3fac1d1df70169dc7f72ef93b500e762e8a03/.github/workflows/gha-cache.yml#L44-L53

        name: Build
        uses: docker/build-push-action@master
        with:
          context: .
          file: ./Dockerfile
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.docker_meta.outputs.tags }}
          labels: ${{ steps.docker_meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha

[crazy-max]

@JanJakes @douglasduteil We need the server-side first moby/buildkit/pull/1974

Is it already working somehow

I was using a staging build to test the feature.

[cdeneen]

Does anyone have a complete example of using this, are you still required to setup the cache in an action?

[adrienjoly]

Here's how I ended up caching of our "admin-api" Docker layers, on GitHub Actions:

  e2e:
    name: End-to-end tests against Docker containers
    runs-on: ubuntu-latest
    env:
      DOCKER_BUILDKIT: '1' # to enable persistent docker cache
      COMPOSE_DOCKER_CLI_BUILD: '1' # so docker-compose commands benefits from buildkit/buildx's docker cache
    steps:
      - uses: actions/checkout@v2

      # Use GitHub's Docker registry to cache layers and reduce image build time from a run to the other
      - run: echo ${{ secrets.GITHUB_TOKEN }} | docker login docker.pkg.github.com -u $GITHUB_ACTOR --password-stdin
      - run: docker pull docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build || true
      - run: docker build . --file Dockerfile-local --tag admin-api --cache-from=docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build --build-arg BUILDKIT_INLINE_CACHE=1
      - run: docker tag admin-api docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build && docker push docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build || true
      # source: https://dev.to/dtinth/caching-docker-builds-in-github-actions-which-approach-is-the-fastest-a-research-18ei

      - name: 'Build admin-api with docker-compose' # so the test suite does not have to
        run: docker-compose -f docker-compose-local.yml build admin-api
      - name: Run end-to-end tests
        run: yarn test:e2e:docker # starts the APIs and services with docker-compose before running tests

It adds and updates the resulting Docker image to the "Packages" section of our GitHub repository.

Hope this helps.

[valentijnscholten]

@adrienjoly this PR is about a new "GitHub native"cache backend being implemented that should be faster / better with regards to expiration etc. Caching to github package registry is kinda old school and slow in my experience.

[adrienjoly]

My bad, thanks for the clarification.

[cdeneen]

Yeah, I was just looking to see the entire cache system in use in an example. @douglasduteil posted a piece of code from @crazy-max but the repo is no longer exists or is public.

name: Build
        uses: docker/build-push-action@master
        with:
          context: .
          file: ./Dockerfile
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.docker_meta.outputs.tags }}
          labels: ${{ steps.docker_meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha

[crazy-max]

Will be available when moby/buildkit#1974 is merged.

[benesch]

If anyone else is looking to give this a shot without having to build docker buildx from source, I put together https://github.com/benesch/setup-docker-caching. It can set up buildx/Buildkit with the new GHA backend in <10s. It also exports the ACTIONS_ env vars so that you can use docker build directly instead of going through docker/build-push-action.

[patroza]

If anyone else is looking to give this a shot without having to build docker buildx from source, I put together https://github.com/benesch/setup-docker-caching. It can set up buildx/Buildkit with the new GHA backend in <10s. It also exports the ACTIONS_ env vars so that you can use docker build directly instead of going through docker/build-push-action.

Thanks.

Im getting:

time="2021-07-13T18:39:55Z" level=warning msg="No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"
15
error: failed to solve: rpc error: code = Unknown desc = url not set for github actions cache

[crazy-max]

@patroza You can test the gha cache exporter using this workflow from the official Build Push Action while waiting for buildx 0.6 and BuildKit 0.9 to be GA.