drone: use emptyDir for /var/lib/docker filesystem and prevent repetitive docker pulls

[webvictim]

Explicitly mounts Docker-in-Docker's /var/lib/docker using an emptyDir, which will be backed by the worker's SSD filesystem. This will remove a layer of abstraction so we're not writing a Docker overlay filesystem inside a container which is itself running on an overlay filesystem.

Also adds a check when running in CI to see whether a Docker image with a matching name:tag is already present and avoid pulling if so. This prevents an edge case I introduced recently where a second image pull will reset the image digest after a changed build. The effect is described below.

Current pipeline:

• make buildbox
  • triggers docker pull buildbox
  • triggers docker build buildbox
    • if cache matches: this is basically a no-op
    • if cache doesn't match: buildbox gets rebuilt
• make lint
  • triggers docker pull buildbox again (which will change the digest back if docker build changed it)
  • triggers docker build buildbox again
    • if cache matches: no-op
    • if cache doesn't match: buildbox needlessly gets rebuilt again
• make test (same thing happens)
• make integration (same thing happens again)

New workflow with this PR (changes in bold):

• make buildbox
  • triggers docker pull buildbox as there is no image with that name:tag locally
  • triggers docker build buildbox
    • if cache matches: this is basically a no-op
    • if cache doesn't match: buildbox gets rebuilt
• make lint
  • no longer triggers docker pull buildbox again as there is a matching image locally
  • triggers docker build buildbox again
    • this will always be a no-op using cache
• make test (uses local image)
• make integration (uses local image)

[awly]

optional: is it possible to squash all these permutations of the buildbox with something like:

buildbox-fips: BUILDBOX="$(BUILDBOX_FIPS)"
buildbox-fips: BUILDBOX_NAME="$(BUILDBOX_FIPS_NAME)"
buildbox-fips: buildbox

[webvictim]

I tried to do this lots of different ways before I started doing it this way, but it doesn't seem possible without removing the buildbox target as a dependency of release (which would probably hamper people's ability to use this locally). Each buildbox has a different Dockerfile and slightly different build arguments too which makes it harder.

There's probably a much smarter way to do this that I'm missing.

[awly]

I'm not seeing a huge difference in pipeline runtimes compared to a few open PRs I checked.
Will this change show up as faster pipelines or only as lower bandwidth usage?

[webvictim]

It should be marginally faster. Using in-memory tmpfs would be faster still, but I'm not confident that we wouldn't exhaust the amount of worker RAM Kubernetes allows you to use for that (which is 50%)

I expect this to excel in situations where we have multiple test runs going at once - it should improve reliability and consistency.

I think some of the slowdowns we're seeing are also related to Drone committing test logs back to its database, which will need some downtime to fix.

[russjones]

Bot.