Suboptimal caching on Actions CI

[patrickt]

👋 Hey, everyone. I’m running into some weird behavior when integrating Bazel into Semantic CI. Though Bazel caches the Stackage snapshot correctly on CI when combined with https://github.com/actions/cache, the projects defined in the semantic repository are being rebuilt every time. I’m hoping that someone can point out something I’m doing wrong, even though this is perhaps beyond the purview of rules_haskell itself (N.B. I based the config file off of @aherrmann’s one here.

Some links that (hopefully) demonstrate what I'm talking about:

• The workflow YAML file I’m using
• An example build log

As per the examples, I am using .bazelrc to specify a local directory for disk_cache and repository_cache, then adding that directory to the path key in the cache step.

Note that in the build log, there’s an extensive Analyzing… phase that happens before the projects are compiled; since that doesn’t happen on a local build with full caching, I presume that this analysis step is the culprit, but I’m at a loss as to how to introspect as to why this analysis phase is occurring.

Interestingly enough, the test phases cache correctly—it’s only the main builds that aren’t caching.

Things I’ve tried:

• Downloading a bazel executable myself so that the bazel that ships with the builder image doesn’t have to do the decompression step;
• caching bazel-out / /home/runner/cache: this works, but balloons the cache greatly;
• ensuring there are no git_repository calls, since git_repository appears not to respect the repository cache;

Would bazelisk help here?

Any insight would be much appreciated!

[aherrmann]

Hi @patrickt,
Thanks for the detailed report and the links.

Though Bazel caches the Stackage snapshot correctly on CI when combined with https://github.com/actions/cache, the projects defined in the semantic repository are being rebuilt every time.

Looking at the build log I see it reports

...
INFO: 427 processes: 427 remote cache hit.
...
INFO: 2 processes: 2 remote cache hit.
...

which looks like you're getting cache hits on all builds. Note that Bazel also caches build logs and reproduces them when fetching targets from cache. So, you'll see GHC output even on a cached build. IIRC the motivation for this behavior is to not hide compiler warnings due to caching.

Note that in the build log, there’s an extensive Analyzing… phase that happens before the projects are compiled; since that doesn’t happen on a local build with full caching, I presume that this analysis step is the culprit, but I’m at a loss as to how to introspect as to why this analysis phase is occurring.

Bazel's caching of repository rules is currently limited to downloads, e.g. http_file and http_archive, see bazelbuild/bazel#5086. In particular Bazel cannot directly cache stack_snapshot or haskell_register_ghc_bindists. For stack_snapshot we've recently implemented pinning which enables Bazel to cache stack_snapshot in the repository cache. For haskell_register_ghc_bindists we currently don't have an option to cache them in the repository cache. One way to avoid this issue is to use a nixpkgs provided toolchain with haskell_register_ghc_nixpkgs. Other options are not yet implemented, see #1320 and #1393.

[patrickt]

Pinning definitely helped! We’re under 4 minutes now, which is really superb (and most of that is from restoring the binary cache).

[aherrmann]

Thank you for reporting back, that sounds great!

It sounds like this can be closed. Please reopen if there's another issue or something new comes up.