cargo clippy is slower with sccache (after caches have been populated!) than a plain build

[nagisa]

On my system I'm seemingly observing sccache having a very hard time being quick.

I’m using https://github.com/near/nearcore/pulls as the test case here on a machine with 48 cores and 96 threads.

To establish a baseline, from the root of repository, I run:

$ rm -rf target/clippy/
$ env RUSTC_WRAPPER= CARGO_TARGET_DIR="target/clippy" cargo clippy --all-features --all-targets --locked --timings

which reports a total time of 105.7s (1m 45.7s).

---

Then running an sccache-enabled check for the first time (to build up the cache):

$ rm -rf target/clippy $PWD/../sccache-test
$ pkill sccache
$ env RUSTC_WRAPPER=$HOME/.cargo/bin/sccache SCCACHE_DIR=$PWD/../sccache-test CARGO_INCREMENTAL=0 CARGO_TARGET_DIR="target/clippy" cargo clippy --all-features --all-targets --locked --timings

During this run I observe the sccache process consuming multiple cores worth of CPU time (~600+%) at points of the build process. The total time here is 158.7s (2m 38.7s)

---

Second sccache-enabled run:

$ rm -rf target/clippy
$ env RUSTC_WRAPPER=$HOME/.cargo/bin/sccache SCCACHE_DIR=$PWD/../sccache-test CARGO_INCREMENTAL=0 CARGO_TARGET_DIR="target/clippy" cargo clippy --all-features --all-targets --locked --timings

Unfortunately is still slower than the non-cache build, coming in at 122.4s (2m 2.4s).

---

Looking at the sccache stats:

Compile requests                   4090
Compile requests executed          3106
Cache hits                         1551
Cache hits (C/C++)                  481
Cache hits (Rust)                  1070
Cache misses                       1551
Cache misses (C/C++)                481
Cache misses (Rust)                1070

it would seem that after the first build we’re seeing perfect cache utilization (cache hits == cache misses.) This suggests the 2nd build should get fetched from the disk, pretty much, but that either isn’t happening, is very slow for some reason or sccache is not able to handle the 96 build processes of parallelism.

Here are the build timings from cargo (NB: to see graphs you need to slide the "min unit time slider to non-zero and reload the page):

• no-sccache.html.gz
• sccache-initial.html.gz
• sccache-2nd.html.gz

sccache version: v0.7.4, installed via cargo install just now.
rustc: 1.75.0

I would appreciate some suggestions where to look to figure out what the problem here really is.

[sylvestre]

This is the expected behavior.
Sccache caches compilations. Clippy is static analysis. If you try sccache with clippy, add sccache overhead but still have to run the full analysis.

Maybe we should break the build to make this explicit.

[nagisa]

Well, hm, the reason I started looking into this at all was because my regular builds felt slower too. I can try to make this issue report about that instead, it just felt that clippy was a much better demo of the problem. Thanks for letting me know that isn't the case.

EDIT: though that will have to wait for a little bit.

[nagisa]

Hey,

Thank you for the patience! I tried it out with cargo build and was able to get build times in the order of 1 minute after a populated cache (as opposed to 1m 47s for a regular build.) It still feels like a lot of time is being spent somewhere in sccache, but at least this is not slower and I don’t have any other actionable insights so I’ll close this.

Thank you for making this tool!