Separate target directories by a hash of flags and other inputs?

[anp]

I've done some searching in the issues on the cargo repo and haven't seen anything related to this -- if this is a duplicate please forgive me.

Has anyone ever proposed the use of an intermediate directory under target but above debug/release which is specific to a set of flags/build options? Something like target/HASH_OF_INPUTS/debug?

There are a couple of situations where I've recently found myself toggling flags on and off for both debug and release builds, where triggering a full rebuild is problematic:

• Measuring performance of different code changes with LTO on and then off, debuginfo on and then off, etc.
• Performing general compiler-driven coding, followed by running line coverage on tests. This means passing no-gc-sections in flags, where I wouldn't want that to be the default in my development environment.

Both of these workflows depend on comparing the effects small code changes across different sets of flags, but to do so currently kicks off a full rebuild of dependencies (which is a really killer amount of time to spend in between edits).

From a user's perspective, I can see a definite downside from disk usage growing as they experiment with different flags. At the same time, it would be nice to have these kinds of workflows "just work" without needing full dependency rebuilds or having to play games with CARGO_TARGET_DIR.

Hopefully I'm making sense here. Thoughts?

[alexcrichton]

Thanks for the report! This definitely makes sense to me, and I don't recall seeing any existing issues for it!

In some sense Cargo could just be a massive content-addressed storage mechanism. That is, all output aritfacts have some key which they're created with which includes information like:

• Dependencies
• Compiler flags/options
• Actual files used to produce the artifact
• Compiler used

We could have a database which in the limit reflects all of these and just caches basically everything we can find. The downside to this, of course, is that it requires manual intervention to do a gc which may need to happen often depending on the size of the build directory. So on one hand I do think this is useful as Cargo's definitely built to take these features into account, but I'm not sure whether it'd be worth the tradeoff in disk space.

[anp]

Indeed. I can imagine configuring the amount of "heap" space to be allowed for caching before triggering "GC," but that could definitely get messy. From #rust-tools I also just saw your comment on #1997 (comment), which seems like it could definitely cover the use I've proposed (and then some). Does it seem like there's a fair amount of overlap there? Or am I just seeing "CAS with hashing" and glossing over a key distinction?

(my main concern is not contributing to issue bloat if this is going to get built as part of a larger caching initiative)

[alexcrichton]

Ah yeah I think that would end up being pretty similar to this. In the limit this feature ends up building what I think is what's needed for #1997, a globally unique key for all artifacts. Once we're caching on that layer then we may as well make it distributed (e.g. just fetch/upload things to S3).

[ehuss]

Flags were added to the filename hash in #6503 which effectively addresses this issue by allowing you to swap between different flags without recompiling. The filename hash contains a fair bit of information now, so I'm going to close this as effectively implemented.

Details about the filename hash can be found in Metadata and the relationship to fingerprints is described in fingerprint.