Improve build cache handling that does not rely on Cargo build-plan feature

[celinval]

Proposed change: I don't have a great solution for this; thus, I decided to use the Cargo's --build-plan feature for now. However, we need to figure out a more robust solution for the long term.

Ideally, we should use some API provided by Cargo.

Motivation: Build plan seems to be deprecated (rust-lang/cargo#7614), so we need a better solution ASAP. I posted a comment asking for some guidance: rust-lang/cargo#7614 (comment)

As I mentioned in the comment, I was hoping to use compiler artifact messages for that, but unfortunately the information provided is incomplete. I tried to extract the artifact name from the artifact messages, but for things like binaries, static / shared libraries, the list of filenames do not include the original files that were generated by the compiler.

E.g.: For a static library, this is what I get:

"filenames":["/tmp/test/static_lib/target/debug/libstatic_lib.a"]

while for a rust library, I get:

"filenames":["/tmp/test/libA/target/debug/liblibA.rlib","/tmp/test/libA/target/debug/deps/liblibA-728fd0386e8e49d7.rmeta"]

On the other hand, cargo build --build-plan includes the name of the file:

"outputs": ["/tmp/test/static_lib/target/debug/deps/libstatic_lib-edb6526891c85cb2.a" ],

[celinval]

Other very hacky possibilities that I thought so far are:

1. Change the compiler to generate a dummy artifact. Get the artifact provided by artifact messages. Compare the inode of this artifact will all candidates. Cargo uses a hardlink for artifacts in the same HD partition.
2. Change the compiler to generate a dummy artifact that contains the name of the original file. The driver could read the artifact and extract the name of the original file.

Both of these hacky solutions really bother me since we will generate "invalid" files. I.e.: ELF files that are in fact text files or empty files.

[celinval]

It seems that build-plan doesn't work for us either. Since we use cargo rustc instead of cargo build, there are differences when we manually select the target, which happens when we use --tests option. 😕

[celinval]

As I documented in #2246, I gave up on using --build-plan to retrieve the artifacts built by cargo kani. I tried to implement that, but it didn't work when we executed Kani with --tests option. I manually validated that the hash for the following invocations are actually different:

cargo build --profile tests

and

cargo rustc --profile tests

Unfortunately, --build-plan only works for cargo build, and since it is being deprecated, there is no plan to extend it to other Cargo commands (rust-lang/cargo#5855).