Use cargo artifacts in dev environment

[SkamDart]

Crane, similar to the rest of the nix-rust builders, relies on crates-io for dependencies in the nix-shell
This is a bit off the beaten path for nix users where you (usually) get the dependencies from your nix-store or cache before rebuilding from source.

Cargo and crates-io are wonderful tools but this is a bit non-standard for nix users.

Has this project considered exposing cargoArtifacts to cargo in the nix-shell environment?

One rather janky way I envision this happening is adding a shellHook to set the target directory and symlinking the dependencies from the nix-store to said directory.

Cargo used to have a "build-plan" which would be ideal here but it seems to deprecated.
rust-lang/cargo#7614

Thoughts?

[dpc]

I'm not sure if I understand, but cartoArtifacts (or shall we say: most stuff created in ./target/ directory) are not very reusable. They vary based on OS, environment variables, available system C libraries, cargo config options, toolchain version, enabled cargo features, etc.

[SkamDart]

We can control all of these with Nix!

[dpc]

Yes, but the chances that two users will have exactly same combination are becoming increasingly small, making the caching largely pointless.

The re-usability of cargo artifacts makes sense only within the same organization, largely around just working on the exactly same project, pinned to the same inputs. It's trivial to plug in cachix (or other private Nix cache) and achieve the same results, without the overhead of global caching infrastructure.

On top of it crane is specifically designed to avoid the complexity of largely reimplementing chunks of cargo building logic to allow per-crate derivations. For that see cargo2nix.

[luveti]

One rather janky way I envision this happening is adding a shellHook to set the target directory and symlinking the dependencies from the nix-store to said directory.

I have actually done just this in my own nix cargo/rust solution, as I had trouble using https://github.com/twistedfall/opencv-rust with many of these other solutions. I also needed workspace support, which many of these don't support.

CARGO_HOME, Cargo.toml and Cargo.lock can indeed be symlinked into your project directory. You can also mostly prevent cargo from touching these by using export CARGO_NET_OFFLINE="true". But the target directory needs to be extracted from the nix store as it needs to be writable for cargo build to work. Cargo uses fingerprints to determine when/if things to be rebuilt, and a nix-shell is quite different than the nix sandbox. Many packages that run a build script will rebuild after being extracted. It would appear changing CARGO_HOME always results in crates being rebuilt. I've ran into many issues with cargo picking up system libraries or using system binaries for building stuff. So I actually clean the PATH variable in my shellHook using some python code.

This has been an uphill battle. What we really need for a pure cargo/rust shell is chrooted shells. Combining that with overlayfs would allow us to reuse the target directory. CARGO_HOME could also always be set to /build/.cargo.

[SkamDart]

Yes, but the chances that two users will have exactly same combination are becoming increasingly small, making the caching largely pointless.

The re-usability of cargo artifacts makes sense only within the same organization, largely around just working on the exactly same project, pinned to the same inputs. It's trivial to plug in cachix (or other private Nix cache) and achieve the same results, without the overhead of global caching infrastructure.

I don't agree with these points. Caching build artifacts between your development environment, CI, and other projects is valuable outside of the 'same organization'.

On top of it crane is specifically designed to avoid the complexity of largely reimplementing chunks of cargo building logic to allow per-crate derivations

It sounds the feature I am suggesting is a non-starter for this project.

[ipetkov]

Has this project considered exposing cargoArtifacts to cargo in the nix-shell environment?

@SkamDart yes I have considered it, the issue has been finding a robust solution. The last time I tried looking into that there were issues with cargo rebuilding things anyway. Come to think of it, the last time I even tried building in one directory then copying the artifacts somewhere else it would crash the compiler (though maybe this is related to incremental builds?)...

I have a suspicion something might be sensitive to the full path to the ./target directory. It works out fine in the Nix builder since they usually get built in the same (chrooted) path, but this isn't the case for "local" builds. @luveti has also highlighted the additional sources of cache invalidation which are harder to control for outside of the Nix sandbox

It sounds the feature I am suggesting is a non-starter for this project.

I wouldn't go quite that far. If someone is willing to investigate and find a solution that works I'd be happy to include it in the documentation! Even though crane's design is not solely focused on as granular caching of individual crate files, caching is still a design goal to strive for; so far I've been wanting to avoid potential "false advertising" if the theoretical applications don't work out in practice...

[szlend]

yes I have considered it, the issue has been finding a robust solution. The last time I tried looking into that there were issues with cargo rebuilding things anyway. Come to think of it, the last time I even tried building in one directory then copying the artifacts somewhere else it would crash the compiler (though maybe this is related to incremental builds?)...

I've been doing this in CI on a few projects for a while now with the cache working perfectly. I think it's actually pretty neat. The main reason I began looking into this was that I was unhappy with running tests in the Nix sandbox.

The reasons were:

• Integration tests might might need to reach out to the (localhost) network
• It's tricky to extract out test artifacts like junit reports without making the build always succeed
• Making the build always succeed caches failures which is really bad if you ever run into a flaky test

So I just decided to build buildDepsOnly and then extract the build artifacts and run the rest in a dev shell. The key thing is that this shouldn't be your regular dev shell that you work in. It needs to be purposely built to exactly match your buildDepsOnly/buildPackage args, with a few additional crane hooks, and then it works fine.

Btw there's no guarantee that the Nix sandbox build directory stays the same between depsOnly and package either. So this is not unique to dev shells. In fact on Darwin it changes very often because of /tmp collisions.

I'll see if I can clean up and upstream some of this when I find some free time.