Improve isolation of runtime benchmark compilation #1621
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR improves the "isolation" of runtime benchmarks, in preparation of executing them in CI. This is done in two ways:
Benchmark groups (crates) are now compiled in a temporary directory by default, avoiding potential artifact conflicts between different versions of
rustc
. For compile benchmarks, we copy the whole crate to a temporary directory. I'm not sure if that's needed, so here I just decided to redirect the--target-dir
when invokingcargo
.Runtime benchmarks are no longer part of the same workspace. While this did speed up their compilation by sharing dependencies, I realized that this could be problematic. Because a dependency change in some runtime benchmark crate (or the addition of a new crate) could change the existing benchmarks, which is something that we want to avoid. The runtime benchmark crates still depend on the
benchlib
crate, but they do so outside of the main workspace, so any dependency changes will be obvious and reflected in the lockfiles of the individual runtime benchmark crates.Sadly, currently due to a limitation of Cargo, every runtime benchmark has to contain an empty
[workspace]
section (same as for compile benchmarks basically). I removed theexclude
section in the last commit, because it was confusing, as it wasn't actually doing anything.Best reviewed commit by commit. Most changes come from the newly added lockfiles, the actual diff is about 200 lines.