Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve isolation of runtime benchmark compilation #1621

Merged
merged 6 commits into from
Jun 28, 2023

Conversation

Kobzol
Copy link
Contributor

@Kobzol Kobzol commented Jun 23, 2023

This PR improves the "isolation" of runtime benchmarks, in preparation of executing them in CI. This is done in two ways:

  1. Benchmark groups (crates) are now compiled in a temporary directory by default, avoiding potential artifact conflicts between different versions of rustc. For compile benchmarks, we copy the whole crate to a temporary directory. I'm not sure if that's needed, so here I just decided to redirect the --target-dir when invoking cargo.

  2. Runtime benchmarks are no longer part of the same workspace. While this did speed up their compilation by sharing dependencies, I realized that this could be problematic. Because a dependency change in some runtime benchmark crate (or the addition of a new crate) could change the existing benchmarks, which is something that we want to avoid. The runtime benchmark crates still depend on the benchlib crate, but they do so outside of the main workspace, so any dependency changes will be obvious and reflected in the lockfiles of the individual runtime benchmark crates.

    Sadly, currently due to a limitation of Cargo, every runtime benchmark has to contain an empty [workspace] section (same as for compile benchmarks basically). I removed the exclude section in the last commit, because it was confusing, as it wasn't actually doing anything.

Best reviewed commit by commit. Most changes come from the newly added lockfiles, the actual diff is about 200 lines.

@Kobzol Kobzol requested a review from nnethercote June 23, 2023 15:30
Kobzol added 6 commits June 23, 2023 17:39
This should make the builds more isolated. Most importantly, this makes sure that each benchmark group has a separate lockfile, and thus updating a version of a dependency of another (or a new) runtime benchmark will not affect the other benchmarks.
…arks

For CI, we want as large isolation as possible, but for local usage it is nice to just rerun the benchmarks as fast as possible.
These don't really do anything due to a Cargo limitation (rust-lang/cargo#6745).
@Kobzol Kobzol force-pushed the runtime-compilation-isolation branch from fe7b5a8 to a61443f Compare June 23, 2023 15:40
@Kobzol Kobzol merged commit 255dd73 into rust-lang:master Jun 28, 2023
@Kobzol Kobzol deleted the runtime-compilation-isolation branch June 28, 2023 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants