Skip to content

Commit

Permalink
Add better diagnostics for rebuilding
Browse files Browse the repository at this point in the history
This commit overhauls how a `Fingerprint` is stored on the filesystem and
in-memory to help provide much better diagnostics as to why crates are being
rebuilt. This involves storing more structured data on the filesystem in order
to have a finer-grained comparison with the previous state. This is not
currently surfaced in the output of cargo and still requires
`RUST_LOG=cargo::ops::cargo_rustc::fingerprint=info` but if it turns out to be
useful we can perhaps surface the output.

There are performance considerations here to ensure that a noop build is still
quite speedy for a few reasons:

1. JSON decoding is slow (these are just big structures to decode)
2. Each fingerprint stores all recursive fingerprints, so we can't just "vanilla
   decode" as it would decode O(n^2) items
3. Hashing is actually somewhat nontrivial for this many items here and there,
   so we still need as much memoization as possible.

To ensure that builds are just as speedy tomorrow as they are today, a few
strategies are taken:

* The same fingerprint strategy is used today as a "first line of defense" where
  a small text file with a string contains the "total fingerprint" hash. A
  separately stored file then contains the more detailed JSON structure of the
  old fingerprint, and that's only decoded if there's a mismatch of the short
  hashes. The rationale here is that most crates don't need to be rebuilt so we
  shouldn't decode JSON, but if it does need to be rebuilt then the work of
  compiling far dwarfs the work of decoding the JSON.
* When encoding a full fingerprint as JSON we don't actually include any
  dependencies, just the resolved u64 of them. This helps the O(n^2) problem in
  terms of decoding time and storage space on the filesystem.
* Short hashes continue to be memoized to ensure we don't recompute a hash if
  we've already done so (e.g. shared dependencies).

Overall, when profiling with Servo, this commit does not regress noop build
times, but should help diagnose why crates are being rebuilt hopefully!

Closes #2011
  • Loading branch information
alexcrichton committed Oct 5, 2015
1 parent 8524efb commit c447e9d
Show file tree
Hide file tree
Showing 5 changed files with 244 additions and 75 deletions.
6 changes: 3 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion src/cargo/ops/cargo_rustc/context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ pub struct Context<'a, 'cfg: 'a> {
pub compilation: Compilation<'cfg>,
pub build_state: Arc<BuildState>,
pub exec_engine: Arc<Box<ExecEngine>>,
pub fingerprints: HashMap<Unit<'a>, Fingerprint>,
pub fingerprints: HashMap<Unit<'a>, Arc<Fingerprint>>,
pub compiled: HashSet<Unit<'a>>,
pub build_config: BuildConfig,
pub build_scripts: HashMap<Unit<'a>, Arc<BuildScripts>>,
Expand Down
Loading

0 comments on commit c447e9d

Please sign in to comment.