Switch benchmarks to criterion #145
Conversation
Advantages over the previous system: * Allows running on stable Rust * Compares benchmarks between runs using statistics See https://github.com/japaric/criterion.rs for more.
benches/highlighting.rs
Outdated
|
|
||
| criterion_group!{ | ||
| name = benches; | ||
| config = Criterion::default().sample_size(10); |
There was a problem hiding this comment.
Note that more samples is better, but it takes a much longer time to run. But this can be adjusted if the measured runtime varies too much between runs of cargo bench.
There was a problem hiding this comment.
Tried it out locally, it's pretty neat, I like it. One of the sample sizes needs fixing though.
And also for the highlighting and parsing tests, there's a bug in criterion where it writes the results to paths like target/criterion/parse/"testdata/highlight_test.erb"/new/sample.json expecting the quoted part to be one directory but it actually creates it as a "testdata directory with the rest inside it. I'm not sure if this can only be fixed in criterion, or if there's a workaround to not put slashes in the benchmark names.
benches/loading.rs
Outdated
|
|
||
| criterion_group! { | ||
| name = benches; | ||
| config = Criterion::default().sample_size(100); |
There was a problem hiding this comment.
This sample size is too high, the load_syntaxes benchmark has an estimated time of 10 minutes, and others are pretty long.
|
Good pickup with the results (I only looked at the output, not the files). I'll look into it tomorrow and either raise an issue or a fix. To work around it in syntect we could instead use numerical indexes or simpler strings and then look up the test file in a Map. What do you think? |
|
Yah a string map seems like a fine workaround, numerical indexes would probably lead to the files being numbers and thus harder to navigate. |
So that the file paths in target are a bit less weird, see bheisler/criterion.rs#142.
|
Done using a match, and raised an issue for criterion: bheisler/criterion.rs#142 |
Advantages over the previous system:
See https://github.com/japaric/criterion.rs for more.