ci: Benchmark with CodSpeed.io (#187)

Experimentally replace the
[bencher.dev](https://bencher.dev/perf/portgraph) CI with
[codspeed's](https://codspeed.io/CQCL/portgraph).

On first review I had overlooked codspeed's criterion wrapper that
replaces the multi-run benchmarks with a single-run equivalent, similar
to iai's.

.github/workflows/ci.yml

@@ -54,7 +54,7 @@ jobs:
         run: cargo miri test
 
   benches:
-    name: continuous Benchmarking
+    name: continuous benchmarking
     # Not required, we can ignore it for the merge queue check.
     if: github.event_name != 'merge_group'
     runs-on: ubuntu-latest
@@ -67,56 +67,18 @@ jobs:
       - uses: Swatinem/rust-cache@v2
         with:
           prefix-key: v0
-
-      # The installed iai-callgrind-runner version must match the
-      # version of iai-callgrind in the Cargo.toml
       - uses: cargo-bins/cargo-binstall@main
-      - name: Install iai-callgrind-runner
-        run: |
-          version=$(cargo metadata --format-version=1 |\
-            jq '.packages[] | select(.name == "iai-callgrind").version' |\
-            tr -d '"'
-          )
-          cargo binstall --no-confirm iai-callgrind-runner --version $version --force
-
-      - uses: bencherdev/bencher@main
-      - name: Install valgrind
-        run: sudo apt update && sudo apt install -y valgrind
-
-      - name: Track base branch IAI benchmarks
-        if: github.event_name == 'push'
-        run: |
-          bencher run \
-          --project portgraph \
-          --token '${{ secrets.BENCHER_API_TOKEN }}' \
-          --branch main \
-          --testbed ubuntu-latest \
-          --threshold-measure instructions \
-          --threshold-test t_test \
-          --threshold-max-sample-size 64 \
-          --threshold-upper-boundary 0.99 \
-          --thresholds-reset \
-          --err \
-          --github-actions '${{ secrets.HUGRBOT_PAT }}' \
-          --adapter rust_iai_callgrind \
-          "cargo bench --bench iai_benches"
-
-      - name: Track PR IAI benchmarks
-        if: github.event_name == 'pull_request'
-        run: |
-          bencher run \
-          --project portgraph \
-          --token '${{ secrets.BENCHER_API_TOKEN }}' \
-          --branch "${{ github.event.pull_request.head.ref }}" \
-          --testbed ubuntu-latest \
-          --start-point "${{ github.event.pull_request.base.ref }}" \
-          --start-point-clone-thresholds \
-          --start-point-reset \
-          --err \
-          --github-actions '${{ secrets.HUGRBOT_PAT }}' \
-          --adapter rust_iai_callgrind \
-          "cargo bench --bench iai_benches"
-        # --start-point-hash '${{ github.event.pull_request.base.sha }}' \
+      - name: Install cargo-codspeed
+        run: cargo binstall cargo-codspeed --force
+      - name: Override criterion with the CodSpeed harness
+        run: cargo add --dev codspeed-criterion-compat --rename criterion
+      - name: Build benchmarks
+        run: cargo codspeed build criterion_benches --profile bench
+      - name: Run benchmarks
+        uses: CodSpeedHQ/action@v3
+        with:
+          token: ${{ secrets.CODSPEED_TOKEN }}
+          run: "cargo codspeed run criterion_benches"
 
   tests:
     runs-on: ubuntu-latest

DEVELOPMENT.md

@@ -59,11 +59,15 @@ This is the simplest kind of benchmark. To run the, use:
 cargo bench --bench criterion_benches
 ```
 
+We run these on CI to track historical performance using a special single-shot test harness,
+and upload them to [codspeed.io](https://codspeed.io/CQCL/portgraph).
+
 ### Single-shot benchmarking
 
-These benchmarks are useful when running in noisy environments, in addition to
-being faster than criterion. We run these on CI to track historical performance
-in [bencher.dev](https://bencher.dev/perf/portgraph).
+We use [`iai-callgrind`](https://iai-callgrind.github.io/iai-callgrind) for
+single-shot benchmarking, measuring instruction counts rather than wall-clock
+time. These benchmarks are more precise when running in noisy environments, in
+addition to being faster than criterion.
 
 To run these, you must have [`valgrind`](https://valgrind.org/) installed.
 Support for Apple Silicon (M1/M2/...) macs is

README.md

@@ -5,7 +5,7 @@ portgraph
 [![crates][]](https://crates.io/crates/portgraph)
 [![msrv][]](https://github.com/CQCL/portgraph)
 [![codecov][]](https://codecov.io/gh/CQCL/portgraph)
-[![bencher][]](https://bencher.dev/perf/portgraph)
+[![codspeed][]](https://codspeed.io/CQCL/portgraph)
 
 Data structure library for directed graphs with first-level ports. Includes
 secondary data structures for node and port weights, and node hierarchies.
@@ -38,5 +38,5 @@ This project is licensed under Apache License, Version 2.0 ([LICENSE][] or http:
   [LICENSE]: LICENCE
   [msrv]: https://img.shields.io/badge/rust-1.75.0%2B-blue.svg?maxAge=3600
   [codecov]: https://img.shields.io/codecov/c/gh/CQCL/portgraph?logo=codecov
-  [bencher]: https://img.shields.io/badge/bencher-.dev-blue.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAMAAABEpIrGAAAAtFBMVEVHcEwEBAT+/v4AAADi4uKenp7u7/ABAQEAAAAAAAAAAAAwMDAAAAAAAAAAAACGh4e2tLJfX18AAAAAAAD///++wMEAAABoaGiRkZEAAABucXMAAAAAAAB1dXUxMTHMzMzh4eH/0rRJSUmsrKz29vb////+q3v8bQD9+PP/8+kZGhp9fn/8dAD/3cXk4OT/tIL9p2z8ZwD+vI+UlJTPmnn/5tT/xqH/2cD9k1D9kkL8ijj8fhszkVn/AAAAJXRSTlMA4fgz7ePq/o3zRv0hfcfb7dJoVubyDvvpqfjTt+Dl+tTz3uPjxce2cgAAAaJJREFUOI2lklmTqjAQhcMSOpFFQHB3nJl7O2wii46z/f//daOOJYkvU3XPA6Q4X53udEPIbxWv1+x2ZsF6ovsLd05n/PJ5wmd07jLVj7mDaFKwCPE4NREdrmbwEOu+MOerKQnmZtHXGBpDfwp28dV9YgKWxf/iZ/dV2OV0CLjO227X1cgnE451t9u9Oa43jChtCWQ9vsbxK/aZBOxS6cFI848sE7jx/Q2KLPvIU6UHwiJszsCYsfEZaDBS72nxJO/uQJcn3FIHEVBsuxvQtUgDbZIj1ym+BUZBEKH4Lhx3pM+aUzy1GBlGhO0JqaH7ZARy2Dc58BBAGDzfgWdguu+7gwAZoW/T4jYqsrUiK4qa1DanpQw4VObVM6uDjCiHy4pnJqYA1RWoAFI0Z/HwChW+HwEOV+AAcHzHiilA3TbH8FY/PDZtrQD+8kWIppBecXk0Qrws/QHglfZJ/oeIuRC5fNX9SW2SGBTPRo77/c9hrG7D4uml+l7qctNQ/x9GsEnuU0qWj9vyFvwp3NpS2z9PsPB0/1zGDziUJfCV/7js/9A/nA48HxVN/KwAAAAASUVORK5CYII=
+  [codspeed]: https://img.shields.io/endpoint?url=https://codspeed.io/badge.json
   [CHANGELOG]: CHANGELOG.md