forked from SciTools/iris
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into votable_issues
* main: (69 commits) Revert plotting-vs-y (SciTools#4601) Bump peter-evans/create-pull-request from 3.13.0 to 3.14.0 (SciTools#4608) Support false-easting and false-northing when loading Mercator-projected data (SciTools#4524) Bump peter-evans/create-pull-request from 3.12.1 to 3.13.0 (SciTools#4607) Stop using nc_time_axis.CalendarDateTime (SciTools#4584) Utility class in netcdf loader should not be public. (SciTools#4592) Overnight benchmarks (SciTools#4583) Yaml fixes + clarifications. (SciTools#4594) Bump actions/script from 5.1.0 to 6 (SciTools#4586) Add missing commit to v3.2.x and update version number (SciTools#4593) docs linkcheck skip (SciTools#4590) Finalise whatsnew and version string update (SciTools#4588) [pre-commit.ci] pre-commit autoupdate (SciTools#4587) fix test (SciTools#4585) Loading Benchmarks (SciTools#4477) Fix load_http bug, extend testing, and note to docs (SciTools#4580) New tool-agnostic ASV environment management (SciTools#4571) Fix refresh lockfile worrkflow pull request title (SciTools#4579) gha: lockfiles labels and auto-pr details (SciTools#4578) Bump actions/script from 4 to 5.1.0 (SciTools#4576) ...
- Loading branch information
Showing
170 changed files
with
9,639 additions
and
6,019 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Reference: | ||
| # - https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/keeping-your-actions-up-to-date-with-dependabot | ||
| # - https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates | ||
|
|
||
| version: 2 | ||
| updates: | ||
|
|
||
| - package-ecosystem: "github-actions" | ||
| directory: "/" | ||
| schedule: | ||
| # Check for updates to GitHub Actions every weekday | ||
| interval: "daily" | ||
| labels: | ||
| - "New: Pull Request" | ||
| - "Bot" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,7 +22,9 @@ on: | |
| default: "no" | ||
| schedule: | ||
| # Run once a week on a Saturday night | ||
| - cron: 1 0 * * 6 | ||
| # N.B. "should" be quoted, according to | ||
| # https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onschedule | ||
| - cron: "1 0 * * 6" | ||
|
|
||
|
|
||
| jobs: | ||
|
|
@@ -35,7 +37,7 @@ jobs: | |
| # the lockfile bot has made the head commit, abort the workflow. | ||
| # This job can be manually overridden by running directly from the github actions panel | ||
| # (known as a "workflow_dispatch") and setting the `clobber` input to "yes". | ||
| - uses: actions/script@v4 | ||
| - uses: actions/script@v6 | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| script: | | ||
|
|
@@ -71,7 +73,7 @@ jobs: | |
|
|
||
| strategy: | ||
| matrix: | ||
| python: ['37', '38'] | ||
| python: ['38'] | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v2 | ||
|
|
@@ -108,13 +110,25 @@ jobs: | |
| rm -r artifacts | ||
| - name: Create Pull Request | ||
| uses: peter-evans/create-pull-request@052fc72b4198ba9fbc81b818c6e1859f747d49a8 | ||
| id: cpr | ||
| uses: peter-evans/create-pull-request@18f7dc018cc2cd597073088f7c7591b9d1c02672 | ||
| with: | ||
| commit-message: Updated environment lockfiles | ||
| committer: "Lockfile bot <[email protected]>" | ||
| author: "Lockfile bot <[email protected]>" | ||
| delete-branch: true | ||
| branch: auto-update-lockfiles | ||
| title: Update CI environment lockfiles | ||
| title: "[iris.ci] environment lockfiles auto-update" | ||
| body: | | ||
| Lockfiles updated to the latest resolvable environment. | ||
| labels: | | ||
| New: Pull Request | ||
| Bot | ||
| - name: Check Pull Request | ||
| if: steps.cpr.outputs.pull-request-number != '' | ||
| run: | | ||
| echo "pull-request #${{ steps.cpr.outputs.pull-request-number }}" | ||
| echo "pull-request URL ${{ steps.cpr.outputs.pull-request-url }}" | ||
| echo "pull-request operation [${{ steps.cpr.outputs.pull-request-operation }}]" | ||
| echo "pull-request head SHA ${{ steps.cpr.outputs.pull-request-head-sha }}" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # Iris Performance Benchmarking | ||
|
|
||
| Iris uses an [Airspeed Velocity](https://github.com/airspeed-velocity/asv) | ||
| (ASV) setup to benchmark performance. This is primarily designed to check for | ||
| performance shifts between commits using statistical analysis, but can also | ||
| be easily repurposed for manual comparative and scalability analyses. | ||
|
|
||
| The benchmarks are automatically run overnight | ||
| [by a GitHub Action](../.github/workflows/benchmark.yml), with any notable | ||
| shifts in performance being flagged in a new GitHub issue. | ||
|
|
||
| ## Running benchmarks | ||
|
|
||
| `asv ...` commands must be run from this directory. You will need to have ASV | ||
| installed, as well as Nox (see | ||
| [Benchmark environments](#benchmark-environments)). | ||
|
|
||
| [Iris' noxfile](../noxfile.py) includes a `benchmarks` session that provides | ||
| conveniences for setting up before benchmarking, and can also replicate the | ||
| automated overnight run locally. See the session docstring for detail. | ||
|
|
||
| ### Environment variables | ||
|
|
||
| * ``DATA_GEN_PYTHON`` - required - path to a Python executable that can be | ||
| used to generate benchmark test objects/files; see | ||
| [Data generation](#data-generation). The Nox session sets this automatically, | ||
| but will defer to any value already set in the shell. | ||
| * ``BENCHMARK_DATA`` - optional - path to a directory for benchmark synthetic | ||
| test data, which the benchmark scripts will create if it doesn't already | ||
| exist. Defaults to ``<root>/benchmarks/.data/`` if not set. | ||
|
|
||
| ## Writing benchmarks | ||
|
|
||
| [See the ASV docs](https://asv.readthedocs.io/) for full detail. | ||
|
|
||
| ### Data generation | ||
| **Important:** be sure not to use the benchmarking environment to generate any | ||
| test objects/files, as this environment changes with each commit being | ||
| benchmarked, creating inconsistent benchmark 'conditions'. The | ||
| [generate_data](./benchmarks/generate_data/__init__.py) module offers a | ||
| solution; read more detail there. | ||
|
|
||
| ### ASV re-run behaviour | ||
|
|
||
| Note that ASV re-runs a benchmark multiple times between its `setup()` routine. | ||
| This is a problem for benchmarking certain Iris operations such as data | ||
| realisation, since the data will no longer be lazy after the first run. | ||
| Consider writing extra steps to restore objects' original state _within_ the | ||
| benchmark itself. | ||
|
|
||
| If adding steps to the benchmark will skew the result too much then re-running | ||
| can be disabled by setting an attribute on the benchmark: `number = 1`. To | ||
| maintain result accuracy this should be accompanied by increasing the number of | ||
| repeats _between_ `setup()` calls using the `repeat` attribute. | ||
| `warmup_time = 0` is also advisable since ASV performs independent re-runs to | ||
| estimate run-time, and these will still be subject to the original problem. | ||
|
|
||
| ### Scaling / non-Scaling Performance Differences | ||
|
|
||
| When comparing performance between commits/file-type/whatever it can be helpful | ||
| to know if the differences exist in scaling or non-scaling parts of the Iris | ||
| functionality in question. This can be done using a size parameter, setting | ||
| one value to be as small as possible (e.g. a scalar `Cube`), and the other to | ||
| be significantly larger (e.g. a 1000x1000 `Cube`). Performance differences | ||
| might only be seen for the larger value, or the smaller, or both, getting you | ||
| closer to the root cause. | ||
|
|
||
| ## Benchmark environments | ||
|
|
||
| We have disabled ASV's standard environment management, instead using an | ||
| environment built using the same Nox scripts as Iris' test environments. This | ||
| is done using ASV's plugin architecture - see | ||
| [asv_delegated_conda.py](asv_delegated_conda.py) and the extra config items in | ||
| [asv.conf.json](asv.conf.json). | ||
|
|
||
| (ASV is written to control the environment(s) that benchmarks are run in - | ||
| minimising external factors and also allowing it to compare between a matrix | ||
| of dependencies (each in a separate environment). We have chosen to sacrifice | ||
| these features in favour of testing each commit with its intended dependencies, | ||
| controlled by Nox + lock-files). |
Oops, something went wrong.