CI tweaks

[dnicolodi]

The main goal here is to cut down the total number of jobs we run from 41 to to something more sensible.

I got them down to:

• CPython 3.7 and 3.11, and on Windows, macOS, and Ubuntu: 6 jobs
• CPython 3.8, 3.9, 3.10, and PyPy 3.9 on Ubuntu: 4 jobs
• CPython 3.9 in Cygwin: 1 job
• Pyston 3.8 on Ubuntu: 1 job
• CPython 3.7 and 3.11 on Homebrew: 2 jobs
• a mypy job,
• the precommit tests,
• the Sage build job and the 8 Sage test jobs.

That's 25 jobs total.

I've experimented with running the Sage workflow only of the tests workflow succeeds but that has the annoying consequence of removing the Sage workflow test results from the PR view. The only way I know to avoid this is to have the Sage jobs defined in the same workflow as the other tests, but it does not seem very tidy.

[FFY00]

Would it be possible to run everything on the main branch, but a fewer jobs on PRs? Sometimes there are version specific issues, that we still want to catch, but as they are not common enough it might be hard to justify doing that in PRs.

[dnicolodi]

Sure, but you would most likely end up enabling the skipped tests in the PR that tries to fix the issues founds on main. Not a big deal, if infrequent enough. We could have the Sage tests run on push to main only. These are the ones that take the most time and are unlikely to catch anything not caught by the other tests.

[FFY00]

I think the Sage tests are useful in PRs, as they test real work scenarios, but I don't think we need these many in PRs. We could have only archlinux and conda-forge, for eg.

[dnicolodi]

Can do that. Not running the gentoo test on PRs is already a big saving.

[dnicolodi]

The workflow definition look a bit convoluted, but it achieves what we want. I'm surprised that the Sage jobs take so long, but that's a fight for another time. @FFY00 Unless there is something you don't like, I'll like to merge this soon as it is a big improvement upon the current status.

[FFY00]

@dnicolodi yes, sounds good. I am gonna tag a release, feel free to merge afterwards.

[dnicolodi]

I would appreciate having the documentation for the new options before the release. But I can live without that.

[FFY00]

I did not do that because we really needed a release, and we are planning a documentation rewrite anyway (#197) 😅

[webknjaz]

@dnicolodi moreover — this implementation was broken anyway.

https://github.com/marketplace/actions/alls-green#why

[henryiii]

It was not faulty. It just gets marked as skip instead of fail, but it's required to merge, so that's the same result. It would be better if it always ran, and reported the result (via alls-green) probably, but it still lets you do things like "merge when CI passes".

Why was this commented out? It doesn't slow the job by reducing parallism since it always runs at the end. So there's an extra at most 7 seconds or something tacked on the end. And it lets you makes the "passing" requirement in GitHub much simpler, and changes to it don't cause previous jobs to turn orange.

[webknjaz]

but it still lets you do things like "merge when CI passes".

Not exactly. It also lets you do things “merge when CI fails” because skipped==pass in branch protection if you start relying on this implementation.

[webknjaz]

It was not faulty.

Read the whole page at https://github.com/marketplace/actions/alls-green#why for explanation.

[henryiii]

Ahah, so that action is removing the "merge when CI fails" feature. 👍

[dnicolodi]

Why was this commented out? It doesn't slow the job by reducing parallism since it always runs at the end. So there's an extra at most 7 seconds or something tacked on the end. And it lets you makes the "passing" requirement in GitHub much simpler, and changes to it don't cause previous jobs to turn orange.

I commented it out because it is not used. Because all the projects within the mesonbuild organization share the GitHub Actions runners quota, it happened several times that the first batch of jobs from the meson-python is queued then some long-running jobs from the meson or wrapdb projects are queued and the summary job is left waiting for a long time for a runner just to execute an echo.

And it lets you makes the "passing" requirement in GitHub much simpler

In general I don't think that having something enforcing the policy that a PR is to be merged only when the test suite passes is very useful. I think we can trust our pairs to behave. The test suite was failing on several configurations and these tests were simply excluded from the required set. The fact that merging was allowed made no one paid attentions to what was actually a bug. Just merge when all the tests are green is a much simpler and much more effective policy.

changes to it don't cause previous jobs to turn orange

I don't know what you mean by this. I don't think the status of open PRs changes when the CI jobs definitions on the main branch change. If it would, that would be a welcome feature: PRs should be tested against the current main branch.