Prototype: Inherit the maxParallelForks from Jenkins CPU count

[gharris1727]

The maxParallelForks is usually inherited from the CPU count on developer machines, and hardcoded to 2 in Jenkins.
This is an experiment to try increasing the parallelism in Jenkins.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[chia7712]

[2024-05-21T20:30:08.150Z] BUILD SUCCESSFUL in 2h 17m 22s
[2024-05-21T20:30:08.150Z] 321 actionable tasks: 119 executed, 202 up-to-date

almost speedup 50% 😄

[gharris1727]

It's hard to tell for sure. The load on Jenkins varies a lot, so this is a top 30th percentile build, not too out of the ordinary.
If running this build under low load can beat 1h5m (5th percentile) I would confidently say it's better. I need some more runs.

Take a look at this graph:

This PR isn't any of the short bars on this graph, it's 5/5 of the tallest bars. This is a serial execution graph, so measuring the total time taken multiplied by the parallelism, roughly. That means despite maybe saving some overall execution time, running the tests at high parallelism wasted a lot of time, waiting for the CPU or whatever else.

[gharris1727]

Here's some total task times across some different builds:

task	fast build	slow build	this build
org.gradle.api.tasks.testing.Test	3h 35m	16h 28m	1d 13h 53m
com.github.spotbugs.snom.SpotBugsTask	41m 13.667s	1h 16m 1.518s	4h 15m 30.979s
com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar	27.909s	1m 13.334s	21.103s
org.gradle.api.tasks.compile.JavaCompile	21.038s	47.985s	29m 53.841s
org.gradle.api.tasks.Copy	14.474s	24.524s	10.837s
org.gradle.api.tasks.bundling.Jar	12.142s	19.986s	11.885s
org.gradle.api.plugins.quality.Checkstyle	7.710s	9.435s	5.250s
org.gradle.language.jvm.tasks.ProcessResources	3.956s	9.788s	10.564s
org.gradle.api.tasks.scala.ScalaCompile	1.933s	2.731s	18.151s
org.gradle.api.DefaultTask	0.533s	1.860s	1.198s
org.gradle.api.tasks.JavaExec	0.286s	0.634s	0.800s

The 30minute compile and 4h spotbugs are very strange to me, plus just to cherry-pick from the build:

:streams:upgrade-system-tests-36:test | 1h 35m 50.821s

There's no tests in this package! it took junit 95 minutes to scan the classes and not find any tests??!

And i don't even know what this line means, because it can't mean it took 28 minutes for gradle to determine the task was up to date, right?

:streams:upgrade-system-tests-37:compileTestJava UP-TO-DATE | 28m 52.150s

I think all of the above is only consistent with an extremely overloaded test runner.

[gharris1727]

It looks like 2 (or perhaps 3 or 4) is appropriate for the current CI conditions. I don't see any substantial difference that would make this a worthwhile change for us, or worth the risk of being worse CI neighbors than we already are.