Load test classes with runtime classloader

[holly-cummins]

Bugs fixed by this PR

• Resolves QuarkusTest: consider removing the test profile support for @Nested tests #45349
• Resolves Broken QuarkusTestExtension #8446
• Resolves Kotlin junit ParameterizedTest with functions as argument, not working anymore (Kotlin 2.0) #42000
• Resolves Regression in 3.13.0.CR1: Lambda expression from custom serializable interface used as @QuarkusTest method parameter fails with ClassNotFoundException #42006
• Resolves @ParameterizedTest with @MethodSource do not work when using @QuarkusTest annotation #44320
• Resolves Mockito fails to mock non-public inner class in continuous testing due to classloading issues #38987
• Resolves ConfigProvider.getConfig().getOptionalValue("..").get() does not work in JUnitExtension any more #46383
• Resolves Ecosystem CI failure NoClassDefFoundError: io/quarkiverse/pact/devmodetest/farm/FarmContractTest quarkiverse/quarkus-pact#272

Bugs created by this PR (doh!)

• Creates Dev Services should be started during startup, not augmentation #45785
• Creates Maven surefire.rerunFailingTestsCount option does not work in the case where there are test profiles or resources, unless the test order falls such that the failing test uses the last profile/resource #46048
• Creates Connection problems to Oracle in tests when another db is used in another resource first #46542

Outstanding issues/breaking changes (input to release notes)

• @TestProfile on @Nested tests are not supported anymore (fixes QuarkusTest: consider removing the test profile support for @Nested tests #45349).
• Changes the order of some tests; tests should not depend on execution order if they don't set one explicitly, so this shouldn't cause customer failures, but it did cause failures for us (see, for example, Tests in opentelemetry-reactive project are sensitive to execution order #45955 and Improve MetricsTest reliability #45960)
• Dev services start in the discovery phase (Dev Services should be started during startup, not augmentation #45785, we know how to fix this quickly)
• Maven surefire.rerunFailingTestsCount option does not work in the case where there are test profiles or resources, unless the test order falls such that the failing test uses the last profile/resource (Maven surefire.rerunFailingTestsCount option does not work in the case where there are test profiles or resources, unless the test order falls such that the failing test uses the last profile/resource #46048, harder to fix)
• Increased memory footprint running tests. For suites using multiple profiles and resources, more metaspace may be needed.

What problem is this solving?

We see a lot of problems caused by the fact that we load test classes with the deployment classloader, and then intercept the execution and reload the classes with the runtime classloader. Although the new test is loaded with the runtime classloader, its arguments are still loaded with the system classloader. To work around that we sometimes need to clone the arguments by serializing and de-serializing. This was always brittle and no longer worked at all on Java 17+ (until #40601 fixed that). We also see issues because parts of the test infrastructure see the 'wrong' instance of the class. See, for example, quarkiverse/quarkus-pact#73 and #22611.

We have several feature raised against the JUnit team to allow us more control over classloading. The first of this features was introduced in JUnit 5.10, and allows an interceptor to be registered before any tests are launched. This interceptor can set a thread context classloader, which is then used by JUnit to load tests.

My experiments with this feature were thoroughly disappointing. It turns out, setting a TCCL early in the test lifecycle doesn't really help us, because we overwrite our 'early' TCCL with other TCCLs later in the test lifecycle. The following diagram shows some of the places we set the TCCL.

Source: https://excalidraw.com/#json=HFPHIKx8wv0iiyXgNhAzw,8IlEmPcMRvm9pfCGShdClQ

What if we just used one of the existing interception points to set the 'right' classloader, before tests are loaded? If the tests were loaded with our preferred classloader, we wouldn’t need to intercept the factory. Loading the tests with the runtime classloader needs us to move some of our app initialisation earlier in the lifecycle, but I don't think there's any fundamental barrier to this. (We would have had to do this with a solution based on the new JUnit Launcher Interceptor anyway.)

The logic for starting Quarkus needs to be in the test discovery phase, rather than in the extension. This allows us to create the runtime classloader before the test is loaded. The JUnitTest runner already knows about the Quarkus Extension, so it’s only a small extra bit of knowledge to do some of the startup actions.

This only gets us part of the way, though. @stuartwdouglas raised the point that if we have to set only a single classloader, that's not very flexible, because we have a runtime classloader for each test profile. A Quarkus test run doesn't just use one classloader, it uses several. Every resource/unique profile triggers an app relaunch, which means a new classloader. What I've done to handle this is create a FacadeClassLoader. It takes the classloading requests, and then either routes them on to the quarkus application (for vanilla @QuarkusTests), or, if there's a profile/resource, it makes a new app + classloader and sends the request to that.

What we used to before was load a throwaway copy of the the test, pass it to JUnit discovery, let JUnit launch it, and then intercept the execution, figure out what profiles+resources the test declares, create a quarkus app with that information, start the quarkus app, reload the test with the runtime classloader of the quarkus app (and clone its parameters), and execute the test.

The new model is load a throwaway copy of the the test, figure out what profiles+resources the test declares, create a quarkus app with that information, reload the test with the runtime classloader of the quarkus app, pass the ‘right’ class to JUnit discovery, let JUnit launch it, and then intercept the execution, start the quarkus app, and execute the test.

One fundamental limitation of "load tests with the classloader used to execute them" is that a single test cannot run with multiple classloaders, which means it cannot support multiple profiles. We know some people do use this feature, but we also know there have been suggestions that we drop support for it, since it is complex to support (#45349). There is an easy workaround, which is to use one test per profile.

Thoughts on serialization and cloning

A big initial goal of this PR was to get rid of the xstream serialization, since it didn't work on Java 17+. #40601 fixes this issue by switching to use the JBoss marshaller for serialization. Does that mean this work item isn't needed any more? No, although it does mean its benefits are smaller. Here's why it's still useful:

• Even with the JBoss serializer, higher-level test infrastructure (such as @TestTemplate) does not see Quarkus bytecode transformations done by extensions
• Although the JBoss serializer works a lot better than xstream with the Java 17 access restrictions (as in, it works), serialization may continue to be a challenge going forward. See https://bugs.openjdk.org/browse/JDK-8164908 for some context. Most serializers use sun.misc.unsafe, but unsafe is shrinking. It seems certain the JDK team will have to come up with some solution and API to open up access for serializers, but the final design could have security implications (perhaps opening up access in a blanket way), or performance implications (reflection fun), or user experience implications (a need to manually set flags such as --enable-serialization?). If we can avoid serialization, we avoid all that.

Todo before this merges

• Start investigating parallel tests
• Start looking at classloader leaks
• Find Holly's flaky test reproducer that she lost somewhere in a branch
• Fix the ecosystem CI for pact

Todo after this merges

• Monitor ecosystem CI for new failures; we do not have coverage of every code path in our current suite (for example, the tests added in Continuous Testing: add support for build system like test selection #46389 broke with this change, by going down an uncovered code path)
• Address the 'do not start dev services in augmentation' issue
• Removal of all dead code in QuarkusTestExtension
• Consolidation of duplicated code between AppMakerHelper and TestSupport
• More automated tests (will come in a PR that goes in first)
• Tests for interaction with QuarkusProdModeTest, particularly for the tests in Add tests which exercise more complex JUnit extensions #35124
• Continued improvements to fix hacky config usage in existing tests (such as relying on mutable system properties, etc)
• Consolidation + streamlining of logic in the test order (unify around a key-based approach?)
• Adding tests based on various outstanding issues which do not yet have reproducers in our test suite

[The-Funk]

Still watching this one. I've created a minimal not-working-example to see if this fixes it. :)

[holly-cummins]

Still watching this one. I've created a minimal not-working-example to see if this fixes it. :)

Still going on it ... :)

The profile support in normal mode turned out to be a bit thorny, so looking at that now. I haven't checked for a while, but at one point I had reproducers for three of the test-classloading-related defects. One was passing, but two (annoyingly) were failing. If your reproducer is shareable, I'm happy to take it and include it in what I'm checking, or to add it into the test suites, if it's a gap in what we're testing now. (It'd be worth checking what I've added in #35124 to see if one of those covers your scenario, too.)

[geoand]

cc @aloubyansky

[geoand]

Doesn't really work, in what sense?

[holly-cummins]

Good question. That comment is so old I wasn't entirely sure myself, when I was reviewing the TODOs. :)

I think what it's referring to is that at an early stage of development I had good ideas about re-using more of the curated application + base classloaders between different applications. That is, if I decided we needed to restart, instead of throwing everything out, I'd do a light tidy and then re-use the lower levels of the classloader stack. That might still be a good thing to do, and it would reduce the memory footprint of the new 'load everything upfront' pattern. If we can do restarts for continuous testing we should be able to achieve some reuse in normal mode testing. But I never made it work.

So I think that tidy() method is a vestige of my attempts to re-use stuff. The question is whether it's now totally pointless because cleanup happens elsewhere, or whether it's still a useful part of the cleanup that we do between restarts. I'll inspect the code and try and work it out.

[geoand]

🙏🏽

[holly-cummins]

@geoand , FYI I've just spotted that these changes would break the JBeret Ecosystem CI. I think I know what the fix is, but I'll need to make an update. Hopefully it won't be a big update, just a tweak to the quarkus test detection logic to catch whatever edge case is breaking it in the JBeret tests.

[geoand]

Thanks for the heads up!

[geoand]

We need to load all our test classes in one go, during the discovery phase

Is this a current JUnit limitation?

[holly-cummins]

It is, to the best of my knowledge. I banged my head against it for a while, but I don't see a solution that's not writing a new engine, or doing a pre-test-reload. Pre-test-reload sounds appealing, but it's basically the existing approach. It means JUnit doesn't see our modified test code, because if we reload post-discovery, it's too late for some JUnit functionality.

[geoand]

Got it thanks! Any pointers at where I can place debug breakpoints to see this in action?
I would like to understand the current limitations so after we get this in, we can go back to the JUnit 5 people with a details on what we use so far and why it's not enough.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 32496bf.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

---

Flaky tests - Develocity

⚙️ JVM Tests - JDK 17

📦 extensions/smallrye-reactive-messaging-kafka/deployment

✖ io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingWorkingAppPropsTestCase.testContinuousTestingScenario3 - History

• io.quarkus.builder.BuildException: Build failure: Build failed due to errors [error]: Build step io.quarkus.apicurio.registry.devservice.DevServicesApicurioRegistryProcessor\#startApicurioRegistryDevService threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/apicurio/apicurio-registry-mem:2.4.2.Final at io.quarkus.apicurio.registry.devservice.DevServicesApicurioRegistryProcessor.startApicurioRegistryDevService(DevServicesApicurioRegistryProcessor.java:90) at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732) at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856) - java.lang.RuntimeException

java.lang.RuntimeException: 
io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.apicurio.registry.devservice.DevServicesApicurioRegistryProcessor#startApicurioRegistryDevService threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/apicurio/apicurio-registry-mem:2.4.2.Final
	at io.quarkus.apicurio.registry.devservice.DevServicesApicurioRegistryProcessor.startApicurioRegistryDevService(DevServicesApicurioRegistryProcessor.java:90)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:255)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)

---

⚙️ JVM Tests - JDK 17 Windows

📦 extensions/quartz/deployment

✖ io.quarkus.quartz.test.timezone.TriggerPrevFireTimeZoneTest.testScheduledJobs - History

• expected: <2025-03-11T11:16:31Z> but was: <2025-03-11T11:16:32Z> - org.opentest4j.AssertionFailedError

org.opentest4j.AssertionFailedError: expected: <2025-03-11T11:16:31Z> but was: <2025-03-11T11:16:32Z>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
	at io.quarkus.quartz.test.timezone.TriggerPrevFireTimeZoneTest.testScheduledJobs(TriggerPrevFireTimeZoneTest.java:71)