Integration tests using postgresql fail to compile natively with Java 17

[zakkak]

Describe the bug

Native compilation of integration tests using postgresql fails with GraalVM CE 21.3 Java17 (quay.io/quarkus/ubi-quarkus-native-image:21.3-java17)

Namely the following tests fail to build:

• main
• jpa-db2
• jpa-mysql
• jpa-postgresql
• jpa-postgresql-withxml
• hibernate-orm-tenancy
• vault-app
• smallrye-opentracing

See https://github.com/graalvm/mandrel/actions/runs/1445890803

Expected behavior

Tests should build and run.

Actual behavior

Tests fail to build with:

Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: sun.security.krb5.Credentials. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Detailed message:
Trace: 
	at parsing org.postgresql.util.internal.Unsafe.credentialCacheExists(Unsafe.java:18)
Call path from entry point to org.postgresql.util.internal.Unsafe.credentialCacheExists(): 
	at org.postgresql.util.internal.Unsafe.credentialCacheExists(Unsafe.java:17)
	at org.postgresql.core.v3.ConnectionFactoryImpl.credentialCacheExists(ConnectionFactoryImpl.java:413)
	at org.postgresql.core.v3.ConnectionFactoryImpl.enableGSSEncrypted(ConnectionFactoryImpl.java:430)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:245)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
	at org.postgresql.Driver.makeConnection(Driver.java:466)
	at org.postgresql.Driver.access$100(Driver.java:63)
	at org.postgresql.Driver$ConnectThread.run(Driver.java:376)
	at java.lang.Thread.run(Thread.java:833)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:596)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
	at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)
...

How to Reproduce?

./mvnw -Dnative -Dnative.surefire.skip -pl integration-tests/jpa-postgresql package -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-native-image:21.3-java17

Output of uname -a or ver

5.14.15-200.fc34.x86_64 #1 SMP Wed Oct 27 15:53:30 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (build 11.0.13+8, mixed mode)

GraalVM version (if different from Java)

OpenJDK 64-Bit Server VM GraalVM CE 21.3.0 (build 17.0.1+12-jvmci-21.3-b05, mixed mode, sharing)

Quarkus version or git rev

132d9ea

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)

Additional information

The issue is only reproducible on Java 17 and it seems to be resolved by passing:

-Dquarkus.native.additional-build-args="--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED"

Note that this is not a Quarkus issue. I am able to reproduce it using just GraalVM CE and https://github.com/zakkak/issue-reproducers/tree/krb5-credentials-failure-java17

[quarkus-bot]

/cc @Karm, @galderz

[jerboaa]

@zakkak tells me that the relevant code in the PG Java driver is: https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/util/internal/Unsafe.java

It's interesting that it's only surfacing during a native-image build. It seems to suggest it's a latent bug bound to surface in plain JVM mode when run on JDK 17 (which has stronger encapsulation enabled) if the relevant code path is being triggered. Apparently it isn't at this point with existing tests.

To fix this properly, I'd suggest to bring this to the attention of the PG driver developers (file an issue). Then the question is why they are using JDK-internal API for this use-case. It could be that that use-case may be supported via some already existing public JDK API. If there isn't and no equivalent API exists, an OpenJDK bug should get filed to add it. --add-exports should be used as a last resort.

[geoand]

I don't pretend to know what the driver is doing here (@Sanne perhaps knows), but could we not solve this with a simple substitution that returns false for the problematic method (perhaps even higher up the call stack)?

[zakkak]

but could we not solve this with a simple substitution that returns false for the problematic method?

I also don't know what the driver is doing. Since it's checking for the presence of a cache and it handles the case where the cache is not present your suggestion seems safe to me (as a work around). I am not sure what side effects that could have though...

Update: it looks like such a substitution could result in some configurations not working.

FWIW the internal api reference was added in pgjdbc/pgjdbc@705b660

[geoand]

Update: it looks like such a substitution could result in some configurations not working.

But does this happen in practice?

[Sanne]

encryption is commonly used so it would be sad to have to disable support for it. Would be good to know more about this "credentials caching" check specifically though, not sure what its implications are.

[indalaterre]

Same error here. I tried the build both on MacOS and CentOS.

Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: sun.security.krb5.Credentials. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Trace: 
        at parsing org.postgresql.util.internal.Unsafe.credentialCacheExists(Unsafe.java:18)
Call path from entry point to org.postgresql.util.internal.Unsafe.credentialCacheExists(): 
        at org.postgresql.util.internal.Unsafe.credentialCacheExists(Unsafe.java:17)
        at org.postgresql.core.v3.ConnectionFactoryImpl.credentialCacheExists(ConnectionFactoryImpl.java:413)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableGSSEncrypted(ConnectionFactoryImpl.java:430)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:245)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
        at org.postgresql.Driver.makeConnection(Driver.java:466)
        at org.postgresql.Driver.access$100(Driver.java:63)
        at org.postgresql.Driver$ConnectThread.run(Driver.java:376)
        at java.lang.Thread.run(Thread.java:833)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:596)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
        at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)

[Sanne]

I've figured out what is going on;

the "unresolved type" mentioned by the error message is the method sun.security.krb5.Credentials.acquireTGTFromCache, which belongs to the Java module java.security.jgss/sun.security.krb5.

Apparently in Java 17 this module is not exposed by default, so one needs to add an explicit export instruction to make this available to the pg client code.

The "normal" syntax for this would be to pass --add-exports java.security.jgss/sun.security.krb5=ALL-UNNAMED to the JVM; for native-image we need to express this as -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED (note the slightly different syntax)

So a valid workaround is to have

quarkus.native.additional-build-args=-J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED

but we should fix this issue by setting such flags automatically.