Update the mx suite parsing to work with sources of GraalVM version 23.1.0

[entlicher]

The mx project of recent versions of GraalVM can not be successfully parsed. There is an issue in graal/sdk suite.
This change adapts to the mx suite form and a test is updated to work with the new unchained project structure.

[neilcsmith-net]

NB21 or master?

[entlicher]

Well, master. It's too late for 21, I guess, right?

[mbien]

restarting CI with MX [ci] enable "build tools" tests tests active

[neilcsmith-net]

Not yet. There's at least one pending fix marked for NB21, so if you want this fix in there target delivery and it can be reviewed on that basis.

[neilcsmith-net]

This will need rebasing and force pushing on head of delivery.

[entlicher]

Yes, sure, I'm sorry, will do ASAP...

[neilcsmith-net]

No problem, that's normal. Once it's done, feel free to remove the do not merge label.

[JaroslavTulach]

I also realized the MX project support is broken and I created

• NbLaunchDelegate fix and test multi line texts in suite.py #7011

[mbien]

java.mx.project tests are currently failing with:

JDK 11:

...
 test-preinit:
     [exec] JAVA_HOME: /opt/hostedtoolcache/Java_Zulu_jdk/11.0.22-7/x64
     [exec] 110 unsatisfied dependencies were removed from build (use -v to list them)
     [exec] 4 non-default dependencies were removed from build (use -v to list them, mx build --all to build them)
     [exec] Archiving GCC_NINJA_TOOLCHAIN... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/mx/mxbuild/jdk11/dists/gcc-ninja-toolchain.tar does not exist]
     [exec] Downloading NINJA_SYNTAX from ['https://lafo.ssw.uni-linz.ac.at/pub/graal-external-deps/ninja_syntax-1.7.2.tar.gz']
     [exec] Compiling com.oracle.mxtool.compilerserver with javac(JDK 11)... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/mx/mxbuild/jdk11/com.oracle.mxtool.compilerserver/bin/com/oracle/mxtool/compilerserver/JavacDaemon.class does not exist]
     [exec] Downloading NINJA from ['https://lafo.ssw.uni-linz.ac.at/pub/graal-external-deps/ninja-1.10.2-linux-amd64.zip']
     [exec] Downloading ASM_9.5 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm/9.5/asm-9.5.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm/9.5/asm-9.5.jar']
     [exec] Compiling com.oracle.mxtool.checkcopy with javac-daemon(JDK 11)... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/mx/mxbuild/jdk11/com.oracle.mxtool.checkcopy/bin/com/oracle/mxtool/checkcopy/CheckCopyright.class does not exist]
     [exec] Downloading sources ASM_9.5 from ['https://repo1.maven.org/maven2/org/ow2/asm/asm/9.5/asm-9.5-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/ow2/asm/asm/9.5/asm-9.5-sources.jar']
     [exec] Downloading JMH_GENERATOR_ANNPROCESS_1_21 from ['https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21.jar', 'https://search.maven.org/remotecontent?filepath=org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21.jar']
     [exec] Downloading JOPTSIMPLE_4_6 from ['https://repo1.maven.org/maven2/net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.jar', 'https://search.maven.org/remotecontent?filepath=net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.jar']
     [exec] Downloading sources JMH_GENERATOR_ANNPROCESS_1_21 from ['https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/openjdk/jmh/jmh-generator-annprocess/1.21/jmh-generator-annprocess-1.21-sources.jar']
     [exec] Downloading sources JOPTSIMPLE_4_6 from ['https://repo1.maven.org/maven2/net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6-sources.jar', 'https://search.maven.org/remotecontent?filepath=net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6-sources.jar']
     [exec] Downloading COMMONS_MATH3_3_2 from ['https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar', 'https://search.maven.org/remotecontent?filepath=org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar']
     [exec] Downloading sources COMMONS_MATH3_3_2 from ['https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/apache/commons/commons-math3/3.2/commons-math3-3.2-sources.jar']
     [exec] Downloading HAMCREST from ['https://repo1.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar', 'https://search.maven.org/remotecontent?filepath=org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar']
     [exec] Compiling com.oracle.mxtool.junit.jdk9 with javac-daemon(JDK 11)... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/mx/mxbuild/jdk11/com.oracle.mxtool.junit.jdk9/bin/com/oracle/mxtool/junit/ModuleSupport.class does not exist]
     [exec] Downloading LLVM_ORG from ['https://lafo.ssw.uni-linz.ac.at/pub/llvm-org/llvm-llvmorg-16.0.1-4-gad8c248269-bg39f02d0d6a-linux-amd64.tar.gz']
     [exec] Compiling org.graalvm.word with javac-daemon(JDK 11)... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/graal/sdk/mxbuild/jdk11/org.graalvm.word/bin/org/graalvm/word/PointerBase.class does not exist]
     [exec] Downloading sources HAMCREST from ['https://repo1.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar', 'https://search.maven.org/remotecontent?filepath=org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar']
     [exec] Downloading LLVM_ORG_COMPILER_RT_LINUX from ['https://lafo.ssw.uni-linz.ac.at/pub/llvm-org/compiler-rt-llvmorg-16.0.1-4-gad8c248269-bg39f02d0d6a-linux-amd64.tar.gz']
     [exec] Compiling com.oracle.mxtool.webserver with javac-daemon(JDK 11)... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/mx/mxbuild/jdk11/com.oracle.mxtool.webserver/bin/com/oracle/mxtool/webserver/WebServer.class does not exist]
     [exec] Archiving TRUFFLE_NFI_GRAALVM_SUPPORT... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/graal/truffle/mxbuild/jdk11/dists/truffle-nfi-graalvm-support.tar does not exist]
     [exec] Archiving TRUFFLE_ICU4J_GRAALVM_SUPPORT... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/graal/truffle/mxbuild/jdk11/dists/truffle-icu4j-graalvm-support.tar does not exist]
     [exec] Compiling org.graalvm.options with javac-daemon(JDK 11)... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/graal/sdk/mxbuild/jdk11/org.graalvm.options/bin/org/graalvm/options/OptionStability.class does not exist]
     [exec] Building polyglot.image-bash... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/graal/sdk/mxbuild/linux-amd64/polyglot.image-bash/polyglot does not exist]
     [exec] Archiving TRUFFLE_JSON_GRAALVM_SUPPORT... [/home/runner/work/netbeans/netbeans/java/java.mx.project/test/unit/data/graal/truffle/mxbuild/jdk11/dists/truffle-json-graalvm-support.tar does not exist]
     [exec] Building polyglot.image-bash: Failed due to error: 1
     [exec] cannot resolve LAUNCHER_COMMON as a dependency defined by sdk

BUILD FAILED
/home/runner/work/netbeans/netbeans/java/java.mx.project/build.xml:51: exec returned: 1

Total time: 13 seconds

JDK 17/21 have failures in:

testRootsForAnSdkJar: org.netbeans.modules.java.mx.project.SdkSuiteTest

I believe the JDK 11 failure is expected since current graal versions are based on later JDKs.

[mbien]

@entlicher on what JDK versions should the mx project tests now run?

if its JDK 17 and 21 I suggest the following change:

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 8e28440..786f1bc 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -900,6 +900,12 @@
 #      - name: java/ant.grammar
 #        run: ant $OPTS -f java/ant.grammar test
 
+      # only JDK 17 and 21
+      - name: java/java.mx.project
+        if: ${{ matrix.java != '11' }}
+        continue-on-error: ${{ github.event_name != 'pull_request' }}
+        run: .github/retry.sh ant $OPTS -f java/java.mx.project test
+
       - name: Set up JDK 17 for JDK 21 incompatible tests
         if: ${{ matrix.java == '21' }}
         uses: actions/setup-java@v4
@@ -908,10 +914,6 @@
           distribution: ${{ env.default_java_distribution }}
 
       # TODO fix JDK 21 incompatibilites
-      - name: java/java.mx.project
-        continue-on-error: ${{ github.event_name != 'pull_request' }}
-        run: .github/retry.sh ant $OPTS -f java/java.mx.project test
-
       - name: java/gradle.java
         run: .github/retry.sh ant $OPTS -f java/gradle.java test
 

NetBeans 22 will be JDK 17+ anyway, so I can clean this up later when we update the test matrix.

[entlicher]

Many thanks for the suggested change!

[mbien]

i enabled all tests since I had the feeling this might cause more indirect issues.

The "Build all Tests" step fails now since it was used as validator for JDK 11 compatibility.

We could continue the same path and bump that job to 17 too. This is probably not ideal to do in the week before release (given that NB 21 is supposed to build/run on JDK 11 while NB 22 will be 17+) but on the other hand it would be nice if NB 21 would be able to be used for mx projects.

This is also just about building tests, so I would be ok with this early bump - other opinions welcome of course.

relevant line would be here:

netbeans/.github/workflows/main.yml

Line 341 in 1c19e60

java: [ '11' ]

lets hope it builds on JDK 17 ;)

the JDK 21 java/java.mx.project build problem can be probably resolved via @SuppressWarnings or by removing -Werror:

netbeans/java/java.mx.project/nbproject/project.properties

Line 20 in 1c19e60

javac.compilerargs=-Xlint -Xlint:-serial -Xlint:-processing -Xlint:-options -Werror

[jtulach]

Feel free to pick whatever you find useful in https://github.com/apache/netbeans/pull/7011/files#diff-d09b7dca37327eaec02560e4951f9024cf208ad0eacd20755dde663b9b15c612R46 - for example this test. Or I can integrate the test later, when this PR merges.

[entlicher]

Thanks Jaroslav! I've reverted the version of graal/sdk back to 22.3.0, so that it can compile with JDK 11. I do not feel comfortable changing the version of JDK for tests compilation when the rest of the NB still supports JDK 11. It passes fine with JDK 21 now, we should be able to use graal/sdk version 23.1 in master, after we remove the dependency on JDK 11.

[neilcsmith-net]

Thanks! Is the test now stable enough to drop the retry behaviour in the workflow file, or is that change unintentional in here?

[entlicher]

Yes, the test is stable. At least, I've run it locally several times without issues.

[entlicher]

I've removed the .github/retry.sh already, if this is what do you mean.

[mbien]

looks good from my side, please consider the retry script and make sure you read @JaroslavTulach comment #7013 (comment)

I see you reverted the graal version bump -> I think we can bump it again in a week or so for NB 22 when CI is ready for JDK 17.

Thanks a lot!

[mbien]

for the interested: #7019 is now green. If it makes it into master, it should be easily possible to update graal tests to JDK 17 equivalent graal versions - which is great since graal 17 would be still a supported release.

The graal job is still using an EOL graal sdk, I haven't touched that since #6369