Toolchain downloads frequently failing #7492
Assignees
Labels
Comments
cpovirk
added
P3
|
Also consider trying to ensure that you are leveraging the hosted tool cache which avoids downloading if you use one of the pre-installed jdks (temurin 8, 11, 17, 21). You can also use composite github actions for reuse, optionally across repositories, if you find a common setup is helpful (e.g. I wrangle all of my jdk versions in this action) |
|
Slightly different stack trace today: |
copybara-service bot
pushed a commit
that referenced
this issue
Jan 2, 2025
…=21` to set the Java version for tests, and do so in CI. (The property value still _defaults_ to the JDK that you use to run Maven, so passing the flag is not _necessary_.) This lets us use whatever Java version we want\[1\] for executing Maven itself—and thus for executing any plugins that aren't toolchain-aware. (I even hacked up our toolchain configuration to run tests under Java 24, and it worked.) I've changed our CI to do that, too, by always using Java 23. (This makes the explicit choice of Java 23 for _javac_ redundant under CI, as well as on our Google workstations. However, it can be useful for anyone who builds Guava locally while having an older JDK as the default version.) Notably, that means that we can use a new version (like Java 23) for running the GWT plugin, which lets us upgrade to GWT 2.12.1 (cl/711417161), which uses an Eclipse compiler with bytecode version 55 (Java 11)\[2\]. (This means that users who build `guava-gwt` locally with Java 8 will need to [upgrade their build JDK to at least Java 11](https://github.com/google/guava/issues/6549)—or skip building `guava-gwt` if they don't need it.) That GWT upgrade is in turn necessary in order for us to be able to use use [JSpecify annotations](jspecify/jspecify#239) from our GWT artifact. In theory, this could also prepare us for further changes, like more easily running tests under multiple versions during our release process (in addition to how we already do so for CI), though there are already other ways of doing that by hacking up our release script or `pom.xml`. And I'm not sure that I'd really want to do that, anyway. (See also some notes that I've added to cl/639128020.) I also did some related cleanup: - Our `--no-module-directories` configuration probably became Just Wrong back in cl/655647768: It should be based on the version that we use to build Javadoc (which is currently always 11), not the version that Maven is running under. It probably mostly doesn't matter because we probably mostly don't try to run Javadoc under other versions, but I'd like to be able to do so, so this is a step forward. - `-Djava.security.manager=allow` is not necessary now that we've cleaned up our usages of `SecurityManager`. (And that's fortunate, as it seems difficult to [active a profile based on a range of values of an arbitrary "my test JRE version" property](https://issues.apache.org/jira/browse/MNG-3328), as opposed to activate it based on the JDK that Maven is running under.) I expect that this CL is going to fail Gradle integration testing because that testing (when last I checked) required Java 11. I'll try to figure that out once I see the full CI results, which may well reveal bigger problems. Fixes #7492 (by making `setup-java` download all the toolchains that we need) \[1\] I mean, I suspect that Maven support lags new JDKs somewhat, and I've seen problems with other versions (as documented in cl/488902996). But at least we don't have to use the same Java version for executing Maven itself as we do for executing tests. \[2\] That was causing this error: ``` [INFO] --- gwt:2.10.0:compile (gwt-compile) @ guava-gwt --- Error: [ERROR] Unexpected internal compiler error [INFO] java.lang.UnsupportedClassVersionError: org/eclipse/jdt/internal/compiler/classfmt/ClassFormatException has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 [INFO] at java.lang.ClassLoader.defineClass1(Native Method) [INFO] at java.lang.ClassLoader.defineClass(ClassLoader.java:757) [INFO] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [INFO] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) [INFO] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) [INFO] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) [INFO] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) [INFO] at java.security.AccessController.doPrivileged(Native Method) [INFO] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) [INFO] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) [INFO] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [INFO] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) [INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.doBuildFrom(CompilationStateBuilder.java:488) [INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.buildFrom(CompilationStateBuilder.java:464) [INFO] at com.google.gwt.dev.cfg.ModuleDef.getCompilationState(ModuleDef.java:426) [INFO] at com.google.gwt.dev.Precompile.validate(Precompile.java:145) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:184) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:143) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:132) [INFO] at com.google.gwt.dev.Compiler$1.run(Compiler.java:110) [INFO] at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:55) [INFO] at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:50) [INFO] at com.google.gwt.dev.Compiler.main(Compiler.java:113) ``` RELNOTES=n/a PiperOrigin-RevId: 711434202
copybara-service bot
pushed a commit
that referenced
this issue
Jan 2, 2025
…=21` to set the Java version for tests, and do so in CI. (The property value still _defaults_ to the JDK that you use to run Maven, so passing the flag is not _necessary_.) This lets us use whatever Java version we want\[1\] for executing Maven itself—and thus for executing any plugins that aren't toolchain-aware. (I even hacked up our toolchain configuration to run tests under Java 24, and it worked.) I've changed our CI to do that, too, by always using Java 23. (This makes the explicit choice of Java 23 for _javac_ redundant under CI, as well as on our Google workstations. However, it can be useful for anyone who builds Guava locally while having an older JDK as the default version.) Notably, that means that we can use a new version (like Java 23) for running the GWT plugin, which lets us upgrade to GWT 2.12.1 (cl/711417161), which uses an Eclipse compiler with bytecode version 55 (Java 11)\[2\]. (This means that users who build `guava-gwt` locally with Java 8 will need to [upgrade their build JDK to at least Java 11](https://github.com/google/guava/issues/6549)—or skip building `guava-gwt` if they don't need it.) That GWT upgrade is in turn necessary in order for us to be able to use use [JSpecify annotations](jspecify/jspecify#239) from our GWT artifact. In theory, this could also prepare us for further changes, like more easily running tests under multiple versions during our release process (in addition to how we already do so for CI), though there are already other ways of doing that by hacking up our release script or `pom.xml`. And I'm not sure that I'd really want to do that, anyway. (See also some notes that I've added to cl/639128020.) I also did some related cleanup: - Our `--no-module-directories` configuration probably became Just Wrong back in cl/655647768: It should be based on the version that we use to build Javadoc (which is currently always 11), not the version that Maven is running under. It probably mostly doesn't matter because we probably mostly don't try to run Javadoc under other versions, but I'd like to be able to do so, so this is a step forward. - `-Djava.security.manager=allow` is not necessary now that we've cleaned up our usages of `SecurityManager`. (And that's fortunate, as it seems difficult to [active a profile based on a range of values of an arbitrary "my test JRE version" property](https://issues.apache.org/jira/browse/MNG-3328), as opposed to activate it based on the JDK that Maven is running under.) I expect that this CL is going to fail Gradle integration testing because that testing (when last I checked) required Java 11. I'll try to figure that out once I see the full CI results, which may well reveal bigger problems. Fixes #7492 (by making `setup-java` download all the toolchains that we need) \[1\] I mean, I suspect that Maven support lags new JDKs somewhat, and I've seen problems with other versions (as documented in cl/488902996). But at least we don't have to use the same Java version for executing Maven itself as we do for executing tests. \[2\] That was causing this error: ``` [INFO] --- gwt:2.10.0:compile (gwt-compile) @ guava-gwt --- Error: [ERROR] Unexpected internal compiler error [INFO] java.lang.UnsupportedClassVersionError: org/eclipse/jdt/internal/compiler/classfmt/ClassFormatException has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 [INFO] at java.lang.ClassLoader.defineClass1(Native Method) [INFO] at java.lang.ClassLoader.defineClass(ClassLoader.java:757) [INFO] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [INFO] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) [INFO] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) [INFO] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) [INFO] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) [INFO] at java.security.AccessController.doPrivileged(Native Method) [INFO] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) [INFO] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) [INFO] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [INFO] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) [INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.doBuildFrom(CompilationStateBuilder.java:488) [INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.buildFrom(CompilationStateBuilder.java:464) [INFO] at com.google.gwt.dev.cfg.ModuleDef.getCompilationState(ModuleDef.java:426) [INFO] at com.google.gwt.dev.Precompile.validate(Precompile.java:145) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:184) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:143) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:132) [INFO] at com.google.gwt.dev.Compiler$1.run(Compiler.java:110) [INFO] at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:55) [INFO] at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:50) [INFO] at com.google.gwt.dev.Compiler.main(Compiler.java:113) ``` RELNOTES=n/a PiperOrigin-RevId: 711434202
copybara-service bot
pushed a commit
that referenced
this issue
Jan 2, 2025
…=21` to set the Java version for tests, and do so in CI. (The property value still _defaults_ to the JDK that you use to run Maven, so passing the flag is not _necessary_.) This lets us use whatever Java version we want\[1\] for executing Maven itself—and thus for executing any plugins that aren't toolchain-aware. (I even hacked up our toolchain configuration to run tests under Java 24, and it worked.) I've changed our CI to do that, too, by always using Java 23. (This makes the explicit choice of Java 23 for _javac_ redundant under CI, as well as on our Google workstations. However, it can be useful for anyone who builds Guava locally while having an older JDK as the default version.) Notably, that means that we can use a new version (like Java 23) for running the GWT plugin, which lets us upgrade to GWT 2.12.1 (cl/711417161), which uses an Eclipse compiler with bytecode version 55 (Java 11)\[2\]. (This means that users who build `guava-gwt` locally with Java 8 will need to [upgrade their build JDK to at least Java 11](https://github.com/google/guava/issues/6549)—or skip building `guava-gwt` if they don't need it.) That GWT upgrade is in turn necessary in order for us to be able to use use [JSpecify annotations](jspecify/jspecify#239) from our GWT artifact. In theory, this could also prepare us for further changes, like more easily running tests under multiple versions during our release process (in addition to how we already do so for CI), though there are already other ways of doing that by hacking up our release script or `pom.xml`. And I'm not sure that I'd really want to do that, anyway. (See also some notes that I've added to cl/639128020.) I also did some related cleanup: - Our `--no-module-directories` configuration probably became Just Wrong back in cl/655647768: It should be based on the version that we use to build Javadoc (which is currently always 11), not the version that Maven is running under. It probably mostly doesn't matter because we probably mostly don't try to run Javadoc under other versions, but I'd like to be able to do so, so this is a step forward. - `-Djava.security.manager=allow` is not necessary now that we've cleaned up our usages of `SecurityManager`. (And that's fortunate, as it seems difficult to [active a profile based on a range of values of an arbitrary "my test JRE version" property](https://issues.apache.org/jira/browse/MNG-3328), as opposed to activate it based on the JDK that Maven is running under.) Fixes #7492 (by making `setup-java` download all the toolchains that we need) \[1\] I mean, I suspect that Maven support lags new JDKs somewhat, and I've seen problems with other versions (as documented in cl/488902996). But at least we don't have to use the same Java version for executing Maven itself as we do for executing tests. \[2\] That was causing this error: ``` [INFO] --- gwt:2.10.0:compile (gwt-compile) @ guava-gwt --- Error: [ERROR] Unexpected internal compiler error [INFO] java.lang.UnsupportedClassVersionError: org/eclipse/jdt/internal/compiler/classfmt/ClassFormatException has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 [INFO] at java.lang.ClassLoader.defineClass1(Native Method) [INFO] at java.lang.ClassLoader.defineClass(ClassLoader.java:757) [INFO] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [INFO] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) [INFO] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) [INFO] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) [INFO] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) [INFO] at java.security.AccessController.doPrivileged(Native Method) [INFO] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) [INFO] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) [INFO] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [INFO] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) [INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.doBuildFrom(CompilationStateBuilder.java:488) [INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.buildFrom(CompilationStateBuilder.java:464) [INFO] at com.google.gwt.dev.cfg.ModuleDef.getCompilationState(ModuleDef.java:426) [INFO] at com.google.gwt.dev.Precompile.validate(Precompile.java:145) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:184) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:143) [INFO] at com.google.gwt.dev.Compiler.compile(Compiler.java:132) [INFO] at com.google.gwt.dev.Compiler$1.run(Compiler.java:110) [INFO] at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:55) [INFO] at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:50) [INFO] at com.google.gwt.dev.Compiler.main(Compiler.java:113) ``` RELNOTES=n/a PiperOrigin-RevId: 711434202
|
It is possible that we'd accidentally regress that in the future. Probably what we should do is disable |
copybara-service bot
pushed a commit
that referenced
this issue
Jan 2, 2025
- Use Temurin over Zulu. Per [advice](#7492 (comment)) from @ben-manes, LTS versions of Temurin [come pre-installed](https://github.com/actions/setup-java/blob/main/docs/advanced-usage.md#Hosted-Tool-Cache). - But be willing to run with any JDK of the appropriate version that's available, rather than requiring Zulu. (Not that this makes a huge difference, given that we'll still *download* Temurin JDKs. I didn't think this through much.) - Make the test jobs fail [if they need to download a needed JDK](https://github.com/linux-china/toolchains-maven-plugin#how-to-skip-toolchains-maven-plugin-on-cicd-platform) because we didn't already put that one in place with `setup-java`. (Our other CI jobs, which publish snapshots to Sonatype and regenerate snapshot Javadoc, will continue to tolerate missing JDKs. We could change them, too, by making their scripts pass through options to identify the run as a CI run.) This should help us keep our versions in sync, and it should prevent us from silently falling back to slower downloads, per b/334979149. - Use Java 21 to generate Javadoc (as that's the best we can do now without [further trouble](#6790 (comment))). I've made sure to make it available through `setup-java`, including for generating Sonatype snapshots, which [do include Javadoc](https://oss.sonatype.org/content/repositories/snapshots/com/google/guava/guava/HEAD-jre-SNAPSHOT/). Plus, use Java 23 for the Maven VM itself there (so as to standardize on the same version as we already use for the Maven VM itself during our test workflows as of cl/711476575). - Fix a few Javadoc warnings, leaving behind only [a handful that we can't easily do anything about](#6790 (comment)). - Download Java 11 only when it is required. Currently, that's for JDiff and for our Gradle integration tests. There is a decent change that this CL will break something, possibly not until we run the snapshot steps after submission... :) RELNOTES=n/a PiperOrigin-RevId: 711520051
copybara-service bot
pushed a commit
that referenced
this issue
Jan 3, 2025
- Use Temurin over Zulu. Per [advice](#7492 (comment)) from @ben-manes, LTS versions of Temurin [come pre-installed](https://github.com/actions/setup-java/blob/main/docs/advanced-usage.md#Hosted-Tool-Cache). - But be willing to run with any JDK of the appropriate version that's available, rather than requiring Zulu. (Not that this makes a huge difference, given that we'll still *download* Temurin JDKs. I didn't think this through much. I guess you can now at least avoiding downloading a Temurin JDK if you make sure you already have enough JDKs registered and you pass `-Dtoolchain.skip`.) - Make the test jobs fail [if they need to download a needed JDK](https://github.com/linux-china/toolchains-maven-plugin#how-to-skip-toolchains-maven-plugin-on-cicd-platform) because we didn't already put that one in place with `setup-java`. (Our other CI jobs, which publish snapshots to Sonatype and regenerate snapshot Javadoc, will continue to tolerate missing JDKs. We could change them, too, by making their scripts pass through options to identify the run as a CI run.) This should help us keep our versions in sync, and it should prevent us from silently falling back to slower downloads, per b/334979149. - Use Java 21 to generate Javadoc (as that's the best we can do now without [further trouble](#6790 (comment))). I've made sure to make Java 21 available through `setup-java`, including for generating Sonatype snapshots, which [do include Javadoc](https://oss.sonatype.org/content/repositories/snapshots/com/google/guava/guava/HEAD-jre-SNAPSHOT/). Plus, use Java 23 for the Maven VM itself there (so as to standardize on the same version as we already use for the Maven VM itself during our test workflows as of cl/711476575). - Fix a few Javadoc warnings, leaving behind only [a handful that we can't easily do anything about](#6790 (comment)). - Download Java 11 only when it is required. Currently, that's for JDiff and for our Gradle integration tests. There is a decent chance that this CL will break something, possibly not until we run the snapshot steps after submission... :) RELNOTES=n/a PiperOrigin-RevId: 711746683
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
I've had multiple PRs recently that have failed testing because of trouble downloading a JDK. Here's the most recent example, which downloaded 11 but failed on 23, but note that I think I've seen failures for 11 and have definitely seen a failure for 8:
(Aside: GitHub is inserting Markdown when I copy and paste directly from the Actions results, possibly to link to anchors in that page? Not that that would work inside the code block that I'm pasting into. I ended up pasting into a plain-text window first and then copying and pasting again.)
The automatic toolchain downloads are nice, but it would probably be more reliable for CI to download all the toolchains we need with setup-java. (That would likely also be faster, as noted in internal bug 334979149.) The trick would be ensuring that we still use the toolchain that we want for (a) testing and (b, possibly less important) running Maven itself. Error Prone has this worked out by being careful about the order in which they list their downloaded toolchains and then (I'm guessing) by configuring
maven-surefire-pluginmore directly than we do.(We could still keep the toolchain plugin for local builds. It's much nicer for users than if they have to download various toolchains themselves.)
The text was updated successfully, but these errors were encountered: