Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wildfly crashes when using "OpenTelemetry auto-instrumentation Java agent" with the Module "JMX Metric Insights" #11143

Closed
flamiau04 opened this issue Apr 16, 2024 · 0 comments · Fixed by #11151
Closed

Wildfly crashes when using "OpenTelemetry auto-instrumentation Java agent" with the Module "JMX Metric Insights" #11143

flamiau04 opened this issue Apr 16, 2024 · 0 comments · Fixed by #11151
Labels
bug Something isn't working needs triage New issue that requires triage

Comments

@flamiau04
Copy link

flamiau04 commented Apr 16, 2024 •
edited
Loading

Describe the bug

I successfully attach the OpenTelemetry auto-instrumentation Java agent to my wildfly application server. It collects metrics and sends the data to my collector.

But when I use the OTel-Java Agent "JMX Metric Insights" Module (by appending the option "-Dotel.jmx.target.system=wildfly" in the JAVA_OPTS environment variable), the application server crashes during startup.

For some application servers, a few predefined JMX rules are included with the OTel Java agent (JMX Metric Insights Plugin). By providing the option "-Dotel.jmx.target.system=wildfly" the predefined JMX rules for wildfly are executed. And this is exactly where a java stacktrace occurs and the application server crashes:

(...)
[otel.javaagent 2024-04-16 09:48:10:288 +0200] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.3.0                                               
[otel.javaagent 2024-04-16 09:48:12:487 +0200] [main] INFO io.opentelemetry.javaagent.shaded.instrumentation.jmx.yaml.RuleParser - wildfly: found 5 metric rules                                 
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager      
(...)
Caused by: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To u
se JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
(...)

Notice: I can also reproduce the bug by using my own JMX rules with the option "-Dotel.jmx.config".

Details about the module "JMX Metric Insight": https://opentelemetry.io/blog/2023/jmx-metric-insight/

Steps to reproduce

vi /data/wildfly-29.0.1.final/bin/standalone.conf + add this line to the end of the file:

JAVA_OPTS="$JAVA_OPTS -javaagent:/data/wildfly-29.0.1.Final/opentelemetry-javaagent-2.3.0.jar -Dotel.service.name=testapp -Dotel.traces.exporter=none -Dotel.metrics.exporter=none -Dotel.logs.exporter=none -Dotel.jmx.target.system=wildfly"

Start the wildfly application server from console:

/data/wildfly-29.0.1.final/bin/standalone.sh -b 0.0.0.0

I intentionally disabled all metric collections to make the java stacktrace error message more prominent in the application server logs.

The application server crashes when starting and only outputs the error message on the console (stdout) and not in the application server log file.

Expected behavior

I expect the Wildfly application server to start without errors even if the JMX Metric Insights module "-Dotel.jmx.target.system=wildfly" is enabled to collect jmx-metrics.

Actual behavior

[wildfly@server bin]$ ./standalone.sh -b 0.0.0.0                                                                                                                                                    
=========================================================================                                                                                                                                   
                                                                                                                                                                                                            
  JBoss Bootstrap Environment                                                                                                                                                                               
                                                                                                                                                                                                            
  JBOSS_HOME: /data/wildfly-29.0.1.Final                                                                                                                                                                    
                                                                                                                                                                                                            
  JAVA: /usr/lib/jvm/java-11-openjdk-11.0.22.0.7-2.el8.x86_64/bin/java                                                                                                                                      
                                                                                                                                                                                                            
  JAVA_OPTS:  -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -javaagent:/data/wildf
ly-hrm/opentelemetry-javaagent-2.3.0.jar -Dotel.service.name=testapp -Dotel.traces.exporter=none -Dotel.metrics.exporter=none -Dotel.logs.exporter=none -Dotel.jmx.target.system=wildfly  --add-exports=java
.desktop/sun.awt=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldaps=ALL-UNNAME
D --add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAM
ED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/ja
va.util.concurrent=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.naming/javax.naming=ALL-UNNAMED                                                                    
                                                                                                                                                                                                            
=========================================================================                                                                                                                                   
                                                                                                                                                                                                            
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended                                                                           
[otel.javaagent 2024-04-16 09:48:10:288 +0200] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.3.0                                                      
[otel.javaagent 2024-04-16 09:48:12:487 +0200] [main] INFO io.opentelemetry.javaagent.shaded.instrumentation.jmx.yaml.RuleParser - wildfly: found 5 metric rules                                            
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager                                                                                                                     
Apr 16, 2024 9:48:13 AM org.jboss.msc.service.ServiceContainerImpl <clinit>                                                                                                                                 
INFO: JBoss MSC version 1.5.1.Final                                                                                                                                                                         
Apr 16, 2024 9:48:13 AM org.jboss.threads.Version <clinit>                                                                                                                                                  
INFO: JBoss Threads version 2.4.0.Final                                                                                                                                                                     
Apr 16, 2024 9:48:13 AM org.jboss.as.server.ApplicationServerService start                                                                                                                                  
INFO: WFLYSRV0049: WildFly Full 29.0.1.Final (WildFly Core 21.1.1.Final) starting
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.wildfly.extension.elytron.SSLDefinitions (jar:file:/data/wildfly-29.0.1.Final/modules/system/layers/base/org/wildfly/extension/elytron/main/wildfly-elytron-integ$ation-21.1.1.Final.jar!/) to method com.sun.net.ssl.internal.ssl.Provider.isFIPS()
WARNING: Please consider reporting this to the maintainers of org.wildfly.extension.elytron.SSLDefinitions                                                                                                 
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Apr 16, 2024 9:48:15 AM org.wildfly.security.Version <clinit>
INFO: ELY00001: WildFly Elytron version 2.2.1.Final
Apr 16, 2024 9:48:15 AM org.jboss.as.controller.AbstractOperationContext executeStep
ERROR: WFLYCTL0013: Operation ("parallel-extension-add") failed - address: ([])
java.lang.RuntimeException: WFLYCTL0079: Failed initializing module org.jboss.as.logging
        at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:115)                                             
        at [email protected]//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1064)                                                          
        at [email protected]//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:785)                                                         
        at [email protected]//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:476)
        at [email protected]//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1455)
        at [email protected]//org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:523)
        at [email protected]//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:460)
        at [email protected]//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:431)
        at [email protected]//org.jboss.as.server.ServerService.boot(ServerService.java:466)
        at [email protected]//org.jboss.as.server.ServerService.boot(ServerService.java:408)
        at [email protected]//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:370)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has no
t be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:107)
        ... 11 more
Caused by: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To u
se JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
        at [email protected]//org.jboss.as.logging.LoggingExtension.initialize(LoggingExtension.java:186)
        at [email protected]//org.jboss.as.controller.extension.ExtensionAddHandler.initializeExtension(ExtensionAddHandler.java:131)
        at [email protected]//org.jboss.as.controller.extension.ExtensionAddHandler.initializeExtension(ExtensionAddHandler.java:103)
        at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$ExtensionInitializeTask.call(ParallelExtensionAddHandler.java:144)
        at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$ExtensionInitializeTask.call(ParallelExtensionAddHandler.java:127)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:513)

Apr 16, 2024 9:48:15 AM org.jboss.as.server.ServerService$4 logExit
FATAL: WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details. - Server configuration file in use: standalone.xml

Javaagent or library instrumentation version

opentelemetry-javaagent: v2.3.0

Environment

JDK: OpenJDK 11

Details:
openjdk version "11.0.22" 2024-01-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode, sharing)
RPM-Package: "java-11-openjdk.x86_64"

OS: Red Hat Enterprise Linux release 8.9 (Ootpa)

Additional context

I was also able to reproduce the error on other environments, such as (I was testing with opentelemetry-javaagent v2.1.0 at that time):

  • JBoss EAP 7.4.0:
    • ==> Error reproducible
  • wildfly-preview-26.0.1.Final:
    • Java-11-openjdk-11.0.22.0.7-2.el8.x86_64, java-17-openjdk-17.0.10.0.7-2.el8.x86_64, java-21-openjdk-21.0.2.0.13-1.el8.x86_64: ==> Error reproducible
    • Java 1.8.0_402: ====> Just with Java 1.8 it works (?) <=======
  • wildfly-28.0.1.Final:
    • Java-11-openjdk-11.0.22.0.7-2.el8.x86_64, java-17-openjdk-17.0.10.0.7-2.el8.x86_64, java-21-openjdk-21.0.2.0.13-1.el8.x86_64: ==> Error reproducible
  • wildfly-29.0.1.Final:
    • Java-11-openjdk-11.0.22.0.7-2.el8.x86_64, java-17-openjdk-17.0.10.0.7-2.el8.x86_64, java-21-openjdk-21.0.2.0.13-1.el8.x86_64: ==> Error reproducible
  • wildfly-30.0.1.Final:
    • Java-11-openjdk-11.0.22.0.7-2.el8.x86_64, java-17-openjdk-17.0.10.0.7-2.el8.x86_64, java-21-openjdk-21.0.2.0.13-1.el8.x86_64: ==> Error reproducible

I have already looked at issues with a similar error message, but they couldn´t help me. The cause of the error may be related to these past issues:
@flamiau04 flamiau04 added bug Something isn't working needs triage New issue that requires triage labels Apr 16, 2024
@laurit laurit mentioned this issue Apr 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage New issue that requires triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix jmx-metrics on wildfly laurit/opentelemetry-java-instrumentation
1 participant
@flamiau04