Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metrics error with 0.5.2.6 #618

Closed
makghub9 opened this issue Aug 5, 2023 · 3 comments · Fixed by #620
Closed

Metrics error with 0.5.2.6 #618

makghub9 opened this issue Aug 5, 2023 · 3 comments · Fixed by #620

Comments

@makghub9
Copy link

makghub9 commented Aug 5, 2023

Sometimes I'm getting below error with the newer version jar. It happens on startup only sometimes. If I remove the meter registry from the parallel consumer, this won't happen.

2023-08-05 17:44:24,380 [http-nio-8083-exec-2] Failed to apply the value function for the gauge 'pc.partitions.paused'. Note that subsequent logs will be logged at debug level.
java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:2491)
at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2475)
at org.apache.kafka.clients.consumer.KafkaConsumer.paused(KafkaConsumer.java:2066)
at io.confluent.parallelconsumer.internal.ConsumerManager.paused(ConsumerManager.java:133)
at io.confluent.parallelconsumer.internal.BrokerPollSystem.lambda$initMetrics$1(BrokerPollSystem.java:92)
at io.micrometer.core.instrument.StrongReferenceGaugeFunction.applyAsDouble(StrongReferenceGaugeFunction.java:48)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:53)
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$5(PrometheusMeterRegistry.java:331)
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:75)
at io.prometheus.client.Collector.collect(Collector.java:45)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152)
at io.prometheus.client.exporter.common.TextFormat.writeOpenMetrics100(TextFormat.java:202)
at org.springframework.boot.actuate.metrics.export.prometheus.TextOutputFormat$2.write(TextOutputFormat.java:62)
at org.springframework.boot.actuate.metrics.export.prometheus.PrometheusScrapeEndpoint.scrape(PrometheusScrapeEndpoint.java:62)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:281)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74)
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:321)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:428)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:207)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:152)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
@acktsap acktsap mentioned this issue Aug 6, 2023
Merged
2 tasks
@JorgenRingen
Copy link
Contributor

Might be related:
Is it possible to make the micrometer-registry-atlas a "provided" depndency? AtlasRegistry is picked up by spring boot and automatically used in auto-configuration if its on the classpath.

a5bed3f#diff-5729cbacdf946345379af3b300c9fc6b5303b2a4faa646425a9a2c2be6cc1cd2R45

This results in warnings in the application as we're using prometheus.

023-08-09T15:10:16,502 INFO  [spectator-atlas-publish-0] c.n.s.i.h.HttpRequestBuilder traceId=: attempt 1 of 3 failed: POST http://localhost:7101/api/v1/publish
2023-08-09T15:10:16,502 INFO  [spectator-atlas-publish-0] c.n.s.i.h.HttpRequestBuilder traceId=: attempt 2 of 3 failed: POST http://localhost:7101/api/v1/publish

Solution is to exclude the atlas dependency.

@makghub9
Copy link
Author

makghub9 commented Aug 9, 2023

Might be related:

Is it possible to make the micrometer-registry-atlas a "provided" depndency? AtlasRegistry is picked up by spring boot and automatically used in auto-configuration if its on the classpath.

a5bed3f#diff-5729cbacdf946345379af3b300c9fc6b5303b2a4faa646425a9a2c2be6cc1cd2R45

This results in warnings in the application as we're using prometheus.


023-08-09T15:10:16,502 INFO  [spectator-atlas-publish-0] c.n.s.i.h.HttpRequestBuilder traceId=: attempt 1 of 3 failed: POST http://localhost:7101/api/v1/publish

2023-08-09T15:10:16,502 INFO  [spectator-atlas-publish-0] c.n.s.i.h.HttpRequestBuilder traceId=: attempt 2 of 3 failed: POST http://localhost:7101/api/v1/publish

Solution is to exclude the atlas dependency.

You can use management.atlas. metrics.export. disabled=false in springboot application.properties file.It will ignore altas.

@JorgenRingen
Copy link
Contributor

Yes, I just used

    implementation("io.confluent.parallelconsumer:parallel-consumer-core:0.5.2.6") {
        exclude(module = "micrometer-registry-atlas")
    }

But shouldn't micrometer-core be enough as a transitive dependency and then make atlas test or provided scoped?

@JorgenRingen JorgenRingen mentioned this issue Aug 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Return cached pausedPartitionSet
2 participants
@JorgenRingen @makghub9