Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Periodic random kafka-ui crashes. #24

Closed
4 tasks done
PDVJAM opened this issue Dec 18, 2023 · 1 comment
Closed
4 tasks done

Periodic random kafka-ui crashes. #24

PDVJAM opened this issue Dec 18, 2023 · 1 comment

Comments

@PDVJAM
Copy link

PDVJAM commented Dec 18, 2023 •
edited
Loading

Issue submitter TODO list

  • I've looked up my issue in FAQ
  • I've searched for an already existing issues here (legacy) and here
  • I've tried installing latest charts and the issue still persists there
  • I'm running a supported version of the application & chart which is listed here

Describe the bug (actual behavior)

Hello everyone. I'm using the latest version of Kafka UI chart - 0.7.5, k8s version 1.27. The application deployment seems to be working fine, but from time to time, it crashes inexplicably. After a while, it starts working again on its own. Restarting the pod doesn't help. There are no errors in the logs either.
This is the example of logs from start to crash:

12:22:43,654 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.7
12:22:44,750 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:22:44,840 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
12:22:45,040 |-INFO in ch.qos.logback.classic.BasicConfigurator@15ff3e9e - Setting up default configuration.
12:23:04,351 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@5fdcaa40 - URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/logback-spring.xml] is not of type file
12:23:08,847 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
12:23:08,940 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:23:09,946 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
12:23:09,946 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
12:23:09,946 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
12:23:09,948 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO
12:23:09,951 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@6dc17b83 - Propagating INFO level on Logger[ROOT] onto the JUL framework
12:23:10,040 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
12:23:10,040 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@5e0826e7 - End of configuration.
12:23:10,040 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator@32eff876 - Registering current configuration as safe fallback point

 _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
 \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
                                 |_|

2023-12-18 12:23:13,854 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.6 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2023-12-18 12:23:13,943 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v3.0.6, Spring v6.0.8
2023-12-18 12:23:13,944 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"

OR

12:21:43,648 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.7
12:21:44,650 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:21:44,745 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
12:21:44,845 |-INFO in ch.qos.logback.classic.BasicConfigurator@15ff3e9e - Setting up default configuration.
12:22:03,148 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@5fdcaa40 - URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/logback-spring.xml] is not of type file
12:22:05,146 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
12:22:05,147 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:22:05,651 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
12:22:05,651 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
12:22:05,651 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
12:22:05,652 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO
12:22:05,652 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@6dc17b83 - Propagating INFO level on Logger[ROOT] onto the JUL framework
12:22:05,654 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
12:22:05,654 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@5e0826e7 - End of configuration.
12:22:05,655 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator@32eff876 - Registering current configuration as safe fallback point

 _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
 \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
                                 |_|

2023-12-18 12:22:08,949 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.6 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2023-12-18 12:22:08,958 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v3.0.6, Spring v6.0.8
2023-12-18 12:22:08,961 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"
2023-12-18 12:24:06,941 DEBUG [main] c.p.k.u.s.SerdesInitializer: Configuring serdes for cluster kafka-prod-cluster

It's completely unclear what the cause is and how to resolve this. Any ideas?

Expected behavior

Should be working all the time.

Your installation details

chart 0.7.5.
values.yaml:

existingSecret: "kafka-ui"

envs:
  secret: {}
  config:
    KAFKA_CLUSTERS_0_NAME: "kafka-dev-cluster"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka-controller-0.kafka-controller-headless.dev.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.dev.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.dev.svc.cluster.local:9092"
    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_PLAINTEXT
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
    KAFKA_CLUSTERS_0_SCHEMAREGISTRY: "http://schema-registry.dev.svc.cluster.local:8081"
    AUTH_TYPE: "LOGIN_FORM"

ingress:
  enabled: false

resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 200m
    memory: 256Mi

Steps to reproduce

I don't know how to reproduce this. The configuration is fine, but sometimes the application starts crashing suddenly.

Screenshots

No response

Logs

Error with DEBUG logging:

2023-12-18 12:38:53,445 DEBUG [SpringApplicationShutdownHook] o.s.c.s.DefaultLifecycleProcessor: Bean 'webServerStartStop' completed its stop procedure
2023-12-18 12:38:53,445 DEBUG [SpringApplicationShutdownHook] o.s.c.s.DefaultLifecycleProcessor: Stopping beans in phase -2147483647
2023-12-18 12:38:53,446 DEBUG [SpringApplicationShutdownHook] o.s.c.s.DefaultLifecycleProcessor: Bean 'springBootLoggingLifecycle' completed its stop procedure
2023-12-18 12:38:53,447 DEBUG [SpringApplicationShutdownHook] o.s.s.c.ThreadPoolTaskExecutor: Shutting down ExecutorService 'applicationTaskExecutor'
2023-12-18 12:38:53,448 DEBUG [SpringApplicationShutdownHook] o.s.s.c.ThreadPoolTaskScheduler: Shutting down ExecutorService 'taskScheduler'
2023-12-18 12:38:53,451 DEBUG [SpringApplicationShutdownHook] o.s.j.e.MBeanExporter: Unregistering JMX-exposed beans on shutdown
2023-12-18 12:38:53,747 ERROR [scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler: Unexpected error occurred in scheduled task
reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
	at reactor.core.Exceptions.propagate(Exceptions.java:408)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:91)
	at reactor.core.publisher.Mono.block(Mono.java:1710)
	at com.provectus.kafka.ui.service.ClustersStatisticsScheduler.updateStatistics(ClustersStatisticsScheduler.java:30)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.InterruptedException: null
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1048)
	at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
	... 14 common frames omitted
2023-12-18 12:38:55,646 DEBUG [reactor-http-epoll-3] i.n.b.PoolThreadCache: Freed 14 thread-local buffer(s) from thread: reactor-http-epoll-3
2023-12-18 12:38:55,742 DEBUG [reactor-http-epoll-1] i.n.b.PoolThreadCache: Freed 1 thread-local buffer(s) from thread: reactor-http-epoll-1
2023-12-18 12:38:55,744 DEBUG [reactor-http-epoll-2] i.n.b.PoolThreadCache: Freed 5 thread-local buffer(s) from thread: reactor-http-epoll-2
2023-12-18 12:38:55,748 DEBUG [reactor-http-epoll-4] i.n.b.PoolThreadCache: Freed 1 thread-local buffer(s) from thread: reactor-http-epoll-4

Additional context

No response
@Haarolean
Copy link
Contributor

most likely the pod is being killed due to failing health/readiness probes

@PDVJAM PDVJAM closed this as completed Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Haarolean @PDVJAM