running docker compose errors with Kafka

[priamai]

Issue description

Hi there,
I am running the standard docker-compose provided in the tutorial, but Kestra doesn't start.
I am getting this error:

org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:870)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:851)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:821)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:733)
        at io.kestra.runner.kafka.services.KafkaStreamService$Stream.<init>(KafkaStreamService.java:122)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:110)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:69)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:61)
        at io.kestra.runner.kafka.KafkaScheduler.init(KafkaScheduler.java:162)
        at io.kestra.runner.kafka.KafkaScheduler.initStream(KafkaScheduler.java:170)
        at io.kestra.runner.kafka.KafkaScheduler.run(KafkaScheduler.java:188)
        at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

It seems to be related to Kafka but besides that I have no idea how to fix it.

[priamai]

Worth adding the beginning of the logs leading to that:

wait-for-it: waiting 60 seconds for kafka:9092
wait-for-it: kafka:9092 is available after 3 seconds
wait-for-it: waiting 60 seconds for elasticsearch:9200
wait-for-it: elasticsearch:9200 is available after 6 seconds
2022-04-25 09:21:08,210 INFO  standalone   io.kestra.cli.AbstractCommand Starting Kestra with environments [cli]
2022-04-25 09:21:08,863 INFO  standalone   io.kestra.cli.AbstractCommand Server Running: http://a6cffb1a0213:8080, Management server on port http://a6cffb1a0213:8081/health
2022-04-25 09:21:26,308 WARN  standalone   .r.e.AbstractElasticSearchRepository Replication incomplete, expected 2, got 1
LOTS OF THOSE REPEATED
2022-04-25 09:22:20,827 ERROR standalone-runner_1 .c.u.ThreadUncaughtExceptionHandlers Caught an exception in Thread[standalone-runner_1,5,main]. Shutting down.
org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:870)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:851)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:821)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:733)
        at io.kestra.runner.kafka.services.KafkaStreamService$Stream.<init>(KafkaStreamService.java:122)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:110)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:69)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:61)
        at io.kestra.runner.kafka.KafkaScheduler.init(KafkaScheduler.java:162)
        at io.kestra.runner.kafka.KafkaScheduler.initStream(KafkaScheduler.java:170)
        at io.kestra.runner.kafka.KafkaScheduler.run(KafkaScheduler.java:188)
        at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
        at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:127)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:867)
        ... 14 common frames omitted
2022-04-25 09:22:22,846 ERROR standalone-runner_0 .c.u.ThreadUncaughtExceptionHandlers Caught an exception in Thread[standalone-runner_0,5,main]. Shutting down.

[tchiotludo]

Can you give your environnement for running the compose please? (macos, linux, ...)

Seems that you don't write access on the host machine on /tmp/kestra-wd.
Maybe rm -rf /tmp/kestra-wd?

[priamai]

Ah good point, I totally forgot that it is a mounted folder.
The /tmp/kestra-wd has root/root permissions which is odd considering I am running docker as non-sudo.

[priamai]

Info:
Ubuntu 20.04 x64 bit
User: local user with docker group
Docker: run as local user without sudo

(same setup described here: https://docs.docker.com/engine/install/linux-postinstall/)

Could I mount it instead as a docker volume to avoid the permission issue or maybe is there a parameter to change the kestra user to non root?

[tchiotludo]

We have unroot ed05baa the docker image in order to be deployed on more environment. If you have started the docker the permission will be wrong.
It's just a temporary directory, we can safely remove it and it will work

[priamai]

Which image tag should I use with the unroot version?
This one: kestra/kestra:develop-full ?

[tchiotludo]

Yes, this one is unroot

[priamai]

Okay I pulled that image from scratch but somehow the folder gets created again as root/root.
I am beginning to wonder whether it is instead a docker issue?

[priamai]

Yes that was the case, by manually creating those folders via my local user (before starting the docker up), solved the problem!

[tchiotludo]

You have reason ... I'm hitting this very old issue moby/moby#2259 and there is no true workaround.
So I decided to unswitch the docker-compose example to root to avoid the issue.
Thanks for the reprt 👍