Connecting to a secure zeebe cluster

[mihneastaub]

Hello Guys,

I have a secure zeebe cluster deployed in kubernetes using the helm chart, when I want to connect to the cluster using this client, I have received this error:

__  __ _                                  _   
|  \/  (_) ___ _ __ ___  _ __   __ _ _   _| |_ 
| |\/| | |/ __| '__/ _ \| '_ \ / _` | | | | __|
| |  | | | (__| | | (_) | | | | (_| | |_| | |_ 
|_|  |_|_|\___|_|  \___/|_| |_|\__,_|\__,_|\__|
  Micronaut (v3.2.7)

10:32:44.374 [main] INFO  i.n.m.z.c.feature.ZeebeClientFactory - ZeebeClient is configured to connect to gateway: localhost:26500
10:32:44.406 [ForkJoinPool.commonPool-worker-17] INFO  i.n.m.z.c.f.ZeebeWorkerProcessor - Zeebe client (com.worldremit.api.service.ProcessService#createTransaction) subscribed to type 'create_transaction'
10:32:44.669 [main] INFO  io.micronaut.runtime.Micronaut - Startup completed in 1333ms. Server Running: http://localhost:8085/
10:32:44.734 [grpc-default-executor-1] WARN  io.camunda.zeebe.client.job.poller - Failed to activated jobs for worker default and job type create_transaction
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:479)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00001204000000000000037fffffff000400100000000600002000000004080000000000000f0001
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1213)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 common frames omitted

The important error is this one: Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00001204000000000000037fffffff000400100000000600002000000004080000000000000f0001
Do you know why the client cannot connect to the secure cluster?

Thank you!

[tobiasschaefer]

Hi @mihneastaub ,

by default the client connects to localhost:26500 using plain text. If Camunda Cloud Credentials are provided it will use SSL/TLS.

In your case localhost:26500 seems to be correct but it is falsely connecting with plain text. Unfortunately, we don't have a configuration option yet to configure "usePlainText". We'll add that now because it definitely makes sense.

There is a quick solution because you can replace the factory bean. We'll take your issue as a trigger to refactor it to make it easier in future (for similar cases).

For now the following class in your project should solve the problem (note: usePlaintext is commented out)

import info.novatec.micronaut.zeebe.client.feature.Configuration;
import info.novatec.micronaut.zeebe.client.feature.ZeebeClientFactory;
import io.camunda.zeebe.client.ZeebeClient;
import io.camunda.zeebe.client.ZeebeClientBuilder;
import io.micronaut.context.annotation.Factory;
import io.micronaut.context.annotation.Replaces;
import jakarta.inject.Singleton;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.time.Duration;

@Factory
@Replaces(factory = ZeebeClientFactory.class)
public class MyZeebeClientFactory extends ZeebeClientFactory {

    private static final Logger log = LoggerFactory.getLogger(MyZeebeClientFactory.class);

    @Singleton
    public ZeebeClient buildClient(Configuration configuration) {

        ZeebeClientBuilder zeebeClientBuilder = isCloudConfigurationPresent(configuration)
                ? createCloudClient(configuration)
                : ZeebeClient.newClientBuilder(); //.usePlaintext();

        configuration.getDefaultRequestTimeout().ifPresent(timeout -> zeebeClientBuilder.defaultRequestTimeout(Duration.parse(timeout)));
        configuration.getDefaultJobPollInterval().ifPresent(duration -> zeebeClientBuilder.defaultJobPollInterval(Duration.ofMillis(duration)));
        configuration.getDefaultJobTimeout().ifPresent(timeout -> zeebeClientBuilder.defaultJobTimeout(Duration.parse(timeout)));
        configuration.getDefaultMessageTimeToLive().ifPresent( ttl -> zeebeClientBuilder.defaultMessageTimeToLive(Duration.parse(ttl)));
        configuration.getDefaultJobWorkerName().ifPresent(zeebeClientBuilder::defaultJobWorkerName);
        configuration.getGatewayAddress().ifPresent(zeebeClientBuilder::gatewayAddress);
        configuration.getNumJobWorkerExecutionThreads().ifPresent(zeebeClientBuilder::numJobWorkerExecutionThreads);
        configuration.getKeepAlive().ifPresent(keepAlive -> zeebeClientBuilder.keepAlive(Duration.parse(keepAlive)));
        configuration.getCaCertificatePath().ifPresent(zeebeClientBuilder::caCertificatePath);

        ZeebeClient zeebeClient = zeebeClientBuilder.build();
        log.info("ZeebeClient is configured to connect to gateway: {}", zeebeClient.getConfiguration().getGatewayAddress());
        return zeebeClient;
    }
}

Does that solve your issue?

[mihneastaub]

Hi @tobiasschaefer,

Thank you for the quick answer. I will check this and I will comeback to you.

[tobiasschaefer]

Hi @mihneastaub ,

I just released v1.4.0 of the micronaut-zeebe-integration. Setting zeebe.client.cloud.use-plain-text-connection: false should solve your issue.

Note: our release v1.4.0 is built with the latest Micronaut 3.3.0 version. The update from your 3.2.7 version should be trivial, i.e. you only need to switch to the new Micronaut version.

Please confirm that is working for you.

[tobiasschaefer]

Hi @mihneastaub ,

I provided you with a workaround and also a release which should fix your issue.

Please confirm that it works for you.

Thanks

[mihneastaub]

Hi @tobiasschaefer,

Sorry for the late response. For now I did not have time to test this...

Thank you very much for the workaround and for the fix, much appreciated.

I will close this and if something is not working I will open again a new issue.

Thanks!