Consumer lags not showing up on any topic

[poyyi]

Hello all,
Recently upgraded kafka version to 2.12-2.3.1 and also upgraded Kafkamgr (CMAK)version:2.0.0.2 (JDK 1.8),
but it is not showing the consumer Lags, logsize and many consumers are shown tried both with SASL_PLAINTEXT & SASL_SSL
does this CMAK version suppot kafka 2.3.1 if not pl suggest the version that i can pick, some of the resolution mentioned are not clear on path forward..Any help is appreciated ..Thanks

Consumer.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="mabc2345" password="xxxxxxxx18!";
bootstrap.servers=zlpy66645.vci.telco.com:2024
ssl.truststore.location=/opt/app/workload/enabler/kafka-manager-2.0.0.2/conf/kafka.server.truststore.jks
ssl.truststore.password=project@123
exclude.internal.topics=false
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
group.id=KMOffsetCache
auto.offset.reset=earliest
kafka_client_jaas.conf
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="mabc2345"
password="xxxxxxxx18!"
user_mabc2345="xxxxxxxx18!";
};

Server.properties:
listeners=SASL_PLAINTEXT://zlpy66645.vci.telco.com:2022,SASL_SSL://zlpy66645.vci.telco.com2024

Errors :

1..SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/opt/app/workload/enabler/kafka-manager-2.0.0.2/conf/kafka_client_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

2..Failed to process a message from offset topic on cluster

3...org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 30000ms
2020-08-18 15:18:45,802 - [ERROR] k.m.a.c.KafkaStateActor - KafkaTopicOffsetGetter exception
org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 30000ms

[poyyi]

i have upgraded to cmak 3.0.0.4 running into same problem, does cmak 3.0.0.4 version support kafka 2.3.1 in the version dropdown there is not 2.3.0 series at all....can some suggest which version shld i choose to get past this problem of lag not showing up.
Thanks

[poyyi]

i have upgraded to cmak 3.0.0.4 running into same problem, does cmak 3.0.0.4 version support kafka 2.3.1 in the version dropdown there is not 2.3.0 series at all....can some suggest which version shld i choose to get past this problem of lag not showing up.
Thanks

[poyyi]

i have upgraded to cmak 3.0.0.4 running into same problem, does cmak 3.0.0.4 version support kafka 2.3.1 in the version dropdown there is not 2.3.0 series at all....can some suggest which version shld i choose to get past this problem of lag not showing up.
Thanks

[poyyi]

i have upgraded to cmak 3.0.0.4 running into same problem, does cmak 3.0.0.4 version support kafka 2.3.1 in the version dropdown there is not 2.3.0 series at all....can some suggest which version shld i choose to get past this problem of lag not showing up.
Thanks

[poyyi]

i have upgraded to cmak 3.0.0.4 running into same problem, does cmak 3.0.0.4 version support kafka 2.3.1 in the version dropdown there is not 2.3.0 series at all....can some suggest which version shld i choose to get past this problem of lag not showing up.
Thanks

[kacey-r-instaclustr]

Hi, can you try the version 3.0.0.6 which will contain #793 which added 2.3.1 support. Looks like 3.0.0.6 isn't out yet though

[poyyi]

i tried with 3.0.0.5 Master consumer group Lags are still not showing up...
cmak..3.0.0.5
kafka version: 2.3.1 (upgraded from 0.11.0.0)
server.properties
inter.broker.protocol.version=2.3-IV1
log.message.format.version=0.11.0-IV2

connecting thru sasl_ssl and plain
Tried multiple kafkaversion from drop down while setting it up with Addcluster option when i chose 2.3.1(which is kafka version),
It shows this exception no cluster is configured..

Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager/PROD_SSAF_DADC_v231/kafka-state)]] after [2000 ms]. Message of type [kafka.manager.model.ActorModel$KSGetBrokers$] was sent by [Actor[akka://kafka-manager-system/user/kafka-manager/PROD_SSAF_DADC_v231#-1454503522]]. A typical reason for AskTimeoutException is that the recipient actor didn't send a reply.

But with 0.11.0.0 except of consumer lags (Lag unavailable ) everything shows up..

At the start up i see schema exception as

org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'owned_partitions': java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:110)
at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeSubscriptionV1(ConsumerProtocol.java:190)

And below timeout exception thrown repeatedly
org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 30000ms
2020-09-15 21:23:52,779 - [ERROR] k.m.a.c.KafkaStateActor - KafkaTopicOffsetGetter exception

Also tried tweaking acls to give describe and read access Kmoffsetcache group...nothing worked
Any Help is appreciated...

Thanks

[LeoHsiao1]

I also encountered this error, Kafka-manager does not show offset, lag for all topics.

2022-04-18 02:56:16,458 - [ERROR] k.m.a.c.KafkaStateActor - KafkaTopicOffsetGetter exception
org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 30000ms

My environment:

kafka: v2.8.0
cmak: v3.0.0.4
number of topic partitions: 5k

It's hard to believe that it takes more than 30 seconds to get offset.
So I try to get topic offsets one by one:

for consumer in `cat consumers`
do
    docker run -it --rm bitnami/kafka:2.8.0 /opt/bitnami/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 10.0.0.1:9092 --group $consumer --describe &> $consumer
done

grep Error *

I found that several topics could not get offsets:

Error: Executing consumer group command failed due to org.apache.kafka.common.errors.TimeoutException: Call(callName=listOffsets on broker 1, deadlineMs=1650336669074, tries=1, nextAllowedTryMs=1650336669177) timedout at 1650336669077 after 1 attempt(s)
...
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled listOffsets on broker 1 request with correlation id 5754 due to node 1 being disconnected

Then, I found similarities between these topics: a partition is blocked in the Under Replicated phase, so it cannot be connected by producers and consumers.
Finally, I deleted these topics and recreated them. Kafka-manager now displays offsets for all topics.