client/metadata got error from broker while fetching metadata write: broken pipe

[ahamedmulaffer]

i am having these following error in my sarama logs all along

i am using using kafka 2.7.0 and go 1.17 i used sarama version 1.32.0 and after downgraded to 1.30.1 still i see the following errors

is there any workaround for this??
any one could help me on this much appreciated

[sarama] 2022/04/13 23:49:36 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:52774->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/13 23:49:36 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.16.134:46624->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/13 23:51:23 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:49956->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/13 23:59:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:55524->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/13 23:59:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:55522->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/13 23:59:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.16.134:49416->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/13 23:59:36 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:55552->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/13 23:59:36 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:55554->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/13 23:59:36 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.16.134:49450->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/13 23:59:36 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:55562->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/13 23:59:36 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:55556->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/13 23:59:36 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.16.134:49452->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/14 00:01:23 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:56058->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:38:12 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:48010->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/14 04:38:12 client/metadata got error from broker -1 while fetching metadata: kafka: broker not connected
[sarama] 2022/04/14 04:41:27 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:52308->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:44:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:51338->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:44:52 client/metadata got error from broker 2 while fetching metadata: EOF
[sarama] 2022/04/14 04:44:52 client/metadata got error from broker 1 while fetching metadata: EOF
[sarama] 2022/04/14 04:44:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:51334->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:44:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:51342->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:44:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:48014->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/14 04:44:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:51344->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:46:17 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2022/04/14 04:46:17 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2022/04/14 04:46:17 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2022/04/14 04:47:18 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2022/04/14 04:47:18 client/metadata got error from broker -1 while fetching metadata: kafka: broker not connected
[sarama] 2022/04/14 04:49:02 client/metadata got error from broker 1 while fetching metadata: EOF
[sarama] 2022/04/14 04:54:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:50830->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/14 04:54:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:50832->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/14 04:54:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:50838->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/14 04:54:52 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:51346->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:54:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:50834->2.48.2.252:9092: write: broken pipe
[sarama] 2022/04/14 04:54:52 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:51348->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:54:52 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:54160->2.48.2.252:9093: write: broken pipe
[sarama] 2022/04/14 04:57:08 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.16.134:56868->2.48.2.252:9093: write: broken pipe

Thanks
Mike

[ahamedmulaffer]

@dnwe
any thougths on this??

[dnwe]

@ahamedmulaffer broken pipe means the kafka broker / server connection has been forcibly closed at the remote end.

Check your Kafka server logs to see if they give a hint as to what is wrong with your configuration or network environment

[ahamedmulaffer]

@dnwe
thanks for your reply i am running this kafka cluster behind a loadbalnacer which is controlled through HAproxy

i am running a 2 node cluster behind HAProxy

in my kafka broker log i couldnt see any errors logged

or is this related to sarama versions??

[dnwe]

I’d suspect that your haproxy is chopping the connections maybe. Probably worth looking at the stats interface of haproxy to see which of the three (client, proxy, server) are responsible for closing the connections

[ahamedmulaffer]

@dnwe @niamster @slaunay

this is my producer config
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForAll
config.Producer.Retry.Max = 100
config.Producer.Return.Successes = true

here am i missing anything because same issue has been opened also earlier
https://github.com/Shopify/sarama/issues/1954
https://github.com/Shopify/sarama/issues/760

here i am running a missing critical system every data on the kafka is very important for me

Thanks
Mike

[niamster]

@ahamedmulaffer as Dominic above mentioned please try to check HAProxy setup.

Have you ever had this setup worked?
Did you try to remove HAProxy from the equation (make Sarama connect directly to Kafka)?
BTW, why do you need HAProxy here, what are you trying to achieve?

Just a friendly reminder: please do not tag everyone if you'd like to have some meaningful help.

[ahamedmulaffer]

thanks @niamster for your reply

so far i checked the HA proxy log i coudnt see any its chopping any connections
we are using HAProxy load balancer we are using one public ip for two kafka node with differents ports
in my case i can not remove HAProxy if i remove i cant use kafka as a cluster
here i am trying to publish data from aws to another environment(A) this environment A's Kafka clusters are running behind the HA proxy
is the problem related to https://github.com/Shopify/sarama/issues/2173 ??

this is the sarama version im using
github.com/Shopify/sarama v1.32.1-0.20220330173227-d5f076b9e201

Thanks
Mike

[niamster]

It's hard to tell whether your problem is related to #2173
Can you be more specific please? Is you application (that uses Sarama) able to connect to Kafka? If so, do you see any other side-effects except these error logs?

[ahamedmulaffer]

@niamster
yes my application able to connect to kafka and its able to publish the data as well
when this broken pipe message occur i am missing these data most of the data are publishing some are missing
daily im publishing 5million of data using this sarama library
is there any way i can republish the message which was caused by broken pipe error??
am i missing any producer configuration??

in the log i am able to see these errors only
[sarama] 2022/04/15 13:59:17 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:52080->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 13:59:17 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:52082->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 13:59:17 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.16.134:55410->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 13:59:31 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/04/15 13:59:31 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/04/15 13:59:31 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/04/15 13:59:31 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/04/15 13:59:31 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/04/15 14:16:08 client/metadata got error from broker -1 while fetching metadata: kafka: broker not connected [sarama] 2022/04/15 14:16:08 client/metadata got error from broker -1 while fetching metadata: kafka: broker not connected

thanks
mike

[niamster]

Thank you for providing more information.
I suspect there's a kind of connection TTL on HAProxy side that kills your Sarama's link to the broker.
Does this happen randomly or after a specific period of time?

---

is there any way i can republish the message which was caused by broken pipe error??

I'm not quite sure what kind of help you are asking here.
You should be able to handle Sarama producer errors and retry messages. The final solution depends on the architecture of your application.

[ahamedmulaffer]

@niamster
its seems like most probably it happens around every 10 minutes which i see apache kafka's default producer's and consumer's connections.max.idle.ms is 9 minutes and broker config default connections.max.idle.ms is 10 minutes
i am attaching the logs here you can see the time interval as well
[sarama] 2022/04/15 15:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:47326->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:39024->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43990->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:31:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:42556->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:31:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:50362->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:52396->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:41:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52392->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:41:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:41806->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44736->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44730->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44732->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44728->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44734->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:51:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47396->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 15:51:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55208->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55348->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55350->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52550->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55356->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55352->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52548->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 15:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55354->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:01:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50188->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:58138->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:44738->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:50334->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:44746->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:58146->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:58144->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:44744->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:57994->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:11:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:52960->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60772->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:53110->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:53112->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:53114->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60914->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:50332->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60916->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:35352->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:35492->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55924->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:55922->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:35494->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:35496->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:31:21 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58550->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:31:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38128->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:31:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:55774->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38276->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38284->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38278->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38282->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:41:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:40868->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:41:31 client/metadata got error from broker 1 while fetching metadata: EOF [sarama] 2022/04/15 16:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:40918->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:41072->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38280->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:41:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:33260->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:50:09 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:43732->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:51:31 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/04/15 16:51:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:35922->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43874->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:33258->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43878->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:33262->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:36076->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43876->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 16:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:33264->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 16:53:33 client/metadata got error from broker 2 while fetching metadata: EOF [sarama] 2022/04/15 16:54:02 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/04/15 17:01:31 client/metadata got error from broker 1 while fetching metadata: EOF [sarama] 2022/04/15 17:01:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46540->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38886->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38888->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47274->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:36072->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:47272->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:39462->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:46686->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46688->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:04:51 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49392->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:49572->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41770->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41772->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49576->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:49574->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52956->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:21:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:42766->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52958->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:21:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:41588->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:53108->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:53110->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:53114->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55744->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55746->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:53106->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:31:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:48074->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55884->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55890->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55886->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:31:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:45304->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50700->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50702->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50852->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:41:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:50850->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50854->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:58656->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:41:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:48080->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:33076->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:33078->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:33218->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:33222->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:50846->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:33220->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 17:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:53652->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 17:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:33224->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:36000->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:36004->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:56432->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38628->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:11:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:56278->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38632->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:11:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:56280->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35998->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:59210->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38794->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38790->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:56426->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41456->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41458->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:33794->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:41596->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38792->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:33796->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:33788->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:41600->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:59212->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:36414->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:36416->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44366->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:36566->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44370->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:36568->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:47150->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:47152->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:41:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:36560->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47154->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:47158->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47156->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49806->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:51:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:39194->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:51:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:42000->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:47004->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49962->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 18:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:42158->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 18:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49964->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:01:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:52602->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:01:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:52604->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52744->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:42150->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52752->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:42152->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:52748->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:11:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47564->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:11:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47568->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47718->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55520->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47720->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:44938->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:44942->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55518->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55522->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:58184->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58324->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58326->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50524->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50522->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:31:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60946->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:31:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:50374->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:32868->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:32864->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:32862->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:31:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:50520->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:32866->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:35494->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:41:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:53140->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:41:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35492->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35636->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:41:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:53284->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35640->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35642->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:56070->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:51:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58712->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:51:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38290->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58866->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:58854->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:56062->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 19:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38434->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 19:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38430->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:01:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:33244->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41192->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:41196->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:33392->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:41194->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:41198->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:58860->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:11:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:33240->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43806->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43808->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43950->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:36148->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:43954->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:36152->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:46604->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46758->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:46764->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46760->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46762->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:36144->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41566->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:31:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49376->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:31:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38792->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49516->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:31:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38948->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41716->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49518->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:52146->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44490->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52290->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:52288->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:52292->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:41:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:41712->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:41:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:41714->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:54924->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:51:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:44336->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:51:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47118->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55104->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55106->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47304->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:55108->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 20:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:44486->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 20:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:47306->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:01:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:57762->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:01:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:57758->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:57900->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:57904->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:50102->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:57906->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:11:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:52714->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60672->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60668->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:60670->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60666->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:50094->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35062->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35064->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:21:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:52716->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35232->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:52856->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55666->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:36182->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:56608->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:37888->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58458->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38028->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:35236->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58460->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:38030->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:31:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:58456->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:31:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:55658->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:41:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:40654->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:41:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:58308->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:32848->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:40796->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:40800->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:32998->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:32996->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 21:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:40798->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:51:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:43428->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43426->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43592->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:43596->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43594->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 21:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:43598->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:01:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38430->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46382->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46384->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:35792->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:46386->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49006->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:11:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38432->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49148->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38572->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49152->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38578->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:41346->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:21:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:49004->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:21:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:43960->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:51774->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44114->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:51916->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:41342->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:51918->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:44116->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:21:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:41344->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:21:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:51914->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:54566->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:46756->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:54704->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:54706->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:46906->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:46904->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:54708->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:41:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:49524->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 22:41:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:57334->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:57476->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:57484->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:57480->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:57482->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60094->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60244->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60240->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:60246->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 22:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:49670->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:01:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:34676->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:01:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:52290->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:34818->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:52430->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:01:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:55252->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:01:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:34820->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:01:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:52434->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:11:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:37464->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:11:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:55098->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:11:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:37466->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58036->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:55248->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:58038->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:37610->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:11:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:37608->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:11:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:55246->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:11:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:37606->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:40238->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:21:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:60660->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:40376->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:60808->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:60810->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:21:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:40378->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:31:31 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:35262->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:31:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43074->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43216->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43222->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:43224->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:31:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:43220->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:31:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:60804->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:31:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:43218->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:41:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:45844->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:45994->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:41:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:45996->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:41:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:38190->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:51:31 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:48618->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:51:31 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38038->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:48760->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38182->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:38184->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:51:59 client/metadata got error from broker 1 while fetching metadata: write tcp 172.31.32.204:40956->2.48.2.252:9092: write: broken pipe [sarama] 2022/04/15 23:51:59 client/metadata got error from broker 2 while fetching metadata: write tcp 172.31.32.204:48762->2.48.2.252:9093: write: broken pipe [sarama] 2022/04/15 23:51:59 client/metadata got error from broker -1 while fetching metadata: write tcp 172.31.32.204:48766->2.48.2.252:9093: write: broken pipe

from this what i understand sarama library trying to use the connection which is already closed by kafka which has elapsed the idle time

actually how the connection happens
A. from sarama library to HAProxy
B. from HAProxy to actual Kafka brokers
so here B's connection closed state was not known to Connection A that (may be other possibilities also) why sarama using the connection A to produce but B is already closed

here i am attaching my HAProxy config file as well
HAProxy timeouts
timeout connect 5000
timeout client 50000
timeout server 50000
but timeout client already 50 seconds here
@niamster these information might help to you

Thanks
mike

[niamster]

Thanks @ahamedmulaffer for this information.

Just to be clear, I've never tried to setup Kafka behind HAProxy so I'm not a great help to you here except trying to help you debugging your setup.

If I understand correctly, you actually don't loose any data (I don't see any producer errors in your logs). How do you confirm that you loose some data?

I'm speculating here, but what might be happening in your case is that your producer is writing to only one of the configured brokers. Indeed Kafka closes connection after 10 minutes by default if the client does not communicate with it anymore.
How many topics/partitions are you writing to?
Can you please try to reduce metadata refresh period to, say, 5mn (default refresh period is 10mn)?

BTW, did you have a chance to look at HAProxy stats interface as Dominic requested here?

[ahamedmulaffer]

hi @niamister
thanks for your reply
i was able to fix this issue by commenting the timeout or matching the timeout to brokers default timeout in HAProxy after that i dont see any metadata fetching error
between thanks for your help guys @niamster and @dnwe
Thank you guys
keep up the good work

[VipulZopSmart]

Hi @ahamedmulaffer , i'm also encountering this same, how can we resolve this.

But i'm seeing 2 types of errors here

1 . EOF
2. write tcp 10.17.202.73:59500->34.213.193.21:9092: write: broken pipe

Could you please help us what all you've tried in the configs?

[slaunay]

Kafka clients relies on a long lived TCP connection to brokers on the contrary to something like HTTP/1.0.
This means that eventually such TCP connections will be closed (EOF when reading, broken pipe, ...) which can and will happen.
The connections.max.idle.ms on the broker side is set to 10 minutes by default so if your producer is not "talking" (querying metadata or producing) to a specific broker for that long the connection will be closed by the server (it will manifest differently if caught by the producer on read or a write against the TCP connection).

This is where the retry logic comes in and if you have a Producer configured as idempotent it will avoid possible duplicates so seeing those errors might not be an issue if they are properly retried.
There were a few major bugs introduced recently in the Producer where retries (mostly on network errors) would either introduce a deadlock or inject an empty record and stop processing new records:
https://github.com/Shopify/sarama/blob/53e77a4dd7182e049493dbce531d0dc5dbc31567/go.mod#L29-L30
So you should make sure you use version 1.33.0 or later.

Related to HAProxy used in front of Kafka, I don't think it brings much value.
The reason is that you connect to a random bootstrap server/Broker first (possibly HAProxy) but then there is a discovery mechanism to talk directly to a specific Broker for future requests (querying metadata, producing).
So that might help providing some HA for the initial discovery (listing multiple brokers and not necessarily all works too) but it is my understanding that HAProxy will be bypassed after that.

[VipulZopSmart]

Hi @slaunay, we've implemented retry logic(retrying once to form connection again) in case of we get pipe error, EOF error and broker not connected error, but the connection should be reestablished but it's not initiating the connection in 1 retry. Could you please kindly help here?

Like how much retries it need to reestablish the connection. Or if there's any other retry mech that we can use to avoid this error.