Skip to content

Commit

Permalink
Set max.request.size for Kafka
Browse files Browse the repository at this point in the history
Previously, we did not configure this parameter for Kafka producer. However,
setting `buffer.memory` (size of Kafka producer buffer before flushing),
`linger.ms` (windowing period for batching requests) is not enough.
Now, we are adding `max.request.size` that will add another threshold for when
messages are flushed based upon request size.
  • Loading branch information
peel committed Oct 11, 2024
1 parent a738b9b commit 4144da0
Showing 1 changed file with 1 addition and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ object KafkaSink {
"retries" -> kafkaConfig.retries.toString,
"buffer.memory" -> bufferConfig.byteLimit.toString,
"linger.ms" -> bufferConfig.timeLimit.toString,
"max.request.size" -> kafkaConfig.maxBytes.toString,
"key.serializer" -> "org.apache.kafka.common.serialization.StringSerializer",
"value.serializer" -> "org.apache.kafka.common.serialization.ByteArraySerializer",
"sasl.login.callback.handler.class" -> authCallbackClass
Expand Down

0 comments on commit 4144da0

Please sign in to comment.