[rhythm] Implement group-consuming in the metrics-generator #4593
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mdisibio
reviewed
Jan 23, 2025
| // This is so the cycle started exactly at the commit offset, and not at what was (potentially over-) consumed previously. | ||
| // In the end, we remove the partition from the client (refer to the defer below) to guarantee the client always consumes | ||
| // from one partition at a time. I.e. when this partition is consumed, we start consuming the next one. | ||
| g.kafkaClient.AddConsumePartitions(map[string]map[int32]kgo.Offset{ |
There was a problem hiding this comment.
Can you help clarify for my understanding - since this client update is removed, how does PollFetches know which partitions are assigned? Is that now handled by the server (broker, etc)?
There was a problem hiding this comment.
Yes, when joining a group (ie. kgo.ConsumerGroup(kafkaCfg.ConsumerGroup)) Kafka handles partitions assignment. The client will talk to the broker which will share the partition balance for it. In each pull, the client will receive fetches for the partitions that have been assigned to it.
mapno
requested review from
knylander-grafana,
joe-elliott,
yvrhdn,
zalegrala,
electron0zero,
ie-pham,
stoewer and
javiermolinar
as code owners
mdisibio
approved these changes
Jan 28, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
This PR introduces group consuming to the generators, instead of the previously used direct consuming. Group consuming delegates partitions assignment to Kafka, which greatly simplifies operations for the generators.
It uses a custom balancer that combines Kafka's cooperative sticky balancing with partition ring awareness. This implementation is forked from Loki.
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]