Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broker OffsetCommit API support #85

Closed
edenhill opened this issue Feb 22, 2014 · 8 comments
Closed

Broker OffsetCommit API support #85

edenhill opened this issue Feb 22, 2014 · 8 comments
Milestone

Comments

@edenhill
Copy link
Contributor

The offset API was added to Apache Kafka 0.8.1.
librdkafka should support it.

@edenhill edenhill added this to the 0.8.4 milestone Feb 22, 2014
@pookieman
Copy link

Hi

We would really like be able to use this, any ideas when 0.8.4 will be available.

Many thanks

@edenhill
Copy link
Contributor Author

edenhill commented Mar 5, 2014

@pookieman Within 1-2 weeks probably.
Do note though that this is only support for committing offsets to the broker (rather than local offset file), not ZooKeeper or consumer group support. Still of interest?

@pookieman
Copy link

Actually what we are looking for is https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

Finding Starting Offset for Reads - section. It would be great if we could just pick up the latest message on a topic. Currently we have to read from the beginning then disregard everything but the last one.

Cheers

@edenhill
Copy link
Contributor Author

edenhill commented Mar 5, 2014

If you don't care about losing messages produced while your consumer is
down you can simply specify RD_KAFKA_.._OFFSET_END as start offset to
consume_start().
Den 5 mar 2014 12:04 skrev "pookieman" [email protected]:

Actually what we are looking for is
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

Finding Starting Offset for Reads - section. It would be great if we could
just pick up the latest message on a topic. Currently we have to read from
the beginning then disregard everything but the last one.

Cheers

Reply to this email directly or view it on GitHubhttps://github.com//issues/85#issuecomment-36710511
.

@edenhill
Copy link
Contributor Author

edenhill commented Mar 6, 2014

Otherwise use file based offsets (already available) or wait for 0.8.4 and its broker based offsets (requires apache kafka 0.8.1 on the broker side).

@pookieman
Copy link

Not sure what file based offsets are will look at this tomorrow. The problem with RD_KAFKA_.._OFFSET_END is that I want the last message, so when the consumer starts it doesn't have to block until something has been published.

Cheers

@edenhill
Copy link
Contributor Author

edenhill commented Mar 7, 2014

File based offset storage writes the last commited (consumed..) offset to a file (one per topic+partition combo).
On restart the last offset is read from these files and consumer will resume where it left off.

edenhill added a commit that referenced this issue Mar 27, 2014
 * Offsets can now be commited and read from the broker.
 * Requires Apache Kafka 0.8.1 on the broker
 * added config property offset.store.method = file|broker  (default: file)
@edenhill
Copy link
Contributor Author

The OffsetCommit API has now be implemented, enable it by setting the topic configuration property
offset.store.method=broker.

Please verify on your side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants