Exponential backoff not honored by fast leader query

[ajbarb]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

Librdkafka doesn't enforce exponential back-off during fast leader query. At max the backoff is 2*topic.metadata.refresh.fast.interval.ms. The issue seems to be in the rd_kafka_metadata_leader_query_tmr_cb where the backoff value passed is rtmr_interval and rd_kafka_timer_backoff calls the rd_kafka_timer_schedule which updates the rtmr_next to rtmr_interval + rtmr_interval.

Shouldn't the new rtmr_interval get doubled every time we try to backoff in rd_kafka_metadata_leader_query_tmr_cb? Something like this:

                rtmr->rtmr_interval = rtmr->rtmr_interval * 2;
                rd_kafka_timer_backoff(rkts, rtmr,
                    (int)rtmr->rtmr_interval);

How to reproduce

• Fail the rd_kafka_fetch_reply_handle with error RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION continously
• Run the rdkafka_example with:
  conf->set("topic.metadata.refresh.fast.interval.ms", "2000", errstr);
  conf->set("topic.metadata.refresh.interval.ms", "32000", errstr);
• Notice that the max delay between fast fetch request is 4sec.

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.5.0
• Apache Kafka version: <REPLACE with e.g., 0.10.2.3>
• librdkafka client configuration: topic.metadata.refresh.fast.interval.ms=2000, topic.metadata.refresh.interval.ms=32000
• Operating system: Windows 10
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[ajbarb]

@edenhill, is my understanding correct?