Hang in KafkaConsumer shutdown when close() is not called

[gridaphobe]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

I've observed a new hang in the C++ KafkaConsumer in v1.5.2 when shutting down without calling close() explicitly. I believe this is related to the recent change to call rd_kafka_destroy() in the dtor. TSAN also occasionally detects a data race between rd_kafka_toppar_op_serve and rd_kafka_fetch_reply_handle on the rktp_last_error field of a toppar while shutting down this consumer.

How to reproduce

See https://gist.github.com/gridaphobe/a2f20726171eb957358bd2a91123015f for a small example, with debug logs, and the output of TSAN.

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): v1.5.2 and 521f460
• Apache Kafka version: mock broker
• librdkafka client configuration: https://gist.github.com/gridaphobe/a2f20726171eb957358bd2a91123015f#file-kafkaconsumer-unsubscribe-noclose-cpp-L93-L97
• Operating system: macOS 10.15 and ubuntu bionic
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[gridaphobe]

I've updated the gist with line numbers in the TSAN output. It looks like the detected data race is not on the refcnt itself, but on the rktp_last_error field of the toppar.