"Request was for a topic or partition that does not exist on this broker" with v1.22.0 but not previous versions

[Ocaenyth]

Versions

Sarama Version: 1.22.0
Kafka Version: 2.1.0
Go Version: 1.11.4

Configuration

I am using the following environment variables in a docker-compose for Kafka :

environment:
        KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19092,LISTENER_DOCKER_EXTERNAL://localhost:9092
        KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
        KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
        KAFKA_ZOOKEEPER_CONNECT: "zoo:2181"
        KAFKA_BROKER_ID: 1
        KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
        KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
        KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

And here is the code that I am using :

func NewKafkaConnection(uri string, name string, o ConnectionOptions) (*KafkaConnection, error) {
	var conn KafkaConnection

	conn.Options = o
	conn.URI = uri
	conn.Name = name
	// Create producer
	prodConf := sarama.NewConfig()
	prodConf.ClientID = o.ClientID
	prodConf.Producer.Return.Errors = true
	prodConf.Producer.Return.Successes = true
	prodConf.Version = sarama.V2_1_0_0
	prod, err := sarama.NewSyncProducer([]string{uri}, prodConf)
	if err != nil {
		return nil, err
	}
	conn.Producer = prod
        return &conn, nil

^ To get the connection and then call the following method v

func (k *KafkaConnection) Write(topic string, msg []byte, o WriteOptions) error {
	fmt.Println("Trying to write on", topic)
	m := &sarama.ProducerMessage{
		Topic:     topic,
		Value:     sarama.ByteEncoder(msg),
		Partition: 0,
	}
	_, _, err := k.Producer.SendMessage(m)
	if err != nil {
		return err
	}
	fmt.Println("Write successful on", topic)
	return nil
}

v1.21.0 Logs

[sarama] 2019/04/23 16:49:36 Initializing new client
[sarama] 2019/04/23 16:49:36 client/metadata fetching metadata for all topics from broker localhost:9092
[sarama] 2019/04/23 16:49:36 Connected to broker at localhost:9092 (unregistered)
[sarama] 2019/04/23 16:49:36 client/brokers registered new broker #1 at localhost:9092
[sarama] 2019/04/23 16:49:36 Successfully initialized new client
Trying to write on aled.1556030976
[sarama] 2019/04/23 16:49:36 client/metadata fetching metadata for [aled.1556030976] from broker localhost:9092
[sarama] 2019/04/23 16:49:36 client/metadata found some partitions to be leaderless
[sarama] 2019/04/23 16:49:36 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2019/04/23 16:49:36 client/metadata fetching metadata for [aled.1556030976] from broker localhost:9092
[sarama] 2019/04/23 16:49:36 producer/broker/1 starting up
[sarama] 2019/04/23 16:49:36 producer/broker/1 state change to [open] on aled.1556030976/0
[sarama] 2019/04/23 16:49:36 Connected to broker at localhost:9092 (registered as #1)
Write successful on aled.1556030976
[sarama] 2019/04/23 16:49:36 Producer shutting down.
[sarama] 2019/04/23 16:49:36 Closing Client
[sarama] 2019/04/23 16:49:36 producer/broker/1 shut down
[sarama] 2019/04/23 16:49:36 Closed connection to broker localhost:9092
[sarama] 2019/04/23 16:49:36 Closed connection to broker localhost:9092

v1.22.0 Logs

[sarama] 2019/04/23 16:45:28 Initializing new client
[sarama] 2019/04/23 16:45:28 client/metadata fetching metadata for all topics from broker localhost:9092
[sarama] 2019/04/23 16:45:28 Connected to broker at localhost:9092 (unregistered)
[sarama] 2019/04/23 16:45:28 client/brokers registered new broker #1 at localhost:9092
[sarama] 2019/04/23 16:45:28 Successfully initialized new client
Trying to write on aled.1556030728
[sarama] 2019/04/23 16:45:28 client/metadata fetching metadata for [aled.1556030728] from broker localhost:9092
[sarama] 2019/04/23 16:45:28 client/metadata found some partitions to be leaderless
[sarama] 2019/04/23 16:45:28 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2019/04/23 16:45:28 client/metadata fetching metadata for [aled.1556030728] from broker localhost:9092
[sarama] 2019/04/23 16:45:28 client/metadata found some partitions to be leaderless
[sarama] 2019/04/23 16:45:28 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama] 2019/04/23 16:45:29 client/metadata fetching metadata for [aled.1556030728] from broker localhost:9092
[sarama] 2019/04/23 16:45:29 client/metadata found some partitions to be leaderless
[sarama] 2019/04/23 16:45:29 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama] 2019/04/23 16:45:29 client/metadata fetching metadata for [aled.1556030728] from broker localhost:9092
[sarama] 2019/04/23 16:45:29 client/metadata found some partitions to be leaderless
[sarama] 2019/04/23 16:45:29 Producer shutting down.
[sarama] 2019/04/23 16:45:29 Closing Client
[sarama] 2019/04/23 16:45:29 Closed connection to broker localhost:9092

Problem Description

After updating the packages for a project that uses sarama (from 1.20.1 to 1.22.0), I get an error when I try to write on a topic that was not previously created, when I did not get any before (see logs) using a sarama.SyncProducer. I have quickly looked at the 1.22.0 changelogs but could not find anything that would explain my issue.

There seems to be an issue with leaderless partitions that is not resolved fast enough in 1.22.0 (not that I understand exactly how it is resolved), that doesn't occur in 1.21.0 (where it is resolved before it runs out of attempts).

I use time.Now().Unix() as a unique seed after my topic's name for testing purposes, so that is why they are never created before-hand

[Ocaenyth]

This issue was related to #1344 and now works as of v1.22.1