BlockBuilder: Consume at startup only when lagging

[codesome]

We immediately try to consume some kafka records on startup in Block Bulder.

See:

mimir/pkg/blockbuilder/blockbuilder.go

Lines 113 to 122 in f52911d

	// Do initial consumption on start using current time as the point up to which we are consuming.
	// To avoid small blocks at startup, we consume until the <consume interval> boundary + buffer.
	cycleEnd := cycleEndAtStartup(time.Now, b.cfg.ConsumeInterval, b.cfg.ConsumeIntervalBuffer)
	err := b.nextConsumeCycle(ctx, cycleEnd)
	if err != nil {
	if errors.Is(err, context.Canceled) {
	return nil
	}
	return err
	}

The proposal is the consume the records only if the partition was lagging by more than a consumption cycle (can be determined using the last kafka commit information). Otherwise there will be unnecessary fetching of records with no result.

For example if the last commit was for the record at 1:01 for a consumption cycle that started at 1:15, and rollout of block builder happened between 1:15-2:15, then we will try to consume the records from 1:01 to 1:15 (with cycle end of 1:15) on startup, while producing no blocks at this point.