[Rhythm] Block-builder consumption loop (#4480)

* Alternate block-builder consume

* Set timeout on PollFetches, reduce initial poll delay, update 1 test to work using real consumergroup functionality

* restore metrics

* Re-add original partition lag metric, polled in separate goroutine. Fix consume loop to only consume full-duration cycles for more determinism

* merge conflict

* Review feedback

* Review feedback

* Comment

* code cleanup, lint

* logs

* code cleanup

* lint

* Review feedback

* Remove missed lookback_on_no_commit config in e2e tests and regen manifest

* Review feedback

docs/sources/tempo/configuration/manifest.md

@@ -682,7 +682,6 @@ block_builder:
     instance_id: hostname
     assigned_partitions: {}
     consume_cycle_duration: 5m0s
-    lookback_on_no_commit: 12h0m0s
     block:
         max_block_bytes: 20971520
         bloom_filter_false_positive: 0.01

integration/e2e/ingest/config-kafka.yaml

@@ -64,7 +64,6 @@ overrides:
         path: /var/tempo/overrides
 
 block_builder:
-  lookback_on_no_commit: 15m
   consume_cycle_duration: 1m
 
 ingest: