blockbuilder: fix panic when closing tsdb after failing to upload block

[narqo]

What this PR does

This PR fixes a panic, I've noticed in a flaky test. When the TSDBBuilder fails to upload a block, the code of CompactAndUpload goes into the branches, that try to close the block several times. Consider the following case:

1. block B1 is closed before uploading;
2. the uploading fails;
3. the deferred function, that cleans up the tsdbs, sees that B1 wasn't "done", and closes the block again;
4. the deferred function panics with close of closed channel and interrupts the clean-up;
5. ...
6. the deferred call to TSDBBuilder.Close tries to close the blocks in the tsdbs once again, and panics once again 🙈

Also, I've notice there is a "race" condition, where the deferred clean-up function unlocks the mutex, that protects tsdbs before it cleans up the tsdbs map.

This PR fixes both issues and adds a unit test for the above scenario.

panic: close of closed channel
	panic: close of closed channel

goroutine 3553 [running]:
github.com/prometheus/prometheus/tsdb.(*DB).Close(0x14002a310e0)
	/Users/v/Documents/Code/grafana/mimir/vendor/github.com/prometheus/prometheus/tsdb/db.go:2075 +0x3c
github.com/grafana/mimir/pkg/blockbuilder.(*TSDBBuilder).Close(0x14001d2e008)
	/Users/v/Documents/Code/grafana/mimir/pkg/blockbuilder/tsdb.go:413 +0xd8
github.com/grafana/dskit/runutil.CloseWithErrCapture(0x1400aa52928, {0x105e348c0?, 0x14001d2e008?}, {0x10552cef3, 0x14}, {0x0, 0x0, 0x0})
	/Users/v/Documents/Code/grafana/mimir/vendor/github.com/grafana/dskit/runutil/runutil.go:26 +0x194
panic({0x105b2f8a0?, 0x105e2e9c0?})
	/opt/homebrew/Cellar/go/1.23.4/libexec/src/runtime/panic.go:785 +0x124
github.com/prometheus/prometheus/tsdb.(*DB).Close(0x14002a310e0)
	/Users/v/Documents/Code/grafana/mimir/vendor/github.com/prometheus/prometheus/tsdb/db.go:2075 +0x3c
github.com/grafana/mimir/pkg/blockbuilder.(*TSDBBuilder).CompactAndUpload.func1()
	/Users/v/Documents/Code/grafana/mimir/pkg/blockbuilder/tsdb.go:335 +0x208
github.com/grafana/mimir/pkg/blockbuilder.(*TSDBBuilder).CompactAndUpload(0x14001d2e008, {0x105e4bfc0, 0x14006cf80c0}, 0x140025c9a90)
	/Users/v/Documents/Code/grafana/mimir/pkg/blockbuilder/tsdb.go:400 +0x408
github.com/grafana/mimir/pkg/blockbuilder.(*BlockBuilder).consumePartitionSection(0x14001f2f308, {0x105e4bfc0, 0x14006cf80c0}, {0x105e34e60, 0x14000da0b40}, 0x14001d2e008, 0x1, {{{0x14002000ea4, 0x4}, 0x1, ...}, ...}, ...)
	/Users/v/Documents/Code/grafana/mimir/pkg/blockbuilder/blockbuilder.go:588 +0x7a8
github.com/grafana/mimir/pkg/blockbuilder.(*BlockBuilder).consumePartition(0x14001f2f308, {0x105e4bff8?, 0x14001f1cfa0?}, 0x1, {{{0x14002000ea4, 0x4}, 0x1, 0x0, 0x0, {0x0, ...}}, ...}, ...)
	/Users/v/Documents/Code/grafana/mimir/pkg/blockbuilder/blockbuilder.go:438 +0x73c
github.com/grafana/mimir/pkg/blockbuilder.(*BlockBuilder).nextConsumeCycle(0x14001f2f308, {0x105e4bff8, 0x14001f1cfa0}, {0x0, 0xedf11b4c4, 0x106d33220})
	/Users/v/Documents/Code/grafana/mimir/pkg/blockbuilder/blockbuilder.go:327 +0x6b8
github.com/grafana/mimir/pkg/blockbuilder.(*BlockBuilder).runningStandaloneMode(0x14001f2f308, {0x105e4bff8, 0x14001f1cfa0})
	/Users/v/Documents/Code/grafana/mimir/pkg/blockbuilder/blockbuilder.go:235 +0x64
github.com/grafana/dskit/services.(*BasicService).main(0x1400220a1e0)
	/Users/v/Documents/Code/grafana/mimir/vendor/github.com/grafana/dskit/services/basic_service.go:193 +0x170
created by github.com/grafana/dskit/services.(*BasicService).StartAsync.func1 in goroutine 69
	/Users/v/Documents/Code/grafana/mimir/vendor/github.com/grafana/dskit/services/basic_service.go:122 +0xfc

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[codesome]

Nice! I haven't tested the unit test but I believe it catches the panic