Bug: splitblocks: "failed injecting meta for split: invalid argument"

[mpeter50]

What is the bug?

I'm attempting to backfill Prometheus data into Mimir, and for that I needed to use the splitblocks tool to split my large blocks into smaller ones that only span a day.

However, splitblocks tells me the following for each block it processes:

level=info ts=2025-01-25T02:01:13.3734175Z block=01HK2ZFKHRNYFHRN40NZJ403M6 block_min_time=2023-12-31T17:22:34.386Z block_max_time=2024-01-01T12:00:00Z
level=debug ts=2025-01-25T02:01:13.4089947Z block=01HK2ZFKHRNYFHRN40NZJ403M6 msg="not downloading again because a provided path matches this one" file=01HK2ZFKHRNYFHRN40NZJ403M6/meta.json
level=info ts=2025-01-25T02:01:13.4104421Z block=01HK2ZFKHRNYFHRN40NZJ403M6 msg="splitting block" minTime=2023-12-31T17:22:34.386Z maxTime=2024-01-01T00:00:00Z
level=error ts=2025-01-25T02:01:13.4122069Z block=01HK2ZFKHRNYFHRN40NZJ403M6 msg="failed to split block" err="failed injecting meta for split: invalid argument"
failed injecting meta for split: invalid argument

First I was worried that maybe I screwed up the data while moving, but then I decided to give it to a prometheus server, and it can read the metrics I checked fine. Checking was done by running a query with its own web interface, and the graph view was showing information I expected.

How to reproduce it?

1. Build splitblocks from source with go 1.23.5
2. Run splitblocks as ./splitblocks --backend filesystem --filesystem.dir data --output.dir prometheus_stats_split --block-concurrency 1
3. See the error messages as processing goes. For most blocks it takes a longer time

What did you think would happen?

I expected that splitblocks successfully splits the blocks, so I can continue with backfilling it to Mimir

What was your environment?

I was running splitblocks and the test promeheus server on my workstation, in WSL.
The prometheus data was generated over time in a debian system, where prometheus was running in docker.

Any additional context to share?

Could you please release binaries of splitblocks for future releases?