Thanos store: "No block found" but thanos-validator shows blocks #1220
Labels
Comments
|
Hey, i am also facing the same issue with compact. I am also using Thanos Store Logs: Thanos Bucket Inspect: Thanos Bucket Verify: Did you find out any solution for this? |
|
@vaibhavkhurana2018 No, I haven't found a solution. Honestly I haven't really tried querying for a while, but I played around with it again today, and this is what I found: I'm able to query in Thanos back to about 13 days now, which if you notice from the timestamp on this issue is right about when I updated our Thanos version. This makes me think that there may be some backwards-incompatibility issues with Thanos. Everything that was written within the last 13 days is queryable, but when I go beyond that I get the dreaded "No block found" error: If I query within the last 13 days it's slow, but it works. Additionally there are numerous log messages that look concerning, but it at least works: Query Logs
That is just an excerpt, but I don't know that's normal or not. Something to be aware of that has plagued us is that Thanos is VERY memory intensive. Make sure you are giving your instance enough memory: You can see that our instance idles at just under 14GiB, but as soon as I performed a query it quickly jumped up to over 28GiB. We've had stability issues with thanos-store in the past when we didn't give it enough memory. Memory wouldn't cause the "No block found" issue though. Did you recently update Thanos versions? I'm wondering if you're having backwards compatibility issues as well. |
|
Running in absolutely the same issue with Thanos 0.7.0 Basically no downsapled data available, only raw data. For testing, we're using a simple request
|
|
Just thinking out loud: @chrisghill @vaibhavkhurana2018 did you guys change settings of compactor downsampling during the lifetime of data? Like when you already had some data that compactor worked with and then changed amount of time you store raw/5m/1h resolutions? |
|
@homelessnessbo No, i haven't made any changes to the config post enabling compact. But, this should not impact in an ideal case because this is something that can change overtime based on the use case. |
|
@homelessnessbo It's been a long time, but I do remember that we changed the lifecycle of 0m and 5m data (to be shorter - a week for 0m and 1 month for 5m I think). And it very likely coincided with these issues. |
Yup, we are aware (: Strong efforts are made on this front:
Help is welcome (: Anyway, a summary is to put better docs into how to use downsampling. TBH I was opposed to adding any retention to Thanos because of such issues like this. Thanos is designed to store "unlimited" data. To do that we created downsampling to query long long ranges (years) with reasonable performance. Raw data is still very useful to zoom in into old historical data. All the deployments I do, I recommend allowing the same retentions for all resolutions: raw, 5m, and 1h. Maybe we should disable custom retentions like this? 🤔 We would love to enable users to fit their own cases, but looks like we failed on the communication side! |
|
TLDR from me: I was confused all the time working with Thanos, I had a fundamental misunderstanding of how downsampling settings should be set and that's where I created a problem for myself. |
|
Hi there, I'd like to share some data from our configuration to show how we got compaction + retention working for us. We have set the retention to the following values:
The result is the following for us: As you can see we have raw blocks of maximum size How compaction & retention works for us
Also important for us to make querying downsampled data work was to set |
|
Hope that this helps: https://thanos.io/components/compact.md/#downsampling-resolution-and-retention AC:
|
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Running Thanos image 0.4.0
What happened
I'm using prometheus-operator with Thanos. I just modified our Thanos-compactor to have the following retention policy:
Basically we'll use prometheus for up to 30 days of data, then Thanos for anything beyond 30 days (at 1h resolution). Thanos-compactor came up and cleared out many of the 5m and raw blocks. But now when I try to query Thanos, it only appears to see the raw data, and if I query beyond that the thanos-store says "No block found". I ran the thanos-validator to make sure 1h blocks were there, and they are:
You can see we have data going back to March 21. Now if I try to query (through grafana) for something ~30 days ago,
Those epoch's (
mintandmaxt) convert to May 5, 2019 6:35:00 PM through May 6, 2019 6:44:00 PM. Thanos should clearly have 1h resolution data from that time period. From the thanos-validator:I'm wondering if there is something strange with my configuration maybe? If I just query the last 7 days via Thanos I get something like this:
Which appears to only be valid for the most recent 1.5 days. And the Thanos store spits some errors:
But if I just zoom in on one of the days that looks invalid, then the data looks valid:
If its relevant, I did upgrade our Thanos version to 0.4.0 for all images a couple days ago. Are there incompatibilities in the blocks for those versions perhaps?
What you expected to happen
Thanos to seamlessly query and return data from blocks
How to reproduce it (as minimally and precisely as possible):
Query Thanos for any 1h resolution data
Anything else we need to know
The text was updated successfully, but these errors were encountered: