Thanos compactor throw error `open index file: read symbols: read symbols: invalid checksum` in compaction process #1728

smalldirector · 2019-11-06T19:34:20Z

Today I found Thanos compactor keep throwing error open index file: read symbols: read symbols: invalid checksum when doing the block compaction. The compaction process tried to merge those 7 blocks to one final block(level 4):

28.2G	01DQH8S0KZVE6JKX06KXVF2ZBP
28.9G	01DQPDC15EGKDR26H40BKAB0AK
28.9G	01DQVHKVKXPSBHC3WE36PZN85C
29.1G	01DR0R43SH4JCMF08JTDENGZWS
28.9G	01DR5W0R4APZ205EX4YWNFQ767
43.0G	01DRB1R4J33A27V8Y1M84YBV9A
29.5G	01DRGJH1XREVCC0MWPAEMG3C31

I saw the new block was created successfully, however it can not pass the function GatherIndexIssueStats check.
I checked the source code, seems the error come from below codes:

func GatherIndexIssueStats(logger log.Logger, fn string, minTime int64, maxTime int64) (stats Stats, err error) {
	r, err := index.NewFileReader(fn)
	if err != nil {
		return stats, errors.Wrap(err, "open index file")
	}

All the blocks are genereated by compactor, why does it can throw such error? Has anyone faced similar issue before?

BTW, full logs is here:

level=info ts=2019-11-06T18:16:15.206410018Z caller=compact.go:1098 msg="start of GC"
level=info ts=2019-11-06T18:16:15.207942229Z caller=compact.go:1104 msg="start of compaction"
level=debug ts=2019-11-06T18:37:32.593837143Z caller=compact.go:888 compactionGroup="0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}" msg="downloaded and verified blocks" blocks="[/mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DQH8S0KZVE6JKX06KXVF2ZBP /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DQPDC15EGKDR26H40BKAB0AK /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DQVHKVKXPSBHC3WE36PZN85C /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DR0R43SH4JCMF08JTDENGZWS /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DR5W0R4APZ205EX4YWNFQ767 /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DRB1R4J33A27V8Y1M84YBV9A /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DRGJH1XREVCC0MWPAEMG3C31]" duration=21m17.372998762s
level=info ts=2019-11-06T19:18:35.894179775Z caller=compact.go:441 msg="compact blocks" count=7 mint=1571270400000 maxt=1572480000000 ulid=01DS105H16T6RD21ANJPGZ6ACQ sources="[01DQH8S0KZVE6JKX06KXVF2ZBP 01DQPDC15EGKDR26H40BKAB0AK 01DQVHKVKXPSBHC3WE36PZN85C 01DR0R43SH4JCMF08JTDENGZWS 01DR5W0R4APZ205EX4YWNFQ767 01DRB1R4J33A27V8Y1M84YBV9A 01DRGJH1XREVCC0MWPAEMG3C31]" duration=41m3.300225471s
level=debug ts=2019-11-06T19:18:36.859074005Z caller=compact.go:915 compactionGroup="0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}" msg="compacted blocks" blocks="[/mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DQH8S0KZVE6JKX06KXVF2ZBP /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DQPDC15EGKDR26H40BKAB0AK /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DQVHKVKXPSBHC3WE36PZN85C /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DR0R43SH4JCMF08JTDENGZWS /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DR5W0R4APZ205EX4YWNFQ767 /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DRB1R4J33A27V8Y1M84YBV9A /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DRGJH1XREVCC0MWPAEMG3C31]" duration=41m4.265120458s
level=warn ts=2019-11-06T19:18:46.17049971Z caller=prober.go:154 msg="changing probe status" status=unhealthy reason="error executing compaction: compaction failed: compaction failed for group 0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}: invalid result block /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DS105H16T6RD21ANJPGZ6ACQ: open index file: read symbols: read symbols: invalid checksum"
level=error ts=2019-11-06T19:18:46.170664646Z caller=main.go:215 msg="running command failed" err="error executing compaction: compaction failed: compaction failed for group 0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}: invalid result block /mnt/thanos/store/compact/0@{_clusterId_=\"ms-nsn-prod\",_replica_=\"23-ms-nsn-0\",_shard_=\"0\"}/01DS105H16T6RD21ANJPGZ6ACQ: open index file: read symbols: read symbols: invalid checksum"

Used version 0.4, 0.6 and 0.8.1 to test, and all of them throw this error.

The text was updated successfully, but these errors were encountered:

FUSAKLA · 2019-11-08T19:41:53Z

Hi, I suppose running bucket inspect on it ends up same way probably?
Does the new compacted block contain the meta.json file?

smalldirector · 2019-11-08T21:34:25Z

yes, the meta.json file is there.

FUSAKLA · 2019-11-08T23:03:05Z

And out of curiosity how big is the resulting index file of the new block?

smalldirector · 2019-11-08T23:43:23Z

@FUSAKLA Good question here. The compacted index is around 22GB.
-rw-r--r-- 1 110262044 110262044 22G Nov 7 15:36 index

smalldirector · 2019-11-08T23:46:32Z

Do you think Prom TSDB could not support very large index?
Actually had one discussion with @bwplotka on #thanos slack channel before, he mentioned one ticket said Prom TSDB couldn't support index larger than 4G: prometheus-junkyard/tsdb#273
However, this issue seems to be already fixed.

FUSAKLA · 2019-11-09T09:44:42Z

Yeah, it was even discussed on PromCon but the current limit is 64 GiB IIRC
(see nice blog post about it https://grafana.com/blog/2019/10/31/lifting-the-index-size-limit-of-prometheus-with-postings-compression/ it's currently being solved here prometheus/prometheus#5884)
But 22 GiB should be still ok I suppose.

bwplotka · 2019-11-09T13:12:46Z

Just to double check, it is the output block that is 22GB index. How meta.json looks like? (: Might be some other thing that is out of int32 but it's interesting we detect in read time not write time.

smalldirector · 2019-11-09T21:27:12Z

Here is the meta.json:

{
	"ulid": "01DS3HKJ6JAV4GC1SQ393N18Q1",
	"minTime": 1571270400000,
	"maxTime": 1572480000000,
	"stats": {
		"numSamples": 93466711137,
		"numSeries": 39725454,
		"numChunks": 798293377
	},
	"compaction": {
		"level": 4,
		"sources": [
			"01DQBZH62GMXNQAF2RPCYHEQF4",
			"01DQC9TRWNDV4NKX2VYKM35VX4",
			"01DQCM4BPNC4N1S2MJKS17DFPG",
			"01DQCYDYT8XGVBY85S16MDATMW",
			"01DQD8QH9EGBHD2FF24XEDZ3C6",
			"01DQDK143AR5PAF6Q7AF2RWH8F",
			"01DQDXAPXMJGAEF4R2B51GR0BP",
			"01DQE7M9QA8DFFGVE5DQHSXC0J",
			"01DQEHXWG150336D8QDY0NCERN",
			"01DQEW7FB1Q155MZNW979XPNKJ",
			"01DQF6H25GYB5TNA4058RENTX4",
			"01DQFGTMYG8T7M793ZTC7A2MSM",
			"01DQFV47RF99XVYPAW0XKFJNAF",
			"01DQG5DTM3VS3CDCQ44E9TR6TE",
			"01DQGFQDB2NA29P02A17MZGQ95",
			"01DQGT104SGKMHYZY6BWMWJ2AM",
			"01DQH4AJZ0FQKKSVA9DZGZTY87",
			"01DQHEM5S2PVVTZ8DVTH9B4JGE",
			"01DQHRXRJRVR2ARER3DNWZA5C0",
			"01DQJ37BCEV1473BXPZXCATCGJ",
			"01DQJDGY5WW2JZA7WDKSX5JRPF",
			"01DQJQTH0552MA72D1ZAR7JR3W",
			"01DQK243T1XZ58WCFFGV1GF18T",
			"01DQKCDPJFGQHZXF06Y4FNWDYT",
			"01DQKPQ9DS8JCE8YWQNJJVWHZ1",
			"01DQM10WDAN25T5HEV99T209E3",
			"01DQMBAF13YV2AB8TRWMDB6HC1",
			"01DQMNM1TD5NXHKWAXMHKWCG3C",
			"01DQMZXMN7GEMQ3Y83FJMP0R93",
			"01DQNA77ECG68D7CSNYTX7ASX9",
			"01DQNMGT8NBS33K20J4S4FPB1D",
			"01DQNYTD0WN5YG7WKKWBAXJD8W",
			"01DQP93ZVA05GTB3FP3GXKG8Z2",
			"01DQPKDJY8GPBNHJ0QEPXBGQ2A",
			"01DQPXQ5F6FJSKX9QPVRE7DMZ2",
			"01DQQ80R8Z8S0T0X9FJ0FD4RN9",
			"01DQQJAB3JB2TRP6Q71W15PVB7",
			"01DQQWKXW14KH8X3AE030GRZ45",
			"01DQR6XGPAR6W17A2Q17SKSR5C",
			"01DQRH73FWWKBQH1TCE8WN4EKJ",
			"01DQRVGPB2YKNPFNRP6B8D7GGD",
			"01DQS5T964RD900W0K1HBFE580",
			"01DQSG3VYRZ8AW4P1TVCZ9C2TP",
			"01DQSTDERHMDEY612DAAD6GFJ1",
			"01DQT4Q1HBPZ8T2E0J0JG2X1VV",
			"01DQTF0MA9JNPCRXSHY6Z2XVK2",
			"01DQTSA75VNW47JDSKTCXRSXJJ",
			"01DQV3KSZK76GYWFBR19QNXW0H",
			"01DQVDXCR7355BBZWKW0WFBQBQ",
			"01DQVR6ZHWSS64KFCH6GVP2GKX",
			"01DQW2GJBJZQ6DKHCDW9P2VWRS",
			"01DQWCT559K3CMNJ9ENA3WXW30",
			"01DQWQ3R0EDV43BXJDJ7BBVCKD",
			"01DQX1DASMR0CQ8MQ3ATX9PRE2",
			"01DQXBPXM1H3269BEMJG3VQQ30",
			"01DQXP0GD80E8CE4D9M9FCY13G",
			"01DQY0A385GN9Y1KPQEYRT2SME",
			"01DQYAKP0XFD8CPS63AZG5GPJM",
			"01DQYMX8TGXC8DDSTNVD42RZRX",
			"01DQYZ6VM066C5KD5W71RDN0Y0",
			"01DQZ9GEE9D2TFDSEN7W750X6B",
			"01DQZKT18EF69XN1CFJSR8N4GF",
			"01DQZY3M1PSPE5HZBMY0T5DYE9",
			"01DR08D6VQVA5Z02AGJH2Z7B03",
			"01DR0JPSYT58A6C59V0KC1T7SA",
			"01DR0X0CFXMB9S3138421G5WWG",
			"01DR179Z8Q2A6ZFBM3C3DFVX0P",
			"01DR1HKJ2Y2FEEG31YF3E80D7G",
			"01DR1VX5FBNVZ1FNS0HWY1G6PY",
			"01DR266QNS876AFG0ZVJ4YZ3TQ",
			"01DR2GGBA8RNXES6XDWKNYNZ59",
			"01DR2TSX9QS1S709X8GPMJXSCN",
			"01DR353G3347CSZW3X8YFXWA37",
			"01DR3FD330QC9Z94YVMBDFB3C8",
			"01DR3SPNQ5V589C9T8J02VNZPJ",
			"01DR4408H2YZ1C57FV8BZAMNYQ",
			"01DR4E9VBF0ZZT7RB0Q6143E1D",
			"01DR4RKE4S8YMXQNDMZPTYPAWA",
			"01DR52X0ZDCJFPTV4DEH3N6SS4",
			"01DR5D6KRRX5MQACFSY1JH04G1",
			"01DR5QG6JF8V79KESN8Z1JN3E1",
			"01DR61SSCFTNYD25BCV4GTFYS0",
			"01DR6C3CFP2CGS9BTG7Y61VJSG",
			"01DR6PCZ81NZ77QC2RWFSMQY21",
			"01DR70PHSZ7PNMCVVMKP8T3PHW",
			"01DR7B04KJVVFHAWR1H3ZBFW84",
			"01DR7N9QDBYCXK8Q33G6Y09BEH",
			"01DR7ZKA7259NWDAV1WH11W1C3",
			"01DR89WX1EHDH1E9JXZ7NNZAEG",
			"01DR8M6FWPEZQENQHDE50W9FBB",
			"01DR8YG2MF7SN9C6GBWMAZD6Q9",
			"01DR98SNDHH6CRSA0SYF8BW5G6",
			"01DR9K388VV9TG5R35TBNH262V",
			"01DR9XCV2FF46YJ6XS7ZTXMSA4",
			"01DRA7PE01C9KJF66Y77R54Q61",
			"01DRAJ00TV6PVFVGM80492ZTAB",
			"01DRAW9KEKXZBVVPSZ8YM0F9QF",
			"01DRB6K68H0P3YT6FYK0MACBQ9",
			"01DRBGWSFBXP73CD8H8HNVHNBW",
			"01DRBV6BXMX378G33TG0ND0NHN",
			"01DRC5FYNTCHH4DWKMNKADT4ZJ",
			"01DRCFEPW9XSXC2ZTXEWMQEWGE",
			"01DRCT34AFKZ4H7WR2RBF9WQ8Y",
			"01DRD4CQ2VCDFHXF3EWRZB6404",
			"01DRDEP9Y26H98PEQEA6B664EE",
			"01DRDRZWQNYZ901XNANCC0FG4Y",
			"01DRE39FHA0M4Z77P6VRAB9C6M",
			"01DREDK2AMT2P752ZPFA14YMVY",
			"01DREQWN4ZNKNZXEQREH9TXKEE",
			"01DRF267Y78JQV2WP0W4Q34RGG",
			"01DRFCDC36QHT1GVMSPA0STP8A",
			"01DRFPSDJQMFMC0ZVQAH7YFC80"
		],
		"parents": [
			{
				"ulid": "01DQH9PG4AEB27MDZYYHTWW9VZ",
				"minTime": 1571270400000,
				"maxTime": 1571443200000
			},
			{
				"ulid": "01DQPECYQ68YGN2P9DKMKSC62T",
				"minTime": 1571443200000,
				"maxTime": 1571616000000
			},
			{
				"ulid": "01DQVJFMMY76A83DQ9SZ7SESM7",
				"minTime": 1571616000000,
				"maxTime": 1571788800000
			},
			{
				"ulid": "01DR0RWGGJ422PNTTF9TWZDR6X",
				"minTime": 1571788800000,
				"maxTime": 1571961600000
			},
			{
				"ulid": "01DR5WS78E8T5CZ332HGR13DT0",
				"minTime": 1571961600000,
				"maxTime": 1572134400000
			},
			{
				"ulid": "01DRB3JCFCTNNS18M4P115FC6E",
				"minTime": 1572134400000,
				"maxTime": 1572307200000
			},
			{
				"ulid": "01DRGKD6ZR37QQ7BE69FN7G0HM",
				"minTime": 1572307200000,
				"maxTime": 1572480000000
			}
		]
	},
	"version": 1,
	"thanos": {
		"labels": {
			"_clusterId_": "ms-nsn-prod",
			"_replica_": "23-ms-nsn-1",
			"_shard_": "0"
		},
		"downsample": {
			"resolution": 0
		},
		"source": "compactor"
	}
}

bwplotka · 2019-11-10T20:38:04Z

cc @codesome who wanted to look at it. We discussed internally and it really looks like malformed disk writes. What do you use for disk storage? Can you compact locally?

smalldirector · 2019-11-10T23:42:20Z

Not try to compact locally as the blocks size are too large.
The compactor running on our production is using local SSD, and it will download the blocks from Ceph. The final compacted block is generated, somehow we can not read its index.

stale · 2020-01-11T04:42:48Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

smalldirector changed the title ~~Thanos compactor throw error open index file: read symbols: read symbols: invalid checksum~~ Thanos compactor throw error open index file: read symbols: read symbols: invalid checksum in compaction process Nov 6, 2019

FUSAKLA added bug component: compact labels Nov 8, 2019

zygiss mentioned this issue Dec 13, 2019

Store gateway fails to sync block meta.json #1874

Closed

stale bot added the stale label Jan 11, 2020

stale bot closed this as completed Jan 18, 2020

LeviHarrison mentioned this issue Jul 16, 2021

Compaction fails when total symbol size exceeds 2^32 bytes prometheus/prometheus#9078

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Thanos compactor throw error `open index file: read symbols: read symbols: invalid checksum` in compaction process #1728

Thanos compactor throw error `open index file: read symbols: read symbols: invalid checksum` in compaction process #1728

smalldirector commented Nov 6, 2019

FUSAKLA commented Nov 8, 2019

smalldirector commented Nov 8, 2019

FUSAKLA commented Nov 8, 2019

smalldirector commented Nov 8, 2019 •

edited

Loading

smalldirector commented Nov 8, 2019 •

edited

Loading

FUSAKLA commented Nov 9, 2019

bwplotka commented Nov 9, 2019

smalldirector commented Nov 9, 2019

bwplotka commented Nov 10, 2019

smalldirector commented Nov 10, 2019

stale bot commented Jan 11, 2020

Thanos compactor throw error open index file: read symbols: read symbols: invalid checksum in compaction process #1728

Thanos compactor throw error open index file: read symbols: read symbols: invalid checksum in compaction process #1728

Comments

smalldirector commented Nov 6, 2019

FUSAKLA commented Nov 8, 2019

smalldirector commented Nov 8, 2019

FUSAKLA commented Nov 8, 2019

smalldirector commented Nov 8, 2019 • edited Loading

smalldirector commented Nov 8, 2019 • edited Loading

FUSAKLA commented Nov 9, 2019

bwplotka commented Nov 9, 2019

smalldirector commented Nov 9, 2019

bwplotka commented Nov 10, 2019

smalldirector commented Nov 10, 2019

stale bot commented Jan 11, 2020

Thanos compactor throw error `open index file: read symbols: read symbols: invalid checksum` in compaction process #1728

Thanos compactor throw error `open index file: read symbols: read symbols: invalid checksum` in compaction process #1728

smalldirector commented Nov 8, 2019 •

edited

Loading

smalldirector commented Nov 8, 2019 •

edited

Loading