Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky TestIngester_PrepareInstanceRingDownscaleHandler #9494

Closed
dimitarvdimitrov opened this issue Oct 2, 2024 · 10 comments · Fixed by #9663, #9960 or #10528
Closed

Flaky TestIngester_PrepareInstanceRingDownscaleHandler #9494

dimitarvdimitrov opened this issue Oct 2, 2024 · 10 comments · Fixed by #9663, #9960 or #10528
Labels
type/tests

Comments

@dimitarvdimitrov
Copy link
Contributor 

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.00s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:[72](https://github.com/grafana/mimir/actions/runs/11141470952/job/30962417169?pr=9493#step:8:73)90
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors2045435997/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors2045435997/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL
@dimitarvdimitrov
Copy link
Contributor Author

@jhalterman in case this is still fresh after you wrote it. Does anything come to mind?

@dimitarvdimitrov
Copy link
Contributor Author

another flake

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.01s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1375604059/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1375604059/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL

@zenador
Copy link
Contributor

zenador commented Oct 17, 2024

https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.00s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:[72](https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655#step:8:73)90
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors3861206[84](https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655#step:8:85)4/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors3[86](https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655#step:8:87)1206844/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"

@jhalterman jhalterman mentioned this issue Oct 18, 2024
Merged
4 tasks
@jhalterman
Copy link
Member

This type of test where we're waiting for ring state to converge can unfortunately take a while. We might consider mocking the ring communication, but since other tests just perform long polling, we could try increasing the polling duration which should help with flakiness in CI.

@narqo
Copy link
Contributor

narqo commented Oct 23, 2024

Seems that it wasn't fixed. Just saw a flake in this build:

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.01s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1968523047/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1968523047/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL
FAIL	github.com/grafana/mimir/pkg/ingester	590.242s

@narqo narqo reopened this Oct 23, 2024
@seizethedave
Copy link
Contributor

Got one from a freshly updated main. https://github.com/grafana/mimir/actions/runs/11827228735/job/32954945570

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.01s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
method=Ingester.Shipper.Sync level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
method=Ingester.Shipper.Sync level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors6698743[85](https://github.com/grafana/mimir/actions/runs/11827228735/job/32954945570#step:8:86)/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors669[87](https://github.com/grafana/mimir/actions/runs/11827228735/job/32954945570#step:8:88)4385/002"
method=Ingester.Shipper.Sync level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL
FAIL	github.com/grafana/mimir/pkg/ingester	599.710s

@dimitarvdimitrov
Copy link
Contributor Author

dimitarvdimitrov commented Nov 14, 2024 •
edited
Loading

happened again

Details 

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/DELETE_request_should_switch_the_instance_ring_entry_to_not_read-only (0.02s)
        downscale_test.go:100: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:100
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:98
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/DELETE_request_should_switch_the_instance_ring_entry_to_not_read-only

@dimitarvdimitrov
Copy link
Contributor Author

Details 

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (4.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/DELETE_request_should_switch_the_instance_ring_entry_to_not_read-only (0.01s)
        downscale_test.go:100: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:100
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:98
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/DELETE_request_should_switch_the_instance_ring_entry_to_not_read-only
method=Ingester.Shipper.Sync level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
method=Ingester.Shipper.Sync level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1889[80](https://github.com/grafana/mimir/actions/runs/11881110281/job/33105076861?pr=9926#step:8:81)5590/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1889805590/002"
method=Ingester.Shipper.Sync level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL

@jhalterman jhalterman mentioned this issue Nov 20, 2024
Merged
4 tasks
@dimitarvdimitrov
Copy link
Contributor Author

This flaked again :(

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (4.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.00s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:72[90](https://github.com/grafana/mimir/actions/runs/12008433899/job/33471095125?pr=10016#step:8:91)
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
method=Ingester.Shipper.Sync level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
method=Ingester.Shipper.Sync level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors4124[93](https://github.com/grafana/mimir/actions/runs/12008433899/job/33471095125?pr=10016#step:8:94)3474/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors4124933474/002"
method=Ingester.Shipper.Sync level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL

@NickAnge
Copy link
Contributor

NickAnge commented Dec 9, 2024

This flaked again unfortunately (Ref)

-- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (4.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.01s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
method=Ingester.Shipper.Sync level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
method=Ingester.Shipper.Sync level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors291255[85](https://github.com/grafana/mimir/actions/runs/12236674870/job/34130678395?pr=10185#step:8:86)/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors29125585/002"
method=Ingester.Shipper.Sync level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL

@56quarters 56quarters mentioned this issue Jan 28, 2025
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/tests
Projects
None yet
Milestone
No milestone
6 participants
@narqo @seizethedave @jhalterman @zenador @dimitarvdimitrov @NickAnge