Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flaky TestSampleTracker_Concurrency #10482

Open
NickAnge opened this issue Jan 20, 2025 · 3 comments
Open

flaky TestSampleTracker_Concurrency #10482

NickAnge opened this issue Jan 20, 2025 · 3 comments

Comments

@NickAnge
Copy link
Contributor

--- FAIL: TestSampleTracker_Concurrency (0.00s)
    sample_tracker_test.go:154: 
        	Error Trace:	/__w/mimir/mimir/pkg/costattribution/sample_tracker_test.go:154
        	Error:      	Received unexpected error:
        	            	 # HELP cortex_discarded_attributed_samples_total The total number of samples that were discarded per attribution.
        	            	 # TYPE cortex_discarded_attributed_samples_total counter
        	            	-cortex_discarded_attributed_samples_total{reason="__overflow__",team="__overflow__",tenant="user1",tracker="cost-attribution"} 87
        	            	+cortex_discarded_attributed_samples_total{reason="__overflow__",team="__overflow__",tenant="user1",tracker="cost-attribution"} 95
        	            	 # HELP cortex_distributor_received_attributed_samples_total The total number of samples that were received per attribution.
        	            	 # TYPE cortex_distributor_received_attributed_samples_total counter
        	            	-cortex_distributor_received_attributed_samples_total{team="__overflow__",tenant="user1",tracker="cost-attribution"} 87
        	            	+cortex_distributor_received_attributed_samples_total{team="__overflow__",tenant="user1",tracker="cost-attribution"} 95
        	            	 
        	Test:       	TestSampleTracker_Concurrency
FAIL
FAIL	github.com/grafana/mimir/pkg/costattribution	0.090s

https://github.com/grafana/mimir/actions/runs/12872353056/job/35887678041

(In Github CI)

@56quarters
Copy link
Contributor

Another instance:

 --- FAIL: TestSampleTracker_Concurrency (0.00s)
    sample_tracker_test.go:154: 
        	Error Trace:	/__w/mimir/mimir/pkg/costattribution/sample_tracker_test.go:154
        	Error:      	Received unexpected error:
        	            	 # HELP cortex_discarded_attributed_samples_total The total number of samples that were discarded per attribution.
        	            	 # TYPE cortex_discarded_attributed_samples_total counter
        	            	-cortex_discarded_attributed_samples_total{reason="__overflow__",team="__overflow__",tenant="user1",tracker="cost-attribution"} 93
        	            	+cortex_discarded_attributed_samples_total{reason="__overflow__",team="__overflow__",tenant="user1",tracker="cost-attribution"} 95
        	            	 # HELP cortex_distributor_received_attributed_samples_total The total number of samples that were received per attribution.
        	            	 # TYPE cortex_distributor_received_attributed_samples_total counter
        	            	-cortex_distributor_received_attributed_samples_total{team="__overflow__",tenant="user1",tracker="cost-attribution"} 93
        	            	+cortex_distributor_received_attributed_samples_total{team="__overflow__",tenant="user1",tracker="cost-attribution"} 95
        	            	 
        	Test:       	TestSampleTracker_Concurrency
FAIL
FAIL	github.com/grafana/mimir/pkg/costattribution	0.080s

https://github.com/grafana/mimir/actions/runs/12917443513/job/36023847743?pr=10502

@56quarters
Copy link
Contributor

@ying-jeanne Can you take a look at this test?

@56quarters
Copy link
Contributor

I've disabled the test as part of this PR for now since it failed CI a few times in a row.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants