Flaky Test: TestProbesWhileOperating in Unit Test (Race)

[rohit-nayak-ps]

Overview of the Issue

2023-12-13T15:17:59.0244245Z E1213 15:15:44.918054   52087 throttler.go:915] refreshMySQLInventory: context canceled
2023-12-13T15:17:59.0245024Z E1213 15:15:44.918079   52087 throttler.go:915] refreshMySQLInventory: context canceled
2023-12-13T15:17:59.0245545Z ==================
2023-12-13T15:17:59.0245794Z WARNING: DATA RACE
2023-12-13T15:17:59.0246082Z Write at 0x0000031c5050 by goroutine 5736:
2023-12-13T15:17:59.0246672Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.newTestThrottler()
2023-12-13T15:17:59.0247688Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler_test.go:101 +0x21c
2023-12-13T15:17:59.0248631Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.TestProbesWhileOperating()
2023-12-13T15:17:59.0249868Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler_test.go:354 +0x29
2023-12-13T15:17:59.0250552Z   testing.tRunner()
2023-12-13T15:17:59.0251072Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1595 +0x261
2023-12-13T15:17:59.0251561Z   testing.(*T).Run.func1()
2023-12-13T15:17:59.0252108Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1648 +0x44
2023-12-13T15:17:59.0252478Z 
2023-12-13T15:17:59.0252643Z Previous read at 0x0000031c5050 by goroutine 83:
2023-12-13T15:17:59.0253352Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.(*Throttler).refreshMySQLInventory()
2023-12-13T15:17:59.0254402Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler.go:862 +0x18f
2023-12-13T15:17:59.0255309Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.(*Throttler).Operate.func2()
2023-12-13T15:17:59.0256558Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler.go:727 +0xa47
2023-12-13T15:17:59.0257101Z 
2023-12-13T15:17:59.0257230Z Goroutine 5736 (running) created at:
2023-12-13T15:17:59.0257563Z   testing.(*T).Run()
2023-12-13T15:17:59.0258074Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1648 +0x845
2023-12-13T15:17:59.0258562Z   testing.runTests.func1()
2023-12-13T15:17:59.0259193Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:2054 +0x84
2023-12-13T15:17:59.0259651Z   testing.tRunner()
2023-12-13T15:17:59.0260144Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1595 +0x261
2023-12-13T15:17:59.0260616Z   testing.runTests()
2023-12-13T15:17:59.0261106Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:2052 +0x8ad
2023-12-13T15:17:59.0261665Z   testing.(*M).Run()
2023-12-13T15:17:59.0262143Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1925 +0xcd7
2023-12-13T15:17:59.0262597Z   main.main()
2023-12-13T15:17:59.0262886Z       _testmain.go:61 +0x2bd
2023-12-13T15:17:59.0263079Z 
2023-12-13T15:17:59.0263210Z Goroutine 83 (finished) created at:
2023-12-13T15:17:59.0263792Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.(*Throttler).Operate()
2023-12-13T15:17:59.0264774Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler.go:660 +0xbf0
2023-12-13T15:17:59.0265644Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.(*Throttler).Enable()
2023-12-13T15:17:59.0266594Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler.go:441 +0x1d8
2023-12-13T15:17:59.0267431Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.runThrottler()
2023-12-13T15:17:59.0268396Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler_test.go:328 +0x1de
2023-12-13T15:17:59.0269231Z   vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.TestRace()
2023-12-13T15:17:59.0270158Z       /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler_test.go:349 +0x3d
2023-12-13T15:17:59.0270806Z   testing.tRunner()
2023-12-13T15:17:59.0271312Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1595 +0x261
2023-12-13T15:17:59.0271786Z   testing.(*T).Run.func1()
2023-12-13T15:17:59.0272300Z       /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1648 +0x44
2023-12-13T15:17:59.0272742Z ==================
2023-12-13T15:17:59.0273232Z --- FAIL: TestProbesWhileOperating (5.00s)
2023-12-13T15:17:59.0273740Z     testing.go:1465: race detected during execution of test
2023-12-13T15:17:59.0274117Z FAIL
2023-12-13T15:17:59.0274477Z FAIL	vitess.io/vitess/go/vt/vttablet/tabletserver/throttle	10.078s

[rohit-nayak-ps]

Seeing this in again in main today

Goroutine 83 (finished) created at:
  vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.(*Throttler).Operate()
      /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler.go:660 +0xbf0
  vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.(*Throttler).Enable()
      /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler.go:441 +0x1d8
  vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.runThrottler()
      /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler_test.go:330 +0x1de
  vitess.io/vitess/go/vt/vttablet/tabletserver/throttle.TestRace()
      /home/runner/work/vitess/vitess/go/vt/vttablet/tabletserver/throttle/throttler_test.go:351 +0x3d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1648 +0x44
==================
--- FAIL: TestProbesWhileOperating (5.00s)
    testing.go:1465: race detected during execution of test
FAIL
FAIL	vitess.io/vitess/go/vt/vttablet/tabletserver/throttle	10.074s

[shlomi-noach]

Ah, sorry, didn't realize this issue existed. #14852 is a duplicate of this issue.