Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpcc/mixed-headroom/n5cpu16 failed #50439

Closed
cockroach-teamcity opened this issue Jun 19, 2020 · 8 comments
Closed

roachtest: tpcc/mixed-headroom/n5cpu16 failed #50439

cockroach-teamcity opened this issue Jun 19, 2020 · 8 comments
Assignees
@dt
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
20.2

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@2dd15cd26f95801c678b1034e98a15560dc10baa:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:266
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_121944.528_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2023526-1592547695-79-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200619 12:19:46.388982 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200619 12:19:46.694321 95 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 297.60686ms (1000 rows, 0 index entries, 178 KiB)
		  | I200619 12:19:59.038726 133 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 12.642132225s (100000 rows, 0 index entries, 629 KiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2023526-1592547695-79-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: 4228
		1: 4365
		4: 4325
		3: dead
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString
More

Artifacts: /tpcc/mixed-headroom/n5cpu16

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jun 19, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Jun 19, 2020
@andreimatei
Copy link
Contributor

@nvanbenschoten have you seen this timeout while loading tpcc before?

12:19:44 cluster.go:2104: [w5] > ./workload fixtures load tpcc --warehouses=1000  {pgurl:1}
22:18:17 test_runner.go:773: [w5] dumped stacks to __stacks
22:18:17 test.go:190: test status: stopping cluster
22:18:17 cluster.go:348: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod stop --sig=11 teamcity-2023526-1592547695-79-n5cpu16:1-5
teamcity-2023526-1592547695-79-n5cpu16: stopping
22:18:18 test.go:325: test failure: 	test_runner.go:800: test timed out (10h0m0s)

@cockroach-teamcity
Copy link
Member Author

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@17c8048e80935f8a01477416980d18bf39cba1bb:


	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:829: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2046708-1593411872-92-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: 4234
		2: 4226
		4: dead
		1: dead
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 1: dead
		  | (1) attached stack trace
		  |   | main.glob..func13
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:203
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		  | Wraps: (2) 3 safe details enclosed
		  | Wraps: (3) 1: dead
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errors.errorString
		Wraps: (3) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) 3 safe details enclosed
		Wraps: (5) 4: dead
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *safedetails.withSafeDetails (5) *errors.errorString
More

Artifacts: /tpcc/mixed-headroom/n5cpu16

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten
Copy link
Member

No, I haven't seen this. But we just hit it again. We load the first four tables quickly and then stall for 10 hours during the restore:

I200629 12:01:19.894893 104 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 287.425325ms (1000 rows, 0 index entries, 184 KiB)
I200629 12:01:20.283205 110 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 675.671902ms (100000 rows, 0 index entries, 12 MiB)
I200629 12:03:03.671041 105 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 1m44.063521843s (10000 rows, 0 index entries, 9.7 KiB)
I200629 12:03:28.731521 109 ccl/workloadccl/fixture.go:586  loaded 126 MiB table new_order in 2m9.123945655s (9000000 rows, 0 index entries, 996 KiB)

@andreimatei
Copy link
Contributor

@dt do you want to triage this one?
We don't see any SST ingestions after a while.

@andreimatei andreimatei assigned dt and unassigned andreimatei Jun 30, 2020
@cockroach-teamcity cockroach-teamcity mentioned this issue Jul 1, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@456a07cfc1e53b87abc7709052e54efb1450e758:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:266
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_140242.027_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2062841-1593843322-81-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200704 14:02:43.944576 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200704 14:02:44.786154 109 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 836.584785ms (100000 rows, 0 index entries, 9.3 MiB)
		  | I200704 14:03:33.315469 104 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 49.365926437s (10000 rows, 0 index entries, 20 KiB)
		  | I200704 14:07:21.285606 103 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 4m37.335929222s (1000 rows, 0 index entries, 195 B)
		  | I200704 14:08:22.123524 107 ccl/workloadccl/fixture.go:586  loaded 1.3 GiB table order in 5m38.174011466s (30000000 rows, 30000000 index entries, 4.0 MiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:829: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2062841-1593843322-81-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		1: 4307
		2: 4273
		4: 4215
		3: dead
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString
More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@3e0de239121813ea4d47873388a2828a66d9edf7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	test_runner.go:804: test timed out (10h0m0s)

	cluster.go:2040,tpcc.go:136,tpcc.go:138,tpcc.go:149,tpcc.go:266,test_runner.go:757: output in run_144615.214_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2063962-1593929497-86-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2115
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2038
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:266
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_144615.214_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2063962-1593929497-86-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200705 14:46:17.130738 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200705 14:47:00.563003 80 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 43.421342245s (100000 rows, 0 index entries, 183 KiB)
		  | I200705 14:48:37.647977 75 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 2m20.506295436s (10000 rows, 0 index entries, 7.2 KiB)
		  | I200705 14:49:30.404756 74 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 3m13.262953991s (1000 rows, 0 index entries, 280 B)
		  | I200705 14:50:53.688110 79 ccl/workloadccl/fixture.go:586  loaded 126 MiB table new_order in 4m36.546312836s (9000000 rows, 0 index entries, 465 KiB)
		  | I200705 14:53:32.768904 78 ccl/workloadccl/fixture.go:586  loaded 1.3 GiB table order in 7m15.627154221s (30000000 rows, 30000000 index entries, 3.1 MiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString
More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Jul 8, 2020
Closed
@nvanbenschoten nvanbenschoten mentioned this issue Jul 8, 2020
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@542c83e7db1694208e09c7fea7d350db4993b897:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	test_runner.go:804: test timed out (10h0m0s)

	cluster.go:2044,tpcc.go:147,tpcc.go:157,tpcc.go:168,tpcc.go:286,test_runner.go:757: output in run_105937.937_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2079204-1594361493-103-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2119
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2042
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:147
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:286
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_105937.937_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2079204-1594361493-103-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200710 10:59:40.002187 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200710 10:59:40.529598 26 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 515.469778ms (1000 rows, 0 index entries, 103 KiB)
		  | I200710 10:59:41.072382 32 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 1.058162925s (100000 rows, 0 index entries, 7.3 MiB)
		  | I200710 11:00:52.413964 27 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 1m12.399854281s (10000 rows, 0 index entries, 14 KiB)
		  | I200710 11:01:31.082656 31 ccl/workloadccl/fixture.go:586  loaded 126 MiB table new_order in 1m51.068459109s (9000000 rows, 0 index entries, 1.1 MiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString
More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Jul 11, 2020
Closed
@irfansharif irfansharif mentioned this issue Jul 13, 2020
Closed
@dt
Copy link
Member

dt commented Jul 17, 2020 •
edited
Loading

The earlier iterations of this are almost certainly fixed by #51085.

The most recent is after that merged, and it is hung on an AdminSplit call. We see the same split key over and over in the logs with increasing range IDs. Any chance this is the "stuck split" I've heard of? It isn't quite what I heard was the symptom to look for, in that there are no goroutines in redirectOnOrAcquireLease. Closing for now unless KV wants to dig into the stuck AdminSplit call more.

@dt dt closed this as completed Jul 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dt dt

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
20.2
Development

No branches or pull requests
4 participants
@dt @andreimatei @nvanbenschoten @cockroach-teamcity