roachtest: tpcc/multiregion/survive=region/chaos=true failed

[cockroach-teamcity]

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.2 @ 4ae29e143ad60de4ae66314055ab43e8a976c85a:

test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
(test_impl.go:286).Fatal: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 1)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/multiregion

This test on roachdash | Improve this report!

Jira issue: CRDB-24565

[ajwerner]

E230215 16:03:41.973180 1 1@cli/clierror/check.go:35 ⋮ [-] 72 +cockroach server exited with error: error recording initial status summaries: replica unavailable: (n6,s6):5 unable to serve request to r3:‹/System/{NodeLivenessMax-tsd}› [(n1,s1):1, (n2,s2):2, (n3,s3):3, (n9,s9):4, (n6,s6):5, next=6, gen=8]: lost quorum (down: (n1,s1):1,(n2,s2):2,(n3,s3):3); closed timestamp: 1676476712.240471697,0 (2023-02-15 15:58:32); raft status: {"id":"5","term":7,"vote":"5","commit":1288,"lead":"5","raftState":"StateLeader","applied":1288,"progress":{"1":{"match":1288,"next":1289,"state":"StateProbe"},"2":{"match":1288,"next":1289,"state":"StateProbe"},"3":{"match":1288,"next":1289,"state":"StateProbe"},"4":{"match":1562,"next":1563,"state":"StateReplicate"},"5":{"match":1562,"next":1563,"state":"StateReplicate"}},"leadtransferee":"0"}: ‹operation "probe" timed out after 1m0s (given timeout 1m0s)›: result is ambiguous: after 60.00s of attempting command: context deadline exceeded

[ajwerner]

Given how early this is in startup of the test, this is either server or test eng. Moving to test eng.

[blathers-crl]

cc @cockroachdb/test-eng

[renatolabs]

This seems to be an instance of #74714.