roachtest: cdc/w=100/init=false/crdbChaos=true failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/02f35d1dd3507bf34c99b4f5ed6ae05699499799

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/init=false/crdbChaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1012425&tab=buildLog

The test failed on release-2.1:
	test.go:645,cluster.go:1159,cdc.go:527,cdc.go:98,cluster.go:1481,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-1012425-cdc-w-100-init-false-crdbchaos-true:4 -- ./workload run tpcc --warehouses=100 --duration=30m --tolerate-errors {pgurl:1-3}  returned:
		stderr:
		
		stdout:
		.0           17.4      0.0      0.0      0.0      0.0 newOrder
		   4m20s      524            0.0            1.7      0.0      0.0      0.0      0.0 orderStatus
		   4m20s      524            0.0           17.9      0.0      0.0      0.0      0.0 payment
		   4m20s      524            0.0            1.7      0.0      0.0      0.0      0.0 stockLevel
		E181114 20:29:09.617584 1 workload/cli/run.go:398  error in payment: dial tcp 10.128.0.40:26257: connect: connection refused
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   4m21s      551            0.0            1.8      0.0      0.0      0.0      0.0 delivery
		   4m21s      551            0.0           17.4      0.0      0.0      0.0      0.0 newOrder
		   4m21s      551            0.0            1.7      0.0      0.0      0.0      0.0 orderStatus
		   4m21s      551            0.0           17.8      0.0      0.0      0.0      0.0 payment
		   4m21s      551            0.0            1.7      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	test.go:645,cluster.go:1502,cdc.go:174,cdc.go:359: unexpected status: failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/eb8345b19cf6c15b3e4fcb9c156136357d83cb2d

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/init=false/crdbChaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1015656&tab=buildLog

The test failed on master:
	test.go:650,cdc.go:668,cdc.go:181,cdc.go:371: max latency was more than allowed: 10m11.629040916s vs 10m0s

[danhhz]

The "max latency was more than allowed" is being caused by #32799