roachtest: cdc/w=100/nodes=3/init=true failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/5b5738084b8cdc769d5e7973921de5cae4457380

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=995412&tab=buildLog

The test failed on master:
	test.go:644,cdc.go:586,cdc.go:151,cdc.go:300: max latency was more than allowed: 2m7.957251825s vs 1m0s

[danhhz]

@mrtracy 1m is pretty strict for this test. I think we can just relax the success criteria

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/acd1250b15b7ed3c8938dfd53b8bc53bb53c578c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=999222&tab=buildLog

The test failed on master:
	test.go:644,cdc.go:604,cdc.go:151,cdc.go:300: max latency was more than allowed: 2m8.692858216s vs 1m0s

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/fb4a974646b8fd440ed60471e70fbfdb79d95a76

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1000678&tab=buildLog

The test failed on master:
	test.go:644,cdc.go:604,cdc.go:151,cdc.go:300: max latency was more than allowed: 11m21.181221432s vs 1m0s

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/24f6c5c295bc65e7a11b465ed9c2a399a6c19cf2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1002679&tab=buildLog

The test failed on master:
	test.go:644,cdc.go:604,cdc.go:151,cdc.go:300: max latency was more than allowed: 6m27.473186421s vs 1m0s

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/b01468f592f6b12ba147e1c19ea6d8ad7484ff6e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1007980&tab=buildLog

The test failed on release-2.1:
	test.go:644,cdc.go:604,cdc.go:151,cdc.go:300: max latency was more than allowed: 2m39.085421451s vs 1m0s

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/96aafe6226579176f496dfadae78b52d687c3faa

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1008356&tab=buildLog

The test failed on master:
	test.go:639,cluster.go:1152,cdc.go:386,cdc.go:72,cdc.go:300: /home/agent/work/.go/bin/roachprod run teamcity-1008356-cdc-w-100-nodes-3-init-true:4 -- yes | sudo apt-get -q install default-jre returned:
		stderr:
		
		stdout:
		.5 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/p/pulseaudio/libpulse0_8.0-0ubuntu3.10_amd64.deb  503  Service Unavailable [IP: 35.184.213.5 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/j/java-common/default-jre_1.8-56ubuntu2_amd64.deb  503  Service Unavailable [IP: 35.184.213.5 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/f/fonts-dejavu/fonts-dejavu-extra_2.35-1_all.deb  503  Service Unavailable [IP: 35.184.213.5 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/h/hicolor-icon-theme/hicolor-icon-theme_0.15-0ubuntu1.1_all.deb  503  Service Unavailable [IP: 35.184.213.5 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/g/gtk+2.0/libgtk2.0-bin_2.24.30-1ubuntu1.16.04.2_amd64.deb  503  Service Unavailable [IP: 35.184.213.5 80]
		
		E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
		Error:  exit status 100
		: exit status 1

[tbg]

^- I just merged an updated xenial image, which may or may not improve this. Anecdotally it has helped in the past.

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/02f35d1dd3507bf34c99b4f5ed6ae05699499799

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1012425&tab=buildLog

The test failed on release-2.1:
	test.go:650,cdc.go:658,cdc.go:176,cdc.go:325: max latency was more than allowed: 6m28.809955897s vs 1m0s

[danhhz]

The "max latency was more than allowed" is being caused by #32799