roachtest: import/tpch/nodes=8 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/66f13c1d9a12c31e18a198da4ff5ac0bbe2db781

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=import/tpch/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1127849&tab=buildLog

The test failed on provisional_201902062125_v2.2.0-alpha.20190211:
	test.go:743,cluster.go:1585,import.go:150: pq: gs://cockroach-fixtures/tpch-csv/sf-100/lineitem.tbl.2: row 16142300: reading CSV record: read tcp 10.138.0.8:39328->173.194.203.128:443: read: connection reset by peer

[asubiotto]

Seems like a node crashed but can't find anything in the logs or dmesg. My first instinct is to blame #34241 but don't have much context.

[petermattis]

Blaming #34241 without evidence is a bit unfair to that poor issue. The other instances of #34241 have always had a stack trace.

The error message suggests to me a problem reading from GCS. @dt or @mjibson is there anything to be done here. We have retry loops on the reads from cloud providers, right?

[asubiotto]

You're right! Sorry.

Seeing something similar with #34693 and #34700 possibly.

[dt]

we have a retry around the creation of the Reader but probably not everywhere that we read from it -- we don't currently wrap the Read method on any of the SDKs so it would just be whatever they have internally.

[maddyblue]

Yes. This was a failure during read mid file. We can't retry ourselves because we don't know the number of bytes that have been read and can't resume from there. Some SDKs do that automatically, including I think GCS. I'm not sure what to do here besides upgrade to the latest GCS SDK and see if they had any improvements in internal retrying.

[dt]

yeah, I'm fine just closing the occasional failure due to external cloud/connectivity issues.

If it were frequent and came up in actual usage, I might want add more retries. As is, I'd rather assume the cloud readers aren't too flaky and if we get an error from one, say so right away -- since maybe they should fix their network config or something -- instead of sitting in a retry loop.