roachtest: tpcc/mixed-headroom/n5cpu16 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/1ca35fc4a0e2665e7f6efd945e65a0db97984fa7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1396096&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190719-1396096/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1726,tpcc.go:162,tpcc.go:164,tpcc.go:277,test_runner.go:691: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1563517204-52-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		I190719 17:51:34.859159 1 ccl/workloadccl/cliccl/fixtures.go:293  starting load of 9 tables
		I190719 17:51:36.253446 126 ccl/workloadccl/fixture.go:476  loaded 7.8 MiB table item in 1.394008666s (100000 rows, 0 index entries, 5.6 MiB)
		I190719 17:53:04.141747 120 ccl/workloadccl/fixture.go:476  loaded 53 KiB table warehouse in 1m29.281735477s (1000 rows, 0 index entries, 607 B)
		I190719 17:53:45.829269 121 ccl/workloadccl/fixture.go:476  loaded 1006 KiB table district in 2m10.969846486s (10000 rows, 0 index entries, 7.7 KiB)
		Error: restoring fixture: dial tcp 10.128.0.18:26257: connect: connection refused
		Error:  exit status 1
		: exit status 1

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/1ad0ecc8cbddf82c9fedb5a5c5e533e72a657ff7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/mixed-headroom/n5cpu16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1399000&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190722-1399000/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:1726,tpcc.go:162,tpcc.go:164,tpcc.go:277,test_runner.go:691: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1563776264-53-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		stderr:
		
		stdout:
		I190722 20:07:34.335336 1 ccl/workloadccl/cliccl/fixtures.go:293  starting load of 9 tables
		Error: restoring fixture: pq: no minimum transaction timestamp provided: "unnamed" id=c6c1732f key=/Table/15/1/471058529044168705/0 rw=true pri=0.01651538 stat=PENDING epo=0 ts=1563826054.615537440,0 orig=1563826054.615537440,0 min=0.000000000,0 max=1563826054.617756514,0 wto=false seq=4
		Error:  exit status 1
		: exit status 1

[tbg]

@nvanbenschoten

  Error: restoring fixture: pq: no minimum transaction timestamp provided: "unnamed" id=c6c1732f key=/Table/15/1/471058529044168705/0 rw=true pri=0.01651538 stat=PENDING epo=0 ts=1563826054.615537440,0 orig=1563826054.615537440,0 min=0.000000000,0 max=1563826054.617756514,0 wto=false seq=4

I looked at the PR introducing this (#38782) and there isn't a migration. Isn't that an oversight? MinTimestamp is populated when the txn proto is made, i.e. not necessarily by the node evaluating CanCreateTxnRecord.

[nvanbenschoten]

Yes, that's an oversight. I must have been thinking about the behavior we'll want in the next release. It's strange because I was clearly thinking about compatibility in this area of code. For now, we should continue to use InclusiveTimeBounds there. I'll send out a PR.