roachtest: ledger/nodes=6/multi-az failed

[cockroach-teamcity]

roachtest.ledger/nodes=6/multi-az failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

(cluster.go:2419).Run: full command output in run_104636.223518519_n7_workload-run-ledger-.log: COMMAND_PROBLEM: exit status 1
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ledger/nodes=6/multi-az/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-40887

[kvoli]

The fatal is from:

Error: error in deposit: pq: result is ambiguous: error=ba: EndTxn(parallel commit) [/Table/106/1/690/0], [txn: a80a69c5] RPC error: grpc: error reading from server: read tcp 10.128.0.23:44714->10.128.0.18:26257: use of closed network connection [code 14/Unavailable] [propagate] (last error: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=a80a69c5 key=/Table/106/1/690/0 iso=Serializable pri=0.05341051 epo=0 ts=1722682326.039554249,0 min=1722682326.039554249,0 seq=8} lock=true stat=ABORTED rts=1722682326.039554249,0 wto=false gul=1722682326.539554249,0)

Also see async rollbacks failing on both n1 and n2:

I240803 10:52:15.515602 8297970 kv/txn.go:982 ⋮ [T1,Vsystem,n1] 490  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [/Table/106/1/‹288›/‹0›], [txn: c86b2012]
...
I240803 10:52:15.524111 8297977 kv/txn.go:982 ⋮ [T1,Vsystem,n1] 502  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [/Table/106/1/‹763›/‹0›], [txn: bca0e44c]

Overall, I'm not sure what when wrong here, aside from there being signs of overload. The concurrent txn use detected is quite suspect that there could be a bug. I'll triage further and leave the release blocker label for now.

[arulajmani]

We talked about this in yesterdays team meeting, and the concurrent txn use detected bit is benign. (See #75209)

@kvoli, given our discussions yesterday re: ambiguous errors, I'll close this out and chalk it to #107571.