TiKV txn benchmark does not retry retriable error

[yiwu-arbug]

Ran go-ycsb against tikv txn API, and there's significant portion of updates get update error. In particular, these are the command and param I'm using:

./bin/go-ycsb load tikv -P workloads/workloada -p tikv.pd="172.16.30.40:2379" -p tikv.type="txn" -p recordcount=10000000
./bin/go-ycsb run tikv -P workloads/workloada -p tikv.pd="172.16.30.40:2379" -p tikv.type="txn" -p recordcount=10000000 -p readproportion=0 -p updateproportion=1.0 -p operationcount=10000000 -p threadcount=1000

In general this is preload an empty tikv with 10M rows, and then try to do 10M updates from 1k threads. During the run phase, ycsb would report 15% update error:

UPDATE_ERROR - Takes(s): 242.8, Count: 1537013, OPS: 6329.8, Avg(us): 46655, Min(us): 1059, Max(us): 321298, 95th(us): 104000, 99th(us): 113000
UPDATE - Takes(s): 242.8, Count: 8462987, OPS: 34852.4, Avg(us): 19998, Min(us): 1381, Max(us): 499397, 95th(us): 36000, 99th(us): 47000

In fact those errors are all retriable errors of "get timestamp too slow". When I patch with https://github.com/yiwu-arbug/go-ycsb/commit/344243b1a04160cff96d717ef3a4f5fb102b34d7 the error rate drops significantly (but also QPS):

UPDATE - Takes(s): 739.7, Count: 9994297, OPS: 13511.0, Avg(us): 65915, Min(us): 1148, Max(us): 9848521, 95th(us): 15000, 99th(us): 0
UPDATE_ERROR - Takes(s): 731.1, Count: 5703, OPS: 7.8, Avg(us): 8625979, Min(us): 7012194, Max(us): 10176360, 95th(us): 0, 99th(us): 0

The remaining errors are still all of the same retriable "get timestamp too slow" error. I'm feeling that go-ycsb should retry these errors to simulate real scenario, where a normal client would retry before giving up.

[yiwu-arbug]

cc @siddontang

[yiwu-arbug]

I'm not blocking on this, but feeling the high error rate very confusing and would like to leave an issue here if someone else wondering the same.

[siddontang]

@disksing

seem we can support this now with client-go.