storage: scatter fails with "pending snapshot already present"

[petermattis]

Saw this error when testing against sky. The error was returned from ALTER TABLE test.kv SCATTER when doing kv --concurrency 512 --splits 100000.

2017/08/09 01:27:10 pq: [n46,s48,r4368/2:/Table/51/1/-7566{8738…-4507…}]: can't set pending snapshot index to 1; pending snapshot already present: 47

[benesch]

@petermattis think this was fixed by #17644?

[petermattis]

No, I think this is interaction between adminScatter and replicateQueue. It isn't particularly harmful, but sort of a wart. I think we could fix it by adding a Replica.maintenanceMu or perhaps Replica.maintenanceCh that is acquired whenever a snapshot is in flight.

[petermattis]

Haven't seen this in a long time. It is possible something fixed this. Closing as not reproducible.