Shows speculative execution in official detail dashboard

[glommer]

I have debugged countless situations where speculative retries add considerable amount of load to the cluster.

There are workloads that are really adversarial to it, and while it usually adds a bit of load to the cluster , I've just seen a scenario in which it added 200% more load to the cluster. You've read it right: speculative retry added increased the load in the cluster by 200%.

@amnonh these are the metrics that are relevant. I will defer to you about whether or not we should add two graphs or combine them:

• scylla_storage_proxy_coordinator_speculative_data_reads : shows the amount of times we had to do actual reads on behalf of speculative retry (more expensive)
• scylla_storage_proxy_coordinator_speculative_digest_reads: shows the amount of times only a digest was necessary (much cheaper)

Look at this beautiful thing: