[Traceql Metrics] PR 4 - Sampling

[mdisibio]

What this PR does:
This PR introduces subsampling hints for TraceQL metrics queries. I.e. inspect a random 50% of data and then scale the resulting rates/counts by 2x. This allows us to trade a customizable amount of accuracy for speed. In internal testing it can be surprisingly accurate down to even ~10%. Of course it depends on the query.

It is controlled through a new system of query hints which can be applied to any query. The new hint is named sample and takes a float:

Sample 100% (default behavior):
{ } | rate()

Sample 10%:
{ } | rate() with(sample=0.1)

Overall design
The sampling rate is enacted by manipulating the existing job shards to cover less data. For example, take the job for shard 1 of 10. This covers 10% of trace IDs. To sample this by 50%, we convert it to be shard 2 of 20. Now it only covers 5% of traces IDs (exactly the latter half of previous range). Then the results are multiplied back to get the final metrics. I like this approach because it maintains the uniform inspection of data across the entire range, and it happens purely through the query-frontend layer. A drawback is that the total number of jobs doesn't change. This will continue to be a bottleneck for large requests.

Hints
The new hints system is added generically. You can add with(key=val, key2=val2...) to any query. There is no validation, so unsupported hints or using sample=... with a non-metrics query is simply ignored. It supports all TraceQL value types, so we can have hints with strings, ints, etc. I think this could be useful and I already have a few ideas for future hints.

Other Changes

1. Discovered that 8-byte trace IDs may be 63-bits of randomness only. Fixing the trace ID sharding to account for this gives a huge accuracy boost when sampling because shards are correctly weighted. Very important to the usefulness of it.
2. Generator request has a new param mode so we can do sampling rates on the generators too.

Alternatives
There are other ways to inspect only 50% of the data (for example).

1. We could drop half of the jobs. Instead of executing all 10 shards, we only execute 1 through 5. The drawbacks to the approach are: (a) statistically less accurate because we wouldn't be sampling the traceID space uniformly. (b) the performance gains don't materialize as well because the minimum job size remains the same. Increased parallelization and scale won't necessarily make it faster.
2. modulus() or rand(). This would read every-other-trace on average. This sounds great in practice but it has no reduction in I/O, and would end up being both less accurate and no faster.

Notes
This is one entry in a set of chained PRs. The full body of work has been split into separate buckets to make reviews and updates more manageable.

1. [Traceql Metrics] PR 1 - Engine #3251
2. [Traceql Metrics] PR 2 - API #3252
3. [Traceql Metrics] PR 3 - Trace ID sharding #3258
4. [Traceql Metrics] PR 4 - Sampling #3275

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[zalegrala]

This looks good to me. The lint has a few things to say and I left a comment about the query mode string.