opt: performance regression on April 21st

[erikgrinaker]

roachperf graphs are showing a performance regression across the board on April 21st, e.g.:

Verified this to be caused by #63807, benchmarks of 22e4d64 (old) vs df67aa9 (new):

name                           old ops/sec  new ops/sec  delta
kv95/enc=false/nodes=1/cpu=32   73.1k ± 3%   61.1k ± 3%  -16.30%  (p=0.000 n=9+8)

name                           old p50      new p50      delta
kv95/enc=false/nodes=1/cpu=32    0.70 ± 0%    0.70 ± 0%     ~     (all equal)

name                           old p95      new p95      delta
kv95/enc=false/nodes=1/cpu=32    1.80 ± 0%    2.91 ± 6%  +61.81%  (p=0.000 n=8+8)

name                           old p99      new p99      delta
kv95/enc=false/nodes=1/cpu=32    4.70 ± 0%    6.82 ± 4%  +45.21%  (p=0.000 n=8+8)

Notice how the new April 21st graph shows a clear improvement from the start, but then slows down after about a minute to a level significantly below the old April 19th baseline. Statistics collection would be a likely cause for the abrupt change.

/cc @cockroachdb/test-eng

[RaduBerinde]

Wow.. thanks for tracking this down! I think we are missing special logic when the memo is determined to be "stale".

[RaduBerinde]

@erikgrinaker for how long did you run the benchmarks? It must have been more than a minute if it slows down after that?

[tbg]

roachtest usually runs 10 minute intervals, which is also what you can see in the above graph

[erikgrinaker]

Just did roachtest run --count 10 '^kv95/enc=false/nodes=1/cpu=32$', they're 10-minute runs plus possibly some ramp-up time.

[RaduBerinde]

Thanks. I understand the issue now. The problem is that we are using the statistics that are derived using placeholders, which are bogus. Once we have stats, our estimated row count is very large, and that affects the way the query is being executed.

planning time: 259µs
execution time: 4ms
distribution: local
vectorized: true
rows read from KV: 1 (0 B)
cumulative time spent in KV: 679µs
maximum memory usage: 90 KiB
network usage: 0 B (0 messages)

• scan
  columns: (k int, v bytes)
  cluster nodes: n1
  actual row count: 1
  vectorized batch count: 1
  KV rows read: 1
  KV bytes read: 0 B
  estimated row count: 20,653
  table: kv@primary
  spans: /-3812730450502136779/0