kvflowcontrol: annotate and fix perf regressions

- Replace the flow controller level mutex-backed kvflowcontrol.Stream =>
  token bucket map with sync.Map. On kv0/enc=false/nodes=3/cpu=96
  accessing this map contributed to a high amount of mutex contention.
  We observe that this bucket is effectively read-only - entries for
  keys are written once (on creation) and read frequently after. We
  don't currently GC these buckets, but even if we did, the same access
  pattern would hold. We'll note that using a sync.Map is slightly more
  expensive CPU-wise.
- Replace various map accesses with individual variables. We were needly
  using maps to access one of two variables, keyed by work class, for
  example when maintaining metrics per work class, or tracking token
  adjustments. The map accesses appeared prominently in CPU profiles and
  was unnecessary overhead.
- Avoid using log.ExpensiveLogEnabled in hot code paths; it shows up in
  CPU profiles.
- Slightly reduce the surface area of kvflowhandle.Handle.mu when
  returning flow tokens.
- We also annotate various other points in the code where peep-hole
  optimizations exist, as surfaced by kv0/enc=false/nodes=3/cpu=96.

Part of #104154.

Release note: None

pkg/kv/kvserver/kvflowcontrol/kvflowcontrol.go

@@ -307,7 +307,8 @@ type raftAdmissionMetaKey struct{}
 // ContextWithMeta returns a Context wrapping the supplied raft admission meta,
 // if any.
 //
-// TODO(irfansharif): This causes a heap allocation. Revisit as part of #95563.
+// TODO(irfansharif,aaditya): This causes a heap allocation. Revisit as part of
+// #104154.
 func ContextWithMeta(ctx context.Context, meta *kvflowcontrolpb.RaftAdmissionMeta) context.Context {
 	if meta != nil {
 		ctx = context.WithValue(ctx, raftAdmissionMetaKey{}, meta)