storage: emit health alert when >100 Raft snapshots are queued

This is a dangerous condition. Adding this to the health checker has the
additional benefit of logging it during the nightly restore/import
tests, which can in turn help diagnose whether a particular run is
affected by cockroachdb#31409.

Release note: None

Author: tbg

pkg/server/status/health_check.go

@@ -77,6 +77,10 @@ var trackedMetrics = map[string]threshold{
 	"queue.raftsnapshot.process.failure":  counterZero,
 	"queue.tsmaintenance.process.failure": counterZero,
 	"queue.consistency.process.failure":   counterZero,
+
+	// When there are more than 100 pending items in the Raft snapshot queue,
+	// this is certainly worth pointing out.
+	"queue.raftsnapshot.pending": {gauge: true, min: 100},
 }
 
 type metricsMap map[roachpb.StoreID]map[string]float64