Use node fields instead of custom queuing algorithm to count children checked

[chencs]

What this PR does

Updates QuerierWorkerQueuePriorityAlgo to use the Node's childrenChecked field instead of keeping its own count of nodes checked; this lets the tree reset the count after the dequeue operation is completed. This also includes a small bugfix introduced to tenantQuerierAssignments by defining leaf nodes.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[francoposa]

Approved, pending the below patch to the benchmark test which caught some bugs when I was exploring an alternate implementation.
(artistic license allowed on the comment wording)

% git diff pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go
diff --git a/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go b/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go
index 95e7ea8a3..34e455d8d 100644
--- a/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go
+++ b/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go
@@ -347,8 +347,6 @@ func TestMultiDimensionalQueueAlgorithmSlowConsumerEffects(t *testing.T) {
                },
        }
 
-       maxQueriersPerTenant := 0 // disable shuffle sharding
-
        // Increase totalRequests to tighten up variations when running locally, but do not commit higher values;
        // the later test cases with a higher percentage of slow queries will take a long time to run.
        totalRequests := 1000
@@ -365,6 +363,12 @@ func TestMultiDimensionalQueueAlgorithmSlowConsumerEffects(t *testing.T) {
        // an ingester can respond in 0.3 seconds while a slow store-gateway query can take 30 seconds
        slowConsumerLatency := 100 * time.Millisecond
 
+       // enable shuffle sharding; we cannot be too restrictive with only two tenants,
+       // or some consumers will not get sharded to any of the two tenants.
+       // enabling shuffle sharding ensures we will hit cases where a querier-worker
+       // does not find any tenant leaf nodes it can work on under its prioritized query component node
+       maxQueriersPerTenant := numConsumers - 1
+
        var testCaseNames []string
        testCaseReports := map[string]*testScenarioQueueDurationReport{}