Series Index Store: fix race in GetSeries

Collect the individual results per-job in a slice of slices, then
flatten them out when all jobs are finished.

Author: akhilanarayanan

CHANGELOG.md

@@ -77,6 +77,7 @@
 * [9773](https://github.com/grafana/loki/pull/9773) **ssncferreira**: Fix instant query summary statistic's `splits` corresponding to the number of subqueries a query is split into based on `split_queries_by_interval`.
 * [9949](https://github.com/grafana/loki/pull/9949) **masslessparticle**: Fix pipelines to clear caches when tailing to avoid resource exhaustion.
 * [9936](https://github.com/grafana/loki/pull/9936) **masslessparticle**: Fix the way query stages are reordered when `unpack` is present.
+* [10309](https://github.com/grafana/loki/pull/10309) **bboreham**: Fix race condition in series index store.
 
 ##### Changes
 

pkg/storage/stores/series/series_index_store.go

@@ -279,7 +279,7 @@ func (c *indexReaderWriter) chunksToSeries(ctx context.Context, in []logproto.Ch
 		}))
 	}
 
-	results := make([]labels.Labels, 0, len(chunksBySeries))
+	perJobResults := make([][]labels.Labels, len(jobs))
 
 	// Picking an arbitrary bound of 20 numConcurrent jobs.
 	numConcurrent := len(jobs)
@@ -294,14 +294,18 @@ func (c *indexReaderWriter) chunksToSeries(ctx context.Context, in []logproto.Ch
 		func(_ context.Context, idx int) error {
 			res, err := jobs[idx]()
 			if res != nil {
-				results = append(results, res...)
+				perJobResults[idx] = res
 			}
 			return err
 		},
 	); err != nil {
 		return nil, err
 	}
 
+	results := make([]labels.Labels, len(chunksBySeries)) // Flatten out the per-job results.
+	for _, innerSlice := range perJobResults {
+		results = append(results, innerSlice...)
+	}
 	sort.Slice(results, func(i, j int) bool {
 		return labels.Compare(results[i], results[j]) < 0
 	})