Limit the number of report decodings that we do in parallel, in multitenant code

[bboreham]

Currently we have this call sequence:

awsCollector.Report()
  awsCollector.getReports()
    S3Store/MemcacheClient.FetchReports()
      // fetch and decode all reports in parallel
  Merger.Merge()

When you have a lot of reports this is maximally bad for memory consumption.

Since Merge() is now serial we could return compressed blobs from FetchReports() and decode just before they are merged. It's probably more effective to have some parallelism - fetch, decode and merge subsets in parallel, then merge the outputs of those subsets.

[bboreham]

I took a look at this, and it's not simple to get the benefit because we cache all the individual reports, so we're going to keep them all in memory anyway.

Maybe decode and merge reports in 5-second groups then cache the merge of those groups, instead of caching the individual reports?

[rade]

Maybe [...] merge reports in 5-second groups then cache the merge of those groups, instead of caching the individual reports?

That would be similar to what the ordinary (non-AWS) Collector is doing.

[bboreham]

This has been fixed by #3671. It is still unlimited in the sense that if the app receives a large number of requests in parallel it will start to handle them all, but that is a different feature.