Merge compilation jobs less aggressively

[galargh]

This issue was extracted from #6336.

Currently, we merge all the jobs we can when running non-production builds. In practice, that might be too aggressive of a strategy.

We should consider relaxing it a little. Some of the ideas for improvements are:

1. Merge only down to the set concurrency number. I.e. if we have 200 jobs that could be merged and 16 cores available, we should end up with 16 merged compilation jobs. We should try to keep the merged compilation jobs balanced. It will be a little more tricky with more disjoint groups, but it shouldn't be too complicated to handle.
2. Expose shouldMergeJobs and concurrency parameters to the user via config and/or CLI.