Improve execution of jobs and commands

[hubertp]

This is a placeholder ticket for a new design for executing jobs and commands. It comes as part of the investigation into #7005 which discovered several shortcomings in the current implementation. Namely:

1. Locks are used extensively to ensure the corrected of the execution
2. Jobs and commands are two ways of serving user requests. The former often does not guarantee timely response, leading to timeouts and unnecessary confusion

In #7005 we patched the underlying problem by sending replies early and take into account jobs that may still be in-progress while other requests come around. But there is an inherent problem between long-running jobs (e.g. compilation) and requests that should be served quickly, exacerbated by the extensive locking logic.

[jdunkerley]

Closing this one for now.