main loop: configurable delay and more

[hjoliver]

See comments #714 (comment) (and after). Current hardwired 1 sec sleep each time through these loops keeps CPU usage low, but it delays task spawning a bit and suite shutdown by a couple of seconds. Changing to say 0.1 sec results in snappier performance - good for quick test suites in particular - and faster shutdown, but higher CPU usage. So ... might be good to make this configurable as per @matthewrmshin's comment #714 (comment)

[hjoliver]

[meeting] - Or longer! Or dynamic. Set and check a flag. (event-driven scheduling model better)

[matthewrmshin]

Related to this (somewhat) is that I would like a more direct control of the multiprocess pool using the main loop. The use of the standard library multiprocess pool has been causing headache, e.g. having to start the pool early and having child processes alive even when they are doing nothing.

A better approach is to write our own child process management logic and have it integrated with the main loop. This approach will give us more flexibility, e.g. child processes on demand, better external command management, control of number of child processes in run time, prioritisation of items in processing queue, etc.

[matthewrmshin]

See also #2315 (comment)

[matthewrmshin]

(Note: #2590 has introduced dynamic timing to the main loop, based on how busy it was and whether there are items in the process pool.)

[dpmatthews]

@hjoliver - are you happy to close this? We don't have a configurable delay but hopefully the dynamic timing removes the need for this.

[matthewrmshin]

@hjoliver I think this is superseded by #2866, which will address the issue in a more standard way?

[hjoliver]

Yep, I'll close this; thanks for flagging it. Only 166 issues to go now 😬 ...