timeout for external commands?

[matthewrmshin]

We have recently noticed that various external commands (including service commands such cylc jobs-submit, cylc jobs-kill) launched by the suite process can hang (possibly due to OS issues?). Some are local commands, and some are run via SSH. Perhaps we need to wrap all commands with the timeout command to ensure that external commands die after a reasonable amount of time?

See also #2292 and #2302.

[hjoliver]

What's happens if this occurs - one process pool member just hangs forever?

[matthewrmshin]

Yes, they probably just hang forever. One thing I have mentioned before is to have the main loop manages the external command processes directly, instead of via a process pool. This should allow us a much better control over hung processes - e.g. we can have event notification of external command timeouts and allow users to tell the suite to kill them via the suite API.

[oliver-sanders]

See also #2468 (comment)

[matthewrmshin]

Original intention fixed by #2659. With that, I don't think there is a strong requirement the extra stuff mentioned in my previous comment. Closing issue.