auto-restart: enable for stopping workflows

[dpmatthews]

If a host gets condemned, any workflows which are in the stopping state do not currently get migrated. This causes problems because any such workflows can keep running indefinitely. We need to allow such workflows to be migrated.

[oliver-sanders]

Workflows which are in the stopping state cannot be safely migrated as we cannot preserve their state in a restart.

The stopping state should be transient so this problem should never happen. The issues we've seen recently workflows were getting stuck in the stopping state are being addressed as part of other issues.

• ihs: catch NoHostsError #5195 (prevent remote-tidy failure from blocking shutdown).
• Make shutdown more robust #5201 (prevent any errors in shutdown from blocking shutdown).

[oliver-sanders]

The issue is when workflows have been stopped in normal shutdown mode where the workflow waits for active jobs to complete before stopping. In this case the jobs might have a high execution time limit (or even no limit at all).

The tricky thing is restarting the workflow in the stopping state without any spawning / queueing logic being activated. We might be able to abuse something like --stop-after-cycle-point to avoid the need for a new flag.