[feature request] Restart a job after build pods was killed unexpectedly

[tk3fftk]

What happened:
Build pods are killed and users have to restart manually his/her jobs when something troubled in a k8s cluster especially a node.

What you expected to happen:
It looks better that a job restart after detecting a lost build pod.
Some builds must run only once so this feature should be able to opt-in or opt-out.

How to reproduce it:

• drain a k8s node which has build pods or delete a build pod

Related: Conditionally retry a step on failure #1645