Skip to content

Commit

Permalink
On unexpected worker exit, shut down entire pool (gracefully)
Browse files Browse the repository at this point in the history
  • Loading branch information
benjie committed Nov 22, 2024
1 parent afcaaed commit fdd041c
Showing 1 changed file with 13 additions and 1 deletion.
14 changes: 13 additions & 1 deletion src/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1282,12 +1282,24 @@ export function _runTaskList(
workerPool._workers.push(worker);
const remove = () => {
if (continuous && workerPool._active && !workerPool._shuttingDown) {
// TODO: user should choose how to handle this, maybe via a middleware:
// - graceful shutdown (implemented)
// - forceful shutdown (probably best after a delay)
// - boot up a replacement worker
/* middleware.run("poolWorkerPrematureExit", {}, () => { */
logger.error(
`Worker exited, but pool is in continuous mode, is active, and is not shutting down... Did something go wrong?`,
);
_finErrors.push(
new Error(`Worker ${worker.workerId} exited unexpectedly`),
);
workerPool.gracefulShutdown(
"Something went wrong, one of the workers exited prematurely. Shutting down.",
);
/* }) */
}
workerPool._workers.splice(workerPool._workers.indexOf(worker), 1);
if (!continuous && workerPool._workers.length === 0) {
if (workerPool._workers.length === 0) {
if (!workerPool._shuttingDown) {
workerPool.gracefulShutdown(
"'Run once' mode processed all available jobs and is now exiting",
Expand Down

0 comments on commit fdd041c

Please sign in to comment.