Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better graceful shutdown for KeyboardInterrupt #19976

Merged
merged 16 commits into from
Jun 16, 2024
Merged

Better graceful shutdown for KeyboardInterrupt #19976

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f432862
exit
awaelchli Jun 14, 2024
a194065
sigkill
awaelchli Jun 14, 2024
8a285b1
update
awaelchli Jun 14, 2024
4ccbc91
update
awaelchli Jun 14, 2024
8ab4afa
only handle subprocess launcher
awaelchli Jun 14, 2024
8e99d14
update
awaelchli Jun 14, 2024
b5d83d2
update
awaelchli Jun 14, 2024
89bc275
handle atexit
awaelchli Jun 14, 2024
4ec8136
revert
awaelchli Jun 15, 2024
e2627d6
update tests
awaelchli Jun 15, 2024
edb671b
Merge branch 'feature/graceful-exit' of github.com:Lightning-AI/light…
awaelchli Jun 15, 2024
100cc6a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 15, 2024
7a6fb91
add test
awaelchli Jun 16, 2024
1637483
add changelog
awaelchli Jun 16, 2024
9bf5a55
windows doesn't have sigkill
awaelchli Jun 16, 2024
5c87355
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update
  • Loading branch information
@awaelchli
awaelchli committed Jun 14, 2024
commit 8e99d147d6311c73bf30f72ccbc9ea36b9f6bd6c
3 changes: 2 additions & 1 deletion src/lightning/pytorch/trainer/call.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,11 @@ def _call_and_handle_interrupt(trainer: "pl.Trainer", trainer_fn: Callable, *arg
rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")
# user could press Ctrl+C many times, disable KeyboardInterrupt for shutdown
signal.signal(signal.SIGINT, signal.SIG_IGN)
_interrupt(trainer, exception)
launcher = trainer.strategy.launcher
if isinstance(launcher, _SubprocessScriptLauncher):
launcher.kill(signal.SIGKILL)
exit(0)
exit(1)
except BaseException as exception:
_interrupt(trainer, exception)
trainer._teardown()
Expand Down
Loading