Experiment Can Stop Launched Jobs #677
Conversation
launchers how to stop the jobs that they manage.
MattToast
added
type: feature
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## smartsim-refactor #677 +/- ##
=====================================================
+ Coverage 40.45% 43.59% +3.13%
=====================================================
Files 110 110
Lines 7326 7076 -250
=====================================================
+ Hits 2964 3085 +121
+ Misses 4362 3991 -371
|
| raise ValueError("No job ids provided") | ||
| by_launcher = self._launch_history.group_by_launcher(set(ids), unknown_ok=True) | ||
| id_to_stop_stat = ( | ||
| launcher.stop_jobs(*launched).items() |
There was a problem hiding this comment.
One thing I am thinking about here is stop_jobs may raise a smartsim.error.errors.LauncherJobNotFound, and if it does, it seems like the process only partially completes. Should we complete all of the ones that can complete and then (re)raise the error?
There was a problem hiding this comment.
In theory this cannot happen if the _launch_history is never corrupted (e.g. an id is never mapped to a launcher that never issued it), but I do agree that this could become a problem, especially if users start to make their own launchers.
I moved this to a new issue where it can be discussed further because I think this may be a problem with other methods (e.g. get_status, wait) as well as potentially other errors that can arise when stopping over a collection of ids.
Add an
Experiment.stopmethod to stop jobs prematurely. Teach launchers how to stop the jobs that they manage.