Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for cancelled jobs. #23

Open
wants to merge 190 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
190 commits
Select commit Hold shift + click to select a range
e22b4c2
`exceptions.py`: Add `GoToStep` exception.
contrains Oct 17, 2024
c9912e5
`workflow.py`: `Workflow`: Modify to accept optional `label` argument…
contrains Oct 17, 2024
df53ac8
`workflow.py`: `Workflow._labels`: Store indices from `_steps` list r…
contrains Oct 17, 2024
4de9c10
`job.py`: `get_input_value`: Refactor conditional logic in `get_input…
contrains Oct 17, 2024
a008530
Revert "`job.py`: `get_input_value`: Refactor conditional logic in `g…
contrains Oct 17, 2024
6041591
`workflow.py`: `Workflow`: Require step labels to be string values.
contrains Oct 17, 2024
05dab16
`workflow.py`: `Workflow`: Add `get_label_index` method to get list i…
contrains Oct 17, 2024
087d33a
`workflow.py`: `Workflow`: Ensure label is unique before setting.
contrains Oct 17, 2024
acde2c8
`exceptions.py`: `GoToStep`: Allow passing step index or label.
contrains Oct 17, 2024
a226f1a
`exceptions.py`: `GoToStep`: Add `has_step` property. (Returns true …
contrains Oct 17, 2024
425eb3f
`exceptions.py`: Add `GoToEnd` exception.
contrains Oct 17, 2024
3aebb1a
`job.py`: `Job`: Add `mark_step_n_completed` method.
contrains Oct 17, 2024
f5e51d2
`job_runner.py`: Add handler for `GoToStep` exception.
contrains Oct 17, 2024
7843d3e
`job_runner.py`: Add handler for `GoToEnd` exception.
contrains Oct 17, 2024
0acc9ff
Add documentation for workflow step labels and `GoToStep` / `GoToEnd`…
contrains Oct 18, 2024
f587dd0
Add Ergeats logo.
contrains Oct 18, 2024
8c2d799
Update README with Ergeats logo.
contrains Oct 18, 2024
5a62309
Update README acknowledgements.
contrains Oct 18, 2024
c1ad0b3
Fixes for tox.
contrains Oct 18, 2024
ac5f3f9
`job_runner.py`: Add missing log output for `GoToEnd`.
contrains Oct 18, 2024
88dbd27
`job_runner.py`: Use logging string substitution.
contrains Oct 18, 2024
889ca37
Fixes for tox.
contrains Oct 18, 2024
fb1efe4
Update Ergeats logo.
contrains Oct 18, 2024
e615b73
`workflow.py`: `get_label_index`: raise KeyError on missing label.
contrains Oct 18, 2024
387d77f
`workflow.py`: Add `@label` decorator method, split from `@step`.
contrains Oct 18, 2024
a46aa97
`workflow-step-labels.md`: Update with @workflow.label` decorator.
contrains Oct 18, 2024
93aaf93
Fixes for tox.
contrains Oct 18, 2024
c384731
Add debug output code.
contrains Oct 18, 2024
c57184e
`workflow.py`: Modify `@label` decorator to (hopefully) work when pai…
contrains Oct 18, 2024
fb6e7ba
Add debug output code.
contrains Oct 18, 2024
1c4b4ae
`workflow.py`: Modify `@label` decorator to (hopefully) work when pai…
contrains Oct 18, 2024
bf6323f
`job_runner.py`: Fix bug with calling `mark_step_n_completed` pertain…
contrains Oct 18, 2024
bda62a8
Add debug output code.
contrains Oct 18, 2024
ffa64b8
`workflow.py`: Refactor `label._decorate` function to only accept Wor…
contrains Oct 18, 2024
9e58ae5
`workflow.py`: Remove debug code.
contrains Oct 18, 2024
1e8eb6e
`job.py`: Add debug code.
contrains Oct 18, 2024
09c2dd0
`job_runner.py`: Update log output for `GoToEnd` to show `retval`.
contrains Oct 18, 2024
346aee1
`job_runner.py`: Update log output for `GoToStep` to show `retval`.
contrains Oct 18, 2024
e073151
`job.py`: Remove debug code.
contrains Oct 18, 2024
90ba635
Fixes for tox.
contrains Oct 18, 2024
acf7d7e
`workflow-step-labels.md`: Update @workflow.label` decorator details …
contrains Oct 18, 2024
420b9d2
`workflow.py`: `Workflow.label`: change `len(self._steps)` to `len(se…
contrains Oct 21, 2024
207af0e
`exceptions.py`: `GoToStep`: add `next_step` property which returns i…
contrains Oct 21, 2024
b5f7c69
`workflow_step.py`: Add `WorkflowStep.paths` attribute containing a l…
contrains Oct 21, 2024
0e4e3e8
`workflow.py`: Add `Workflow.paths` decorator which defines what exce…
contrains Oct 21, 2024
408ba47
`workflow.py`: `Workflow`: Add `calculate_paths` method and supportin…
contrains Oct 21, 2024
392603a
`job_runner.py`: Implement `Workflow.calculate_paths` into `JobRunner`.
contrains Oct 21, 2024
f394a28
`job.py`: Add and implement `Job.current_step` attribute, as distingu…
contrains Oct 21, 2024
71012d2
`workflow.py`: Add debug code.
contrains Oct 21, 2024
bae01e7
`job_runner.py`: Add debug code.
contrains Oct 21, 2024
eb2c713
Fixes for tox.
contrains Oct 21, 2024
d7f7760
`job_runner.py`: Add missing second argument for `filter`.
contrains Oct 21, 2024
f8a6f25
Update debug output.
contrains Oct 21, 2024
a2fe13d
Update debug output.
contrains Oct 21, 2024
862b311
Update debug output.
contrains Oct 21, 2024
e5a8f0f
`workflow.py`: `_calculate_paths`: Change operator for loop detection…
contrains Oct 21, 2024
c932edf
`job_runner.py`: Fix incorrect workflow plan step index.
contrains Oct 21, 2024
96900e3
`workflow.py`: Fix incorrect reference to step plan index and exception.
contrains Oct 21, 2024
0aa78d4
`workflow.py`: Minor reformatting.
contrains Oct 21, 2024
503e01c
`workflow.py`: `_calculate_paths`: Update path list creation.
contrains Oct 21, 2024
1ed7c07
`workflow.py`: `_calculate_paths`: Add current step to path if no chi…
contrains Oct 21, 2024
ef2f836
`workflow.py`: `_calculate_paths`: Update path list creation.
contrains Oct 21, 2024
2a6aa08
`workflow.py`: `_calculate_paths`: Do not add current step to path wh…
contrains Oct 21, 2024
faf8ef3
`job_runner.py`: (Hopefully) fix logic with calculating workflow comp…
contrains Oct 21, 2024
ca2b65c
`job_runner.py`: Fix logic with calculating workflow completion perce…
contrains Oct 21, 2024
36f89f5
Remove debug code.
contrains Oct 21, 2024
74dc11d
`workflow.py`: Consolidate `@step`, `@label` and `@paths` decorators …
contrains Oct 22, 2024
2b39e7f
Fixes for tox.
contrains Oct 22, 2024
bbb55cf
`workflow.py`: `@step`: If decorated function has type hints with a r…
contrains Oct 22, 2024
7ea5a44
`workflow-step-labels.md`: Update documentation for `label` in `@step…
contrains Oct 22, 2024
f35e97c
`exceptions.py`: Modify `GoToStep` exception to take label or index i…
contrains Oct 22, 2024
9b509b6
Add `workflow-step-paths.md` documentation. Update documentation for…
contrains Oct 22, 2024
4cc2365
`workflow.py`: `step`: Use `types.NoneType` for comparison of return …
contrains Oct 22, 2024
31b1266
`workflow-step-paths.md`: Update wording.
contrains Oct 22, 2024
e05bff2
`workflow.py`: Add debug output.
contrains Oct 22, 2024
1015971
`workflow.py`: Add debug output.
contrains Oct 22, 2024
3a819de
Add debug output.
contrains Oct 22, 2024
6c781e0
`exceptions.py`: Fix bug in `GoToStep.label` method.
contrains Oct 22, 2024
858e41c
`README`: Update acknowledgements.
contrains Oct 22, 2024
68b66a2
Remove debug code.
contrains Oct 22, 2024
afa3d61
Update `README`
contrains Oct 22, 2024
4f21914
`exceptions.py`: Split index and label again, with extra checks for m…
contrains Oct 23, 2024
d71b734
Fix type hints for mypy.
contrains Oct 23, 2024
2be9254
Fix for tox.
contrains Oct 23, 2024
8f66ec9
Refactor `idx` to `index`.
contrains Oct 23, 2024
b50b320
`workflow.py`: Add `@overload` definitions for `Workflow.step` decora…
contrains Oct 30, 2024
cf94ae7
`workflow.py`: Import `overloading.overload`.
contrains Oct 30, 2024
627391d
`workflow.py`: Replace `overloading.overload` with `functools.singled…
contrains Oct 30, 2024
68d44e6
`workflow.py`: Add `TypeAlias` type hinting.
contrains Oct 30, 2024
65527c6
`workflow.py`: Rename `WorkflowPath` to `WorkflowPathTypeHint`.
contrains Oct 30, 2024
bff1ffc
`workflow.py`: Replace `functools.singledispatchmethod` with `typing.…
contrains Oct 30, 2024
d45418b
`workflow.py`: Add debug code.
contrains Oct 30, 2024
2dd3c33
`workflow.py`: Update debug code.
contrains Oct 30, 2024
36895e6
`workflow.py`: Remove debug code.
contrains Oct 30, 2024
25156ed
Fixes for tox.
contrains Oct 30, 2024
1ef2518
Remove Ergeats branding.
contrains Nov 15, 2024
8537f36
`tox.ini`: show diff for formatting errors.
contrains Nov 17, 2024
5124f60
`workflow.py`: Reformat for tox.
contrains Nov 18, 2024
7afb6c9
Store `label` in `WorkflowStep`.
contrains Nov 18, 2024
01b9ab6
Add debug code.
contrains Nov 18, 2024
0ff7d53
Replace `label` with use of the functions defined name.
contrains Nov 18, 2024
17b371a
Update documentation.
contrains Nov 18, 2024
30313ae
`workflow.py`: `Workflow._find_next_step`: Invert ternary as fix for …
contrains Nov 18, 2024
21a9181
Add debug output.
contrains Nov 19, 2024
a850441
`job_runner.py`: Fix bug with `GoToEnd` counting one too few total st…
contrains Nov 19, 2024
0f85789
Update debug output.
contrains Nov 19, 2024
49811c9
`job_runner.py`: Fix bug with `GoToEnd` counting one too few total st…
contrains Nov 19, 2024
92e293c
Remove debug output.
contrains Nov 19, 2024
17b079f
Merge pull request #1 from contrains/replace-labels-with-function-names
contrains Nov 19, 2024
6e1e25a
Some changes cherry-picked from `add-path-hinting` branch.
contrains Nov 22, 2024
a576ebd
More changes cherry-picked from `add-path-hinting` branch.
contrains Nov 22, 2024
18444cc
Some changes cherry-picked from `manual-step-ordering` branch.
contrains Nov 22, 2024
e368fc8
`__init__.py`: Sort `__all__`.
contrains Nov 22, 2024
b7e7e98
Fix whitespace.
contrains Nov 22, 2024
caa0b44
`job_runner.py`: Fix bug (hopefully) with calculating total steps whe…
contrains Nov 22, 2024
7c31978
`job_runner.py`: Update debug code.
contrains Nov 22, 2024
c2dc930
`job_runner.py`: Fix off-by-1 bug in remaining/total steps counts for…
contrains Nov 25, 2024
30e409c
`job_runner.py`: Update debug output.
contrains Nov 25, 2024
74cb01f
`__init__.py`: Add missing exports.
contrains Nov 25, 2024
06a4d0f
`job_runner.py`: Fix off-by-1 bug in remaining/total steps counts for…
contrains Nov 25, 2024
e01d79c
`job.py`: `Job.mark_n_steps_completed`: Fix bug with calculating curr…
contrains Nov 25, 2024
447c2b4
`job.py`: `Job.mark_n_steps_completed`: Attempt 2: Fix bug with calcu…
contrains Nov 25, 2024
e0981d9
Remove debug output code.
contrains Nov 25, 2024
d2343c5
Reformat for tox.
contrains Nov 25, 2024
305a183
Rewrite docs for manual step ordering and workflow path hints.
contrains Nov 25, 2024
c13e6ae
Revert "`tox.ini`: show diff for formatting errors."
contrains Nov 25, 2024
2e363e7
Fixes in documentation.
contrains Nov 25, 2024
572df7d
Fixes in docs.
contrains Nov 25, 2024
eaf944c
Remove leftover TODO.
contrains Nov 25, 2024
374d11c
Rename `UnknownStepNameError` to `UnknownStepError`.
contrains Nov 25, 2024
ea16318
`workflow.py`: Add `get_step_index(WorkflowStep)` method.
contrains Nov 25, 2024
3829159
Modify `GoToStep` to accept `WorkflowStep` rather than `int | str`. …
contrains Nov 25, 2024
24499a1
Rename `Workflow.get_index_by_step_name` to `get_step_index_by_name`.
contrains Nov 25, 2024
745226a
Eliminate `Workflow._step_names` dict. Although there is a performan…
contrains Nov 25, 2024
1e40385
`job_runner.py`: Rearrange `GoToEnd` code.
contrains Nov 25, 2024
243415d
`WorkflowStep`: Move definition of index to within object. Eliminate…
contrains Nov 25, 2024
65b5568
`WorkflowStep`: Prepare list of path hints within class rather than e…
contrains Nov 25, 2024
46ab4b1
`job_runner.py`: Fix bug for calculating remaining steps from `GoToSt…
contrains Nov 26, 2024
b87482b
`workflow.py`: Add debug output.
contrains Nov 26, 2024
ffad748
`job_runner.py`: Add debug output.
contrains Nov 26, 2024
6fc0577
`workflow.py`: Do not call `._steps` directly when avoidable.
contrains Nov 26, 2024
6a948bd
`workflow.py`: Update debug output.
contrains Nov 26, 2024
ed38baa
`workflow.py`: Calculate all paths once at startup rather than when r…
contrains Nov 26, 2024
8ef4cf5
`workflow.py`: Calculate all paths at workflow registration using `re…
contrains Nov 26, 2024
8d96b30
`workflow.py`: Update debug output.
contrains Nov 26, 2024
e1fce94
`job_runner.py`: Update debug output.
contrains Nov 26, 2024
f711d4d
Update documentation.
contrains Nov 26, 2024
939f14c
Remove debug code.
contrains Nov 26, 2024
c32edd0
Remove debug code.
contrains Nov 26, 2024
818c544
`workflow_step.py`: Reformat for tox.
contrains Nov 26, 2024
e9f30bd
`workflow_step.py`: Reformat for tox.
contrains Nov 26, 2024
f0f2c96
`workflow_step.py`: Reformat for tox.
contrains Nov 26, 2024
581a2da
Merge pull request #3 from contrains/gotostep_accepts_steps_rather_th…
contrains Nov 26, 2024
846f7d7
`workflow.py`: Revert variable name change.
contrains Nov 26, 2024
26c9fa3
`workflow.py`: change `IndexError` to `UnknownStepError` in `Workflow…
contrains Nov 26, 2024
646207f
`job_runner.py`: `_run_job`: Minor code rearrangement.
contrains Nov 26, 2024
8c63d88
`workflow.py`: `_calculate_paths`: Minor code rearrangement.
contrains Nov 26, 2024
f50bc80
`exceptions.py`: `GoToStep`: Minor code rearrangement.
contrains Nov 26, 2024
ede4947
`job_runner.py`: Add whitespace.
contrains Nov 26, 2024
0655a8f
`workflow.py`: Change `register()` method to not return the `Workflow…
contrains Nov 26, 2024
f7030eb
Add missing type hints.
contrains Nov 26, 2024
15bc536
Update docs/docs/basics/defining-workflow-path-hints.md
contrains Nov 27, 2024
72442dd
Update docs/docs/basics/defining-workflow-path-hints.md
contrains Nov 27, 2024
d4c320c
Update docs/docs/basics/defining-workflow-path-hints.md
contrains Nov 27, 2024
34a6579
Update docs/docs/basics/manual-step-ordering.md
contrains Nov 27, 2024
f5266b9
Update docs/docs/basics/manual-step-ordering.md
contrains Nov 27, 2024
535deb6
Update docs/docs/basics/manual-step-ordering.md
contrains Nov 27, 2024
b74c789
`defining-workflow-path-hints.md`: Remove unused import from example.
contrains Nov 27, 2024
3b833f5
`workflow.py`: Rename `.register()` method to `.finalize()`.
contrains Nov 27, 2024
a1beeae
`tox.ini`: show diff for formatting errors.
contrains Nov 17, 2024
d97f2d0
`workflow.py`: Reformat for tox.
contrains Nov 27, 2024
c66165c
Revert "`tox.ini`: show diff for formatting errors."
contrains Nov 27, 2024
7301d0e
Merge remote-tracking branch 'upstream/main' into main
contrains Nov 29, 2024
d46b194
Merge remote-tracking branch 'upstream/main' into main
contrains Feb 10, 2025
3b78fa4
`job_status.py`: `JobStatus`: Add `CANCELLED` status.
contrains Feb 10, 2025
e134475
`jobs-overview.md`: Add `JobStatus.CANCELLED`.
contrains Feb 10, 2025
b83edb7
job.py`: `Job.should_be_requeued()`: Add `CANCELLED` status.
contrains Feb 10, 2025
737268b
job.py`: `Job.should_be_requeued()`: Make method a class `@property`.
contrains Feb 14, 2025
bcfe5c7
`job_runner.py`: `JobRunner.run()`: Add guard clause ensuring jobs wh…
contrains Feb 14, 2025
3d19ada
`job.py`: `Job`: sort class methods alphabetically.
contrains Feb 14, 2025
38d18c5
`tox.ini`: show diff for formatting errors.
contrains Nov 17, 2024
88c5dcb
`job.py`: `Job.should_be_requeued()`: Remove `@property` decorator.
contrains Feb 18, 2025
e07ada3
`job.py`: `Job.should_be_requeued()`: Alphabetise statuses.
contrains Feb 18, 2025
6e985c3
`job_runner.py`: `run()`: Fix inverted logic.
contrains Feb 18, 2025
db829fa
`exceptions.py`: Add `CancelJob` exception.
contrains Feb 21, 2025
c4538fa
`job.py`: Add `Job.mark_cancelled()` method.
contrains Feb 21, 2025
33ed863
`job_runner.py`: Catch and handle `CancelJob` exception.
contrains Feb 21, 2025
03fd32d
`job_runner.py`: `JobRunner.run()`: Remove extra `job.should_be_reque…
contrains Feb 21, 2025
b2e21b1
Merge remote-tracking branch 'upstream/main' into add-cancelled-job-s…
contrains Feb 25, 2025
c3b4a07
`README`: Update caution regarding breaking changes.
contrains Feb 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ Workflows made easy, your way.

> [!CAUTION]
> Ergate is still in its early stages of development and therefore breaking changes are possible between every version.
>
> For a more stable experience, please consider using the [Ergeatz fork](https://github.com/contrains/ergeats), which
> will guarantee broken code rather than merely suggesting its possibility.

## Documentation
You can find the documentation for Ergate at [ergate.prryplatypus.dev](https://ergate.prryplatypus.dev).
Expand Down
1 change: 1 addition & 0 deletions docs/docs/basics/jobs-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ A `Job` can have any of the following statuses:
- `JobStatus.COMPLETED`
- `JobStatus.FAILED`
- `JobStatus.ABORTED`
- `JobStatus.CANCELLED`


## Triggering/creating a job
Expand Down
2 changes: 2 additions & 0 deletions ergate/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
from .app import Ergate
from .exceptions import (
AbortJob,
CancelJob,
ErgateError,
GoToEnd,
GoToStep,
Expand All @@ -19,6 +20,7 @@

__all__ = [
"AbortJob",
"CancelJob",
"Context",
"Depends",
"Ergate",
Expand Down
4 changes: 4 additions & 0 deletions ergate/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ def __init__(self, message: str) -> None:
self.message = message


class CancelJob(ErgateError): # noqa: N818
"""Raised when a workflow is detected as having been cancelled."""


class GoToEnd(ErgateError): # noqa: N818
"""Raised from a step to immediately complete the job."""

Expand Down
14 changes: 9 additions & 5 deletions ergate/job.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,18 @@ def get_input_value(self) -> Any:

return copy.deepcopy(input_val)

def mark_running(self, step: WorkflowStep) -> None:
self.status = JobStatus.RUNNING
def mark_aborted(self, message: str) -> None:
self.status = JobStatus.ABORTED

def mark_cancelled(self) -> None:
self.status = JobStatus.CANCELLED

def mark_failed(self, exception: Exception) -> None:
self.status = JobStatus.FAILED

def mark_running(self, step: WorkflowStep) -> None:
self.status = JobStatus.RUNNING

def mark_step_n_completed(
self,
n: int,
Expand All @@ -51,12 +57,10 @@ def mark_step_n_completed(
)
self.last_return_value = return_value

def mark_aborted(self, message: str) -> None:
self.status = JobStatus.ABORTED

def should_be_requeued(self) -> bool:
return self.status not in (
JobStatus.ABORTED,
JobStatus.CANCELLED,
JobStatus.COMPLETED,
JobStatus.FAILED,
)
5 changes: 4 additions & 1 deletion ergate/job_runner.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import Generic, TypeVar

from .exceptions import AbortJob, GoToEnd, GoToStep, ReverseGoToError
from .exceptions import AbortJob, CancelJob, GoToEnd, GoToStep, ReverseGoToError
from .handler import ErrorHookHandler
from .interrupt import DelayedKeyboardInterrupt
from .job import Job
Expand Down Expand Up @@ -102,6 +102,9 @@ def _run_job(self, job: JobType) -> None:
job.mark_step_n_completed(
job.current_step + 1, retval, job.steps_completed + remaining_steps
)
except CancelJob:
LOG.exception("Job was cancelled")
job.mark_cancelled()
except Exception as exc:
LOG.exception("Job raised an exception")
job.mark_failed(exc)
Expand Down
1 change: 1 addition & 0 deletions ergate/job_status.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ class JobStatus(IntEnum):
COMPLETED = auto()
FAILED = auto()
ABORTED = auto()
CANCELLED = auto()
2 changes: 1 addition & 1 deletion tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ extras = test

[testenv:check]
commands =
ruff format --check ergate
ruff format --diff ergate
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to undo this? :D

Copy link
Contributor Author

@contrains contrains Feb 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, anyway. ;) I was leaving it in until this issue was resolved: #23 (comment)

Do you have any thoughts about that?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to merge your PR with that since it does indeed appear unrelated. However I won't release until I get it fixed (possibly in a separate PR).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If my code subsequently is shown to have any issues once that is resolved, I'll plan to make a new PR to address them.

ruff check ergate
mypy ergate
flake8 ergate
Expand Down