feat(plugin): add new task-batching plugin #495
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DrRebus
requested review from
rbeuque74,
loopfz,
rclsilver and
wI2L
as code owners
DrRebus
changed the title
DrRebus
force-pushed
the
dev/rtordjma/task-batch-plugin
branch
2 times, most recently
from
d7f70c1 to
9b076b1
Compare
DrRebus
force-pushed
the
dev/rtordjma/task-batch-plugin
branch
from
fa28ad3 to
08c3af2
Compare
DrRebus
commented
Dec 14, 2023
DrRebus
force-pushed
the
dev/rtordjma/task-batch-plugin
branch
from
e980730 to
ddfde5b
Compare
floriancazals
previously approved these changes
Dec 23, 2023
wI2L
requested changes
Jan 11, 2024
orandin
suggested changes
Jul 22, 2024
DrRebus
force-pushed
the
dev/rtordjma/task-batch-plugin
branch
2 times, most recently
from
fc87e13 to
63bd3e6
Compare
orandin
approved these changes
Aug 7, 2024
wI2L
reviewed
Oct 22, 2024
| |----------------------|-------------------------------------------| | ||
| | `batch_id` | The public identifier of the batch | | ||
| | `remaining_tasks` | How many tasks still need to complete | | ||
| | `tasks_started` | How many tasks were started so far | |
There was a problem hiding this comment.
Might be interesting to add the number of tasks done, and failed.
wI2L
requested changes
Oct 25, 2024
DrRebus
force-pushed
the
dev/rtordjma/task-batch-plugin
branch
3 times, most recently
from
877b552 to
14d3a71
Compare
wI2L
previously approved these changes
Oct 25, 2024
DrRebus
force-pushed
the
dev/rtordjma/task-batch-plugin
branch
2 times, most recently
from
f8c4a01 to
da58b0e
Compare
|
CDS Report test#1912.0 ✘
|
Added a task-batching plugin that creates X tasks of the same template and waits until all of them are in a final state. Moved batch-creation code to the engine package and made the POST batch handler use it. Exposed final tasks' states to be used by the batch plugin. Signed-off-by: Ruben Tordjman <[email protected]>
DrRebus
force-pushed
the
dev/rtordjma/task-batch-plugin
branch
from
da58b0e to
e02edf6
Compare
|
CDS Report package#1913.0 ✘
|
wI2L
approved these changes
Oct 25, 2024
wI2L
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
A new feature: a task batching plugin.
What is the current behavior? (You can also link to an open issue here)
The only way of batching tasks so far is to use a
subtaskplugin with theforeachfeature, which dynamically adds a new step persubtask. This approach isn't efficient DB-wise, because everytime a step is updated, the whole step tree (a JSON formatted string) gets updated, triggering anUPDATEin the database. TheseUPDATEs create new bigger-and-bigger rows and delete the previous ones, bloating the DB and requiring a strongerVACUUM.What is the new behavior (if this is a feature change)?
The task-batching plugin spawns child tasks as independent tasks that wake the parent once completed. Since the parent task no longer embed its children, no hard link exists between them, but they're linked via:
ParentTaskIDtag, the public ID of the parent task, allowing the child to wake its parent.BatchIDproperty, the public ID of the batch the task belongs two.Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
Nope.
Other information:
This plugin doesn't yet aggregate results of the children tasks. It'll come in a subsequent version.
Contrary to what's been discussed, when using sub batches (
sub_batch_size> 0), children tasks don't always wake their parent when they're done. Instead, the last task in the sub batch will wake the parent. That way, we limit even more the amount of UPDATEs made. The downside is that we have to wait for the whole sub batch to be done before spawning more tasks in the batch.