Added pipeline management commands #1320
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds several commands aimed at helping users manage pipeline execution and their command history a la standard Unix shell behaviors. The commands are:
pipeline list
-ps
-style command for viewing pipeline statuspipeline info
- Displays detailed information about a specific pipelinepipeline kill
- Aborts an executing pipelinehistory
- Lists user's command historyhistory exec
- Re-executes a previous command history entryPipeline information is stored in the DB and retained essentially forever. Commands which list data, like
pipeline list
andhistory
, limit the number of entries to 20 unless the user specifies otherwise. Pipeline process information is also stored in the DB, which is a little odd, but avoids having to keep two data stores in sync (DB table and ETS in-memory table).There are a couple of TODOs left which need to be addressed. I'm not going to have time before I travel so I'll explain them here and trust either someone else will handle them or I'll tackle the work once I've got internet access again.
In rough priority order:
pipeline kill
should return a list of killed pipeline IDs in its response so as to play nicely with pipelines.pipeline list
's output should include only pipelines inrunning
orwaiting
states.finished
pipelines can be viewed withhistory
.Cog.Repository.PipelineHistory
could use some refactoring to reduce duplication via query composition.pipeline exec
's output isn't currently spliced into the parent pipeline. In other words, pipelines initiated bypipeline exec
execute in parallel and independently of the pipeline managing the call topipeline exec
. This is a known limitation of Cog's current design and will require internal refactors to address. imo, this work should be considered out of scope for this PR and can be tackled separately.