Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added pipeline management commands #1320

Merged
merged 8 commits into from
Jan 25, 2017
Merged

Conversation

kevsmith
Copy link
Member

@kevsmith kevsmith commented Jan 23, 2017

This PR adds several commands aimed at helping users manage pipeline execution and their command history a la standard Unix shell behaviors. The commands are:

  • pipeline list - ps-style command for viewing pipeline status
  • pipeline info - Displays detailed information about a specific pipeline
  • pipeline kill - Aborts an executing pipeline
  • history - Lists user's command history
  • history exec - Re-executes a previous command history entry

Pipeline information is stored in the DB and retained essentially forever. Commands which list data, like pipeline list and history, limit the number of entries to 20 unless the user specifies otherwise. Pipeline process information is also stored in the DB, which is a little odd, but avoids having to keep two data stores in sync (DB table and ETS in-memory table).

There are a couple of TODOs left which need to be addressed. I'm not going to have time before I travel so I'll explain them here and trust either someone else will handle them or I'll tackle the work once I've got internet access again.

In rough priority order:

  • Template tests - Builds are currently failing because the new commands' templates are lacking tests. Should be pretty easy, if a little tedious, to address.
  • pipeline kill should return a list of killed pipeline IDs in its response so as to play nicely with pipelines.
  • pipeline list's output should include only pipelines in running or waiting states. finished pipelines can be viewed with history.
  • Some of the queries in Cog.Repository.PipelineHistory could use some refactoring to reduce duplication via query composition.
  • Revisit permissions - Currently any user can view the pipeline history for any other user. LIkewise any user can kill any other user's executing pipelines. I'm not sure this is the desired behavior. I can make convincing arguments for both sides.

pipeline exec's output isn't currently spliced into the parent pipeline. In other words, pipelines initiated by pipeline exec execute in parallel and independently of the pipeline managing the call to pipeline exec. This is a known limitation of Cog's current design and will require internal refactors to address. imo, this work should be considered out of scope for this PR and can be tackled separately.

@Shelton
Copy link
Collaborator

Shelton commented Jan 23, 2017

Great stuff for a blog post!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants