Job Controller

[thockin]

We need something like ReplicationController that runs RestartOnFailure and RestartNever pods "to completion", collects results, etc

[bgrant0607]

Also discussed in #503.

[bgrant0607]

See also OpenShift's proposed job concept: https://github.com/openshift/openshift-pep/blob/master/openshift-pep-013-openshift-3.md

Potential features include:

• deadline (Restart policy should be able to specify maximum running time (aka deadline) #829)
• queuing
• cron-like scheduling
• replication
• success/failure aggregation
• gang scheduling and/or admission control
• inter-job ordering dependencies and/or activation control for externalized dependency management

[smarterclayton]

We have not yet implemented the generic concept, instead using pods directly. Initially it seems like each of the individual places run-once pods were used would be related, the actual flows were easy to control. Advantages of a unified job resource would be that you could easily extend it for your own use - downside is that you still have to control / ensure the job resource gets created (so you have two sync loops instead of one). Was hoping to see how others might use jobs before we proceeded further. The pattern was definitely common, but our uses had subtle differences that may not abstract well.

[bgrant0607]

That's interesting. Another option is that we could just make individual pods easier to use for these workflow sorts of scenarios (which we should do regardless).

Some features I could imagine would be useful for that:

• wait for completion via watch on events
• get success/failure information (Figure out how to best convey more detailed pod status info #1370)
• deadlines (Restart policy should be able to specify maximum running time (aka deadline) #829)
• pod templates and bulk creation (Separate the pod template from replicationController #170)
• bulk deletion / stop
• graceful and immediate termination (Consistently support graceful and immediate termination for all objects #1535)
• input/output (Ensure there is an easy way to provide container input (and to get output) #1503, Capture application termination messages/output #139) and/or some means of pod/image parameterization

Anything else?

[smarterclayton]

On Oct 11, 2014, at 12:34 PM, bgrant0607 [email protected] wrote:

That's interesting. Another option is that we could just make individual pods easier to use for these workflow sorts of scenarios (which we should do regardless).

Some features I could imagine would be useful for that:

wait for completion via watch on events
get success/failure information
deadlines
pod templates and bulk creation (#170)
input/output (#1503) and/or some means of pod/image parameterization
Anything else?

Define a standard annotation key(s) for certain job conventions, allow annotations to be atomically updated on PUT (standard if-match resource version is enough).

Logs and the ability to read logs and get pod info long after a pod is deleted.=

[bgrant0607]

Some application frameworks, such as mapreduce/hadoop-style workloads, may take on the controller responsibilities themselves rather than relying upon a shared service. An example might be the Application Master in YARN.

[tmrts]

@thockin @smarterclayton @bgrant0607 After a long journey through the issues about job management in kubernetes, I have submitted a GSOC proposal for this topic. Your feedback would be most appreciated.

[bgrant0607]

We'll take a look @TamerTas. Thanks!

[bgrant0607]

To document later:

Some batch workloads are data processing/analysis workloads of independent utility.

Others support serving workloads, such as:

• data cleanup / GC
• serving data generation / aggregation / indexing / import
• data snapshots / copies / backups / export
• logs processing / billing / audit / report generation
• maintenance (e.g., schema changes / data conversion)
• integrity checking / validation
• defense analysis (spam, abuse, dos, etc.)
• online/offline feedback / adaptation / machine learning
• continuous/periodic build/push

[soltysh]

/sub

[gmarek]

cc @mwielgus

[smarterclayton]

@soltysh please link your ongoing job proposal here so Kube folks can get a chance to look. The proposal will be coming soon here while Maciej prototypes

[soltysh]

It's already linked here, but here you go openshift/origin#3693. It covers both the job controller part and the cron scheduler.

[bgrant0607]

Could we discuss this at an upcoming community hangout?

[smarterclayton]

Absolutely

On Jul 21, 2015, at 3:06 PM, Brian Grant [email protected] wrote:

Could we discuss this at an upcoming community hangout?

—
Reply to this email directly or view it on GitHub
#1624 (comment)
.

[bprashanth]

I'm not working on the job controller, a better owner might be Mike or @soltysh

[davidopp]

@erictune has been shepherding this, so assigning to him.

[resouer]

/sub

[bgrant0607]

@aronchick Continuing from #14186 (comment)

Your primary concern is to clarify that this is a batch job (e.g., LSF, Load Leveler, Printer Job) as opposed to an indefinitely running "Job" (e.g., Borg, Aurora, Nomad)?

[soltysh]

And we do distinguish those, but as:

• job - run-once pods
• replicationcontroller (to be renamed someday Create ReplicaSet #3024) - run indefinitely pods

That's the terminology we have in k8s. @aronchick what's your opinion on that?

[erictune]

We now have this as a beta feature, in head and to appear in 1.1. Closing.

[soltysh]

@erictune are you going to close it?