feat: new pulling mechanism for job statuses

[yhwang]

Update the driver to keep running even the user program finishes. The driver provides two APIs:

• GetStatus(): retrieve job status
• Shutdown(): properly tear down the driver

In the controller side, it uses pod/exec resource to run the driver command to invoke the driver APIs to retrieve the job status and shutdown the driver when job is done.

[yhwang]

Some info about the pulling mechanism:
The controller uses pod/exec to run the driver command on the job pod to retrieve the job status and results periodically (using the lmes-pod-checking-interval setting). The driver command uses the domain socket as the IPC to exchange the information with the driver that runs the evaluation job on the pod. When a job is done, the controller uses the same way to run the driver command on the pod to shut down the running driver.
The communication between the controller and job pods goes through the Kube API server. No more gRPC and ingress traffic on the operator. Therefore, I removed all of the gRPC-related files/code.

[tedhtchang]

/LGTM I tested with my suspend PR

ENABLED_SERVICES=LMES CONTROLLER_TOOLS_VERSION=v0.16.3 IMG=quay.io/tedchang/trustyai-service-operator:latest make run

In another terminal:

cat << EOF| kubectl apply -f -
apiVersion: trustyai.opendatahub.io/v1alpha1
kind: LMEvalJob
metadata:
  labels:
    app.kubernetes.io/name: fms-lm-eval-service
    app.kubernetes.io/managed-by: kustomize
  name: evaljob-sample
  namespace: default
spec:
  suspend: false
  model: hf
  modelArgs:
  - name: pretrained
    value: EleutherAI/pythia-70m
  taskList:
    taskNames:
    - unfair_tos
  logSamples: true
  limit: "5"
EOF

Verified job and pod in running state

oc get po,lmevaljob
NAME                 READY   STATUS    RESTARTS   AGE
pod/evaljob-sample   1/1     Running   0          2m46s

NAME                                               STATE
lmevaljob.trustyai.opendatahub.io/evaljob-sample   Running

[openshift-ci]

@tedhtchang: changing LGTM is restricted to collaborators

In response to this:

/LGTM I tested with my suspend PR

ENABLED_SERVICES=LMES CONTROLLER_TOOLS_VERSION=v0.16.3 IMG=quay.io/tedchang/trustyai-service-operator:latest make run

In another terminal:

cat << EOF| kubectl apply -f -
apiVersion: trustyai.opendatahub.io/v1alpha1
kind: LMEvalJob
metadata:
 labels:
   app.kubernetes.io/name: fms-lm-eval-service
   app.kubernetes.io/managed-by: kustomize
 name: evaljob-sample
 namespace: default
spec:
 suspend: false
 model: hf
 modelArgs:
 - name: pretrained
   value: EleutherAI/pythia-70m
 taskList:
   taskNames:
   - unfair_tos
 logSamples: true
 limit: "5"
EOF

Verified job and pod in running state

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[github-actions]

PR image build and manifest generation completed successfully!

📦 PR image: quay.io/trustyai/trustyai-service-operator-ci:741224e023229d75d963321d5e09c73456706865

📦 LMES driver image: quay.io/trustyai/ta-lmes-driver:741224e023229d75d963321d5e09c73456706865

📦 LMES job image: quay.io/trustyai/ta-lmes-job:741224e023229d75d963321d5e09c73456706865

🗂️ CI manifests

[ruivieira]

LGTM, thanks @yhwang!

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ruivieira, tedhtchang

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment