Add Kubeflow Components Overview Diagram

[andreyvelich]

This diagram shows high-level diagram for Kubeflow components, so users can see which part of MLOps lifecycle Kubeflow covers today.
We can edit this diagram in the future by simply import it to: https://www.drawio.com/

We showed this diagram in the latest Kubeflow Summit: https://youtu.be/HBBJYa4JTNY?list=PL2gwy7BdKoGdrkYIWGeAdKi9ntfxq8FYt

Related: #3536

/hold for review

[andreyvelich]

cc for the review on the Kubeflow high-level architecture.
@kubeflow/wg-training-leads @kubeflow/wg-pipeline-leads @kubeflow/wg-notebooks-leads
@jbottum @thesuperzapper @akgraner @vikas-saxena02

[terrytangyuan]

Should we store the drawio source file as well so it's easier to modify later

[thesuperzapper]

@andreyvelich Also, I dont know why/how but this image is somehow interactively linked to draw.io: https://deploy-preview-3650--competent-brattain-de2d6d.netlify.app/docs/started/architecture/

I recommend we raster it down to an SVG to ensure compatibility.

[thesuperzapper]

Also, your diagram should make it clear that any Kubernetes cluster will work, so perhaps we should replace the minikube logo with the Kubernetes one?

Or, perhaps the Kubernetes logo should replace the "SDKs", "Web UI", and "Kubectl" layer, to indicate that Kubeflow runs on Kubernetes.

[thesuperzapper]

Also, I think the components should be ordered by popularity, or possibly the order in which you use them.

For example:

• Top-Left: Kubeflow Pipelines
• Top-Right: Kubeflow Notebooks
• Mid-Left: Central Dashboard
• Mid-Right: Katib (AutoML)
• Bottom-Left: Training Operator
• Bottom-Right: KServe (Serving)

(You also missed central dashboard, which is a pretty important component).

[andreyvelich]

Should we store the drawio source file as well so it's easier to modify later

@terrytangyuan If you upload this image to drawio, you can modify it

[jbottum]

isn't the Web UI the Central Dashboard ?

[thesuperzapper]

@jbottum @andreyvelich perhaps we should re-word the "Web UI" to "Web Dashboard"?

[thesuperzapper]

@andreyvelich I think we can swap the "layers" to more logically represent what runs "on top" of what.

My idea for ordering (from top to bottom):

1. Libraries (Jupyter / Tensorflow / Torch / ...)
   • Yellow Horizontal Bar
2. Interfaces (SDKs / Web Dashboard / Kubectl)
   • Blue Arrow
3. Kubeflow
   • Blue Arrow
4. Kubernetes
   • Yellow Horizontal Bar
5. Platforms (AWS / GCP / On-Prem / ...)

Some other notes:

• Minikube is not really a platform, so we can remove it (or possibly replace it with "Local Deployment", with a picture of a laptop).
• We should put a dotted line around the "Platforms" group, like the "Libraries" one.

[andreyvelich]

Web UI is the combination of all UI that Kubeflow offers today (e.g. Central Dashboard, Katib, Pipelines, Volumes, Tensorboards, etc.)

[andreyvelich]

@thesuperzapper I agree that Kubernetes should go above the platforms and I can change minikube to local deployment, but I am not sure if SDK/Web UI/kubectl should be between Kubeflow and libraries.

My thoughts are that interfaces (SDK, Web UI, and kubectl) is a layer between Kubeflow components and Kubernetes. For example, when user run create_job Training Operator SDK API, it creates resources in Kubernetes clusters.
Or when user clicks create Experiment button on the Katib UI it creates custom resources in Kubernetes.

Any thoughts @thesuperzapper @jbottum @johnugeorge @vikas-saxena02 ?

[thesuperzapper]

@andreyvelich I thought your diagram going to be used on the "introduction" page rather than the "architecture"?

And that we would use a diagram like this one on the "architecture" page:

[andreyvelich]

I thought your diagram going to be used on the "introduction" page rather than the "architecture"?

Yes, let me move it to introduction for now. I hope, we have time to discuss it during tomorrow's community call to hear the feedback from others.

[hbelmiro]

Small typo.

Suggested change

	The following diagrams shows the main Kubeflow components to cover each step of ML lifecycle
	The following diagram shows the main Kubeflow components to cover each step of ML lifecycle

[vikas-saxena02]

@andreyvelich , the diagram looks great to me!!
I just have one suggestion though, for the ML Libraries section on top. Is it possible to provide names for the ones which do not have it mentioned in the icon. For non-technical users (managers, BAs etc) it will be confusing as they do not know which library is that.

[andreyvelich]

@andreyvelich , the diagram looks great to me!! I just have one suggestion though, for the ML Libraries section on top. Is it possible to provide names for the ones which do not have it mentioned in the icon. For non-technical users (managers, BAs etc) it will be confusing as they do not know which library is that.

That's good point @vikas-saxena02. I will make this change to the diagram.

[andreyvelich]

@thesuperzapper @terrytangyuan @johnugeorge @tenzen-y @hbelmiro @vikas-saxena02 @jbottum Any other suggestions before we can merge this initial diagram ?

[hbelmiro]

/lgtm

[google-oss-prow]

@hbelmiro: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jbottum]

/lgtm

[thesuperzapper]

@andreyvelich I like where you are going but I think it can still be made more clear.

I have taken a stab at it based on your design, I went back and forward on keeping the "Web UI" box, but settled on replacing "ML Metadata" with "Central Dashboard" because that's similar to what we do on the kubeflow.org homepage.

The following file is in SVG draw.io format if you want to modify it further:

PS: we should use SVG rather than PNG for the website (don't forget to click "embed images" when exporting from draw.io)

Here is a compressed version of the PNG format, incase GitHub comments break the image draw.io compatibility:
kubeflow-intro-diagram.drawio.png.zip

[terrytangyuan]

Great work! We can always improve/iterate later.

/lgtm

[andreyvelich]

Sure @thesuperzapper, that sound good, I made the appropriate changes.
I keep components in the appropriate order (e.g. on the left side: ML Training, ML Tuning, ML Serving. On the right side development tools: Notebooks, Pipelines, Central Dashboard )
/hold cancel

[thesuperzapper]

@andreyvelich based on your comment I think we can add the "Model Training" / etc labels underneath each component to help new users understand what each box means.

I also think that it makes sense to put the development tools on the left side (the same order we use on the homepage).

For example:

And the compressed file version:
kubeflow-intro-diagram-2.drawio.svg.zip

[thesuperzapper]

@andreyvelich Based on a few comments I have made the labels a lot less prominent

kubeflow-intro-diagram-3.drawio.svg.zip

[andreyvelich]

I made the final changes based on the discussion.
Please provide your final comments and we can merge it.
/assign @thesuperzapper @terrytangyuan @johnugeorge @tenzen-y @hbelmiro @vikas-saxena02 @jbottum @james-jwu @zijianjoy

[zijianjoy]

/lgtm
/approve

Thank you @andreyvelich and everyone for your help on this PR!

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, hbelmiro, terrytangyuan, zijianjoy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• content/en/docs/started/OWNERS [zijianjoy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

@andreyvelich Thank you for this improvement! This looks great :)