Improve Knative Eventing End-to-End Observability (GSOC)

[aslom]

Problem
We see users struggling to observe what happens inside Knative Eventing and we want that to be improved by providing easy to use tools. It would be good to have command-line interface (CLI) tools that directly address observability needs of Knative Eventing users, in particular tracing events nd understanding how they are processed in groups/clusters.

Context: Knative Eventing has some support for observability 1 2 3 but it is piecewise and user needs for end-to-end observability are not fully addressed 4 5 6. The idea is to find out what are the biggest gaps in Knative Eventing by asking the community. Then to find a few issues that are “low hanging” fruits that can be solved during summer with the overall goal to to simplify end-to-end observability by improving support for OpenTelemetry in Knative Eventing and/or create a new plugin(s) for kn CLI.

Persona:
Targeting GSOC

Exit Criteria
Improvements in observability documented and shared with Knative community

Time Estimate (optional):
Few months (GSOC)

Additional context (optional)
Submitted as idea for GSOC as PR cncf/mentoring#560 for
https://github.com/cncf/mentoring/blob/main/summerofcode/2022.md
Moved form Knative community: https://github.com/knative/community/blob/main/google-summer-of-code/gsoc-2022.md#ideas

[aslom]

It seems we missed deadline for GSOC 2022 in CNCF? cncf/mentoring#560 (comment)

[aslom]

Link to related issue: knative/docs#2931

[aslom]

CNCF discussion is probably the best place to ask questions: cncf/mentoring#618

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[aavarghese]

/remove-lifecycle stale

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[knative-prow-robot]

This issue or pull request is stale because it has been open for 90 days with no activity.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close

/lifecycle stale

[aslom]

I am going to have updated version for GSOC 2023 here based on 2022 version https://github.com/cncf/mentoring/blob/main/summerofcode/2022.md#improve-knative-eventing-end-to-end-observability-by-addressing-top-issues-identified-by-community

Improve Knative Eventing End-to-End Observability by addressing top issues identified by community

Description: TBA
Expected outcome: Improved e2e Knative Eventing observability documented and described in one or more blog posts
Expected size of the project: 350h
Difficulty rating: Medium
Recommended Skills:TBA
Mentor(s): Aleksander Slominski @aslom, Ansu Varghese @aavarghese, and Lionel Villard @lionelvillard
Upstream Issue (URL): https://github.com/knative/eventing/issues/6247

[zeus2611]

@aslom @aavarghese hey, I'm interested in this issue for gsoc'23. can u suggest how i can get started with this?

[aslom]

Hi @zeus2611 the updated version for 2023 is here: https://github.com/cncf/mentoring/blob/main/summerofcode/2023.md#knative - what is your level of experience with observability and knative?

[zeus2611]

@aslom in terms of knative i don't have any experience with it but i'm looking into it. And, I have experience with serverless and event driven application and functions on cloud platforms. And, I have experience with prometheus, grafana, elk stack for monitoring and logging.

[arcAman07]

Hey @aslom @zeus2611 I don't have much experience with Knative Eventing but am really proficient in the world of ML/ Data Science. I am looking forward to contribute to this project pre and post GSOC. How can I get started with it? ( in-terms of any prerequisite tasks or initial issues to work on)

[zeus2611]

@arcAman07 I'm new to this too and am looking to work on this issue as a start for my open source contribution. I can only suggest going through knative docs and getting a understanding of what it is and how it works. And, getting some hands-on experience with this stack as that's what i'm doing currently.

And,@aslom please let us know if their's any prerequisites or initial tasks which we could work on to get better understanding of this project and future tasks.

[arcAman07]

@zeus2611 , really sorry tagged you by mistake.

[MatrixX-X]

@aslom Hi, I am new to open source.
I am interested in this project, is there anything that i can do to increase my chance of getting selected for this project

[Lan-ce-lot]

Hey @aslom I am a master from SEL laboratory of Zhejiang University, familiar with cloud native, golang and eBPF. I have done some observable work before. I am very interested in the Knative community and hope to participate in the community through this issue.

[aslom]

We would like to thank you for your comments and being interested in our project.

To prepare for the project please make sure you know the technologies below — here is the list of tutorials that you can use:

• Golang tutorials: https://go.dev/doc/tutorial/
• Python tutorial: https://www.learnpython.org/
• Kubernetes tutorials: https://kubernetes.io/docs/tutorials/kubernetes-basics/ and https://kubebyexample.com/
• Knative Serving, Functions and Eventing tutorials: https://knative.dev/docs/getting-started/
• Knative Eventing tutorial with Apache Kafka - make sure service and events can be sent and received: https://redhat-developer-demos.github.io/knative-tutorial/knative-tutorial/advanced/eventing-with-kafka.html
• OpenTelemetry tutorial: https://opentelemetry.io/docs/instrumentation/go/getting-started/
• Distributed Tracing in Knative Blog: https://knative.dev/blog/articles/distributed-tracing/

If there is something unclear about the project we proposed please comment in this issue or join Knative Slack and ask question in #knative or #knative-eventing channel https://github.com/knative/community/blob/main/SLACK-GUIDELINES.md

[aslom]

Hi,

We would like to help you to make a successful application for this project before the Gsoc contributor application period ends April 4.

We think that the key for the successful application is to share it with us early so we can give you feedback (sharing with us your google doc draft is the easiest way).

Beyond requirements set by Gsoc for this project we ask you to show one meaningful contribution to open source project. It’s fine if you never contributed to an open source project, we will try to help you. We suggest that you make contributions based on your observations when you followed the tutorials. In particular, think about observability in Knative Eventing, how OpenTelemetry works and how Knative Eventing can be improved in this regard. Please share with us what GitHub issue and PR you are opening by mentioning it in your google doc, here in this issue and in Knative Slack. Then we will work with you to get that contribution accepted. Please follow steps outlined in https://knative.dev/docs/community/contributing/ and https://github.com/knative/eventing/blob/main/CONTRIBUTING.md blog.

In your application in the Synopsis section please make sure to show that you are comfortable with concepts you can see in the tutorials (posted in this Github issue above). Then use that knowledge to go through “Observability'' topics in https://knative.dev/docs/eventing/accessing-traces/ and write few paragraphs in Synopsis section (please keep the section under 400 words limit - in Google Doc check Tools menu “Word count”) about your experience and ideas what can be improved for user experience that uses logging, distributed tracing and metrics to solve user problems in Knative Eventing. Then in the “Benefits to Community '' section describe your ideas about what would be the user experience after the project is finished in particular when using the command line that processes output from Knative Eventing observability to solve user problems - how do you imagine it is working and solving problems for developers?

You can find detailed guide about writing Gsoc proposal in https://google.github.io/gsocguides/student/writing-a-proposal

Please make sure that your application provide the following information:

Proposal Title

Proposal Summary

Project Size large (350 hour)

Project Technologies

Project Topics 

Synopsis

Benefits to Community

Deliverables

Related Work

Biographical Information

Upload your Proposal as a PDF to GSoc website. You can continue to upload a new proposal pdf until the deadline of April 4th 1800 UTC

For ideas how to write successful proposals please review PDFs from https://blogs.python-gsoc.org/en/

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.