- Release Signoff Checklist
- Summary
- Motivation
- Proposal
- Design Details
- Production Readiness Review Questionnaire
- Implementation History
- Drawbacks
- Alternatives
- Infrastructure Needed (Optional)
Items marked with (R) are required prior to targeting to a milestone / release.
- (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
- (R) KEP approvers have approved the KEP status as
implementable - (R) Design details are appropriately documented
- (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- e2e Tests for all Beta API Operations (endpoints)
- (R) Ensure GA e2e tests meet requirements for Conformance Tests
- (R) Minimum Two Week Window for GA e2e tests to prove flake free
- (R) Graduation criteria is in place
- (R) all GA Endpoints must be hit by Conformance Tests
- (R) Production readiness review completed
- (R) Production readiness review approved
- "Implementation History" section is up-to-date for milestone
- User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
- Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
We propose to enhance the observability, troubleshooting, and debugging capabilities of core Kubernetes components by implementing a flagz endpoint for all components. This will provide users with real-time visibility into the command-line flags each component was started with, aiding in the diagnosis of configuration issues and unexpected behavior
This proposal extends the ideas presented in the Component Statusz KEP, however, it proposes a distinct flagz endpoint to enhance visibility into component configurations. By providing real-time visibility into the active configuration of Kubernetes components, flagz enables users to quickly spot misconfigurations that could lead to system instability or outages
- Provide users with the ability to dynamically inspect and understand the active flags of running Kubernetes components
- Replace existing monitoring solutions: metrics, logs, traces
- Provide information about components that are not directly accessible due to network restrictions
We are proposing to add a new endpoint, /flagz on all core Kubernetes components which returns the command-line flags each component was started with.
- As a cluster administrator, I want to be able to query the /flagz endpoint on individual Kubernetes components so I can inspect their runtime flag values and diagnose unexpected behavior
- As a developer, I want to access the /flagz endpoint after deploying a component with modified flags so I can verify that my changes have been applied correctly
-
No sensitive data exposed
We will ensure that no sensitive data is exposed through flagz and that access to this endpoint is gated by using system-monitoring group.
-
Performance impact
We will take care to carefully consider the computational overhead associated with the flagz endpoint to avoid negatively impacting the server's overall performance.
-
Essential data only
We will only display the command-line flag values for a component in this page and nothing more.
-
Premature Dependency on Unstable Format
The alpha release will explicitly support plain text format only, making it clear that the output is not intended for parsing or automated monitoring. The feature will be secured behind a feature gate that is disabled by default, ensuring users opt-in consciously. will support specifying a version as a query parameter to facilitate future transitions to structured schema changes without breaking existing integrations, once the usage patterns and requirements are better understood.
Initially, the flagz page will exclusively support a plain text format for responses. In future, when we have established a structured schema for the response, we can support specifying a version param, like /flagz?version=2, to ensure future compatibility. Through this, we can evolve the endpoint and introduce changes to the response schema without impacting clients relying on previous versions, ensuring backward compatibility.
Access to the endpoint will be limited to members of the system:monitoring group (ref), which is how paths like /healthz, /livez, /readyz are restricted today.
- Method: GET
- Endpoint: /flagz
- Header:
Content-Type: text/plain - Body: empty
- Flag Name: The name of the command-line flag or configuration option
- Value: The current value assigned to the flag at runtime
----------------------------
title: Kubernetes Flagz
description: Command line flags that Kubernetes component was started with.
----------------------------
default-watch-cache-size=100
delete-collection-workers=1
enable-garbage-collector=true
encryption-provider-config-automatic-reload=false
...
[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.
staging/src/k8s.io/component-base/zpages/flagz:2024-10-08-100%
- Ensure existence of the /flagz endpoint
- Feature implemented behind a feature flag
- Feature implemented for apiserver
- Ensure proper tests are in place.
- Gather feedback from developers
- Feature implemented for other Kubernetes Components
- Several cycles of bake-time
In alpha, no changes are required to maintain previous behavior. And the feature gate can be turned on to make use of the enhancement.
The primary purpose of the flagz page is to provide information about the individual component itself. While the underlying data format may be enhanced over time, the endpoint itself will remain a reliable source of flag information across Kubernetes versions.
We will target this feature behind a flag ComponentFlagz
- Feature gate (also fill in values in
kep.yaml)- Feature gate name: ComponentFlagz
- Components depending on the feature gate:
- apiserver
- kubelet
- scheduler
- controller-manager
- kube-proxy
Yes it will expose a new flagz endpoint.
Yes. But it will remove the flagz endpoint
It will expose the flagz endpoint again
Unit test will be introduced in alpha implementation.
This feature should not cause rollout failures. If it does, we can disable the feature. In the worst case, it is possible it could cause runtime failures, but it is highly unlikely we would not detect this with existing tests.
- apiserver_request_duration_seconds metric that will tell us if there's a spike in request latency of kube-apiserver that might indicate that the flagz endpoint is interfering with the component's core functionality or causing delays in processing other requests
This should not be necessary since we're adding a new endpoint with no dependencies. The rollback simply removes the endpoint.
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
No.
They can curl a component's flagz endpoint. However, its important to note that this feature is not directly used by workloads, its a debug feature.
- They can curl a component's
flagzendpoint, which should return data.
This is a debugging feature and not something that workloads depend on. Therefore, any failure in calls to the /flagz endpoint is likely a bug or external issue like a network problem. So the SLO is essentially 100% success rate for the calls to flagz.
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
This enhancement proposes data that can be used to determine the health of the component.
Are there any missing metrics that would be useful to have to improve observability of this feature?
No. We are open to input.
No, each component's flagz is independent.
No
No.
No.
No.
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
Minimal to no impact expected, since the flagz endpoint is designed to provide lightweight, readily available information about a component's command-line flags. Retrieving this data should not introduce significant delays or resource contention that would noticeably affect the performance of existing operations covered by SLIs/SLOs.
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
Enabling and using the /flagz feature is expected to result in a negligible increase in resource usage across the components where it is implemented. The primary reason for this is that the endpoint will mainly serve static, readily available information about the component's command line flags.
Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?
No.
flagz endpoint for apiserver will not be available if the API server itself is down.
Overreliance on flagz for critical monitoring. We will clearly document the intended use cases and limitations of the flagz endpoint, emphasizing that it's primarily for informational and troubleshooting purposes, not real-time monitoring or alerting.
The feature can be disabled by setting the feature-gate to false if the performance impact of it is not tolerable.