Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OCPBUGS-31353: Minimize wildcard privileges #1190

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

OCPBUGS-31353: Minimize wildcard privileges #1190

wants to merge 2 commits into from

Conversation

alebedev87
Copy link
Contributor

@alebedev87 alebedev87 commented Feb 5, 2025 •
edited
Loading

  • Added operand namespaces in the operator payload to be able to create local roles.
  • Added local roles and rolebinding in the operand namespaces to manage ingresscontrollers and canary.
  • Added a local role for the certificate management in openshift-config-managed namespace.
  • Limited cluster permissions on namespace resource to operand namespaces.
  • Moved role and rolebinding permissions from cluster level to openshift-config namespace (local role).
  • Moved secret and configmap permissions from cluster level to openshift-config and openshift-config-managed namespace (local roles).

TODO: clusterroles/clusterrolebinding.

@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Feb 5, 2025
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 5, 2025
@openshift-ci-robot
Copy link
Contributor

@alebedev87: This pull request references Jira Issue OCPBUGS-31353, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.19.0) matches configured target version for branch (4.19.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

  • Added operand namespaces in the operator payload to be able to create local roles.
  • Added local roles and rolebinding in the operand namespaces to manage ingresscontrollers and canary.
  • Added a local role for the certificate management in openshift-config-manager namespace.
  • Limited cluster permissions on namespace resource to operand namespaces.

TODO: clusterroles/clusterrolebinding and roles/rolebindings.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Feb 5, 2025
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Feb 5, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot requested a review from melvinjoseph86 February 5, 2025 14:30
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Feb 5, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from alebedev87. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@alebedev87
Copy link
Contributor Author

/test e2e-aws-operator

@alebedev87
Copy link
Contributor Author

The local role for the ingress operator cannot be created due to the missing openshift-config-manager. This namespace cannot be a part of the cluster-ingress-operator payload like the operand namespaces. So, the permissions required for this have to be in the cluster scope.

E0205 15:10:32.266074       1 task.go:122] "Unhandled Error" err="error running apply for role \"openshift-config-manager/ingress-operator\" (379 of 911): namespaces \"openshift-config-manager\" not found" logger="UnhandledError"

@alebedev87 alebedev87 force-pushed the least-privilege-operand-namespaces-role-bindings branch from d248a94 to f84e759 Compare February 6, 2025 13:55
@alebedev87
Copy link
Contributor Author

/test e2e-aws-operator

@alebedev87
Copy link
Contributor Author

alebedev87 commented Feb 6, 2025 •
edited
Loading 

E0206 15:21:17.315795       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-config-managed\"" logger="UnhandledError"
W0206 15:21:18.879778       1 reflector.go:561] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:openshift-ingress-operator:ingress-operator" cannot list resource "configmaps" in API group "" in the namespace "openshift-config-managed"
E0206 15:21:18.879826       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-config-managed\"" logger="UnhandledError"
W0206 15:21:20.504426       1 reflector.go:561] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:openshift-ingress-operator:ingress-operator" cannot list resource "configmaps" in API group "" in the namespace "openshift-config"

The secret and configmap readonly access should be given on the cluster level since the operator's payload (manifests for CVO) cannot create roles in not existing (at installation time) openshift-config-managed and openshift-config namespaces.

@alebedev87
Copy link
Contributor Author

Also, found another PR which was doing a similar thing with the operand namespace creation. Looks like David Eads was against this idea though.

@alebedev87 alebedev87 force-pushed the least-privilege-operand-namespaces-role-bindings branch from f84e759 to b8c781a Compare February 14, 2025 15:21
@alebedev87
Copy link
Contributor Author

alebedev87 commented Feb 14, 2025 •
edited
Loading

Local role for the certificate management was created in a wrong namespace. Normally openshift-config-managed namespace should exist at the moment of CIO installation, from the release image manifests:

$ oc adm release extract --to manifests quay.io/openshift-release-dev/ocp-release:4.19.0-ec.1-x86_64
$ cd manifests
$ ag 'kind: Namespace' -A20 | grep 'name: ' | grep config
0000_00_config-operator_00_namespace.yaml:14-  name: openshift-config-operator
0000_10_config-operator_01_openshift-config-managed-ns.yaml:13-  name: openshift-config-managed
0000_10_config-operator_01_openshift-config-ns.yaml:16-  name: openshift-config

CIO manifests are applied later: 0000_50_cluster-ingress-operator (manifests are ordered by name).

So, I'm returning to the approach with operand namespaces and local roles after a discussion with CVO people.

@alebedev87
Copy link
Contributor Author

/test e2e-aws-operator

@openshift-ci-robot
Copy link
Contributor

@alebedev87: This pull request references Jira Issue OCPBUGS-31353, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.19.0) matches configured target version for branch (4.19.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

In response to this:

  • Added operand namespaces in the operator payload to be able to create local roles.
  • Added local roles and rolebinding in the operand namespaces to manage ingresscontrollers and canary.
  • Added a local role for the certificate management in openshift-config-managed namespace.
  • Limited cluster permissions on namespace resource to operand namespaces.
  • Moved role and rolebinding permissions from cluster level to openshift-config local role.

TODO: clusterroles/clusterrolebinding.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@alebedev87 alebedev87 force-pushed the least-privilege-operand-namespaces-role-bindings branch from b8c781a to 57eb6a5 Compare February 17, 2025 11:26
@alebedev87
Copy link
Contributor Author

/test e2e-aws-operator

@alebedev87
Copy link
Contributor Author

Read only verbs added for configmap resource in openshift-config namespace to let the "sync error pages controller" watch cluster admin's custom pages.

@alebedev87
Copy link
Contributor Author

"certificate-publisher controller" needs to watch secrets and configmaps from openshift-config-managed namespace to resync the router certificate and CA.

@alebedev87 alebedev87 force-pushed the least-privilege-operand-namespaces-role-bindings branch from 57eb6a5 to 3302c40 Compare February 17, 2025 13:23
@alebedev87
Copy link
Contributor Author

Forgot the role bindings in openshift-config, openshift-config-managed namespaces.

@alebedev87
Copy link
Contributor Author

/test e2e-aws-operator

2 similar comments
@alebedev87
Copy link
Contributor Author

/test e2e-aws-operator

@alebedev87
Copy link
Contributor Author

/test e2e-aws-operator

@alebedev87
Copy link
Contributor Author

Namespaces resource with concrete names didn't work:

W0217 14:25:24.061762       1 reflector.go:561] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:openshift-ingress-operator:ingress-operator" cannot list resource "namespaces" in API group "" at the cluster scope
E0217 14:25:24.061810       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope" logger="UnhandledError"

@alebedev87
Copy link
Contributor Author

alebedev87 commented Feb 18, 2025 •
edited
Loading

Daemonset and deployment resources (which were removed from cluster scope) had to be added to all namespaces: operator, operands and configs.

Operator:

E0217 14:15:37.838650       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"daemonsets\" in API group \"apps\" in the namespace \"openshift-ingress-operator\"" logger="UnhandledError"

Operands:

E0217 14:18:58.291874       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"daemonsets\" in API group \"apps\" in the namespace \"openshift-ingress\"" logger="UnhandledError"

E0217 14:15:35.493573       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"openshift-ingress-canary\"" logger="UnhandledError"

Configs:

E0217 14:12:00.932803       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"openshift-config-managed\"" logger="UnhandledError"
E0217 13:54:04.127305       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"daemonsets\" in API group \"apps\" in the namespace \"openshift-config-managed\"" logger="UnhandledError"

E0217 13:54:12.217328       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"openshift-config\"" logger="UnhandledError"
E0217 14:11:54.670664       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.DaemonSet: failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"daemonsets\" in API group \"apps\" in the namespace \"openshift-config\"" logger="UnhandledError"

@alebedev87
Copy link
Contributor Author

alebedev87 commented Feb 18, 2025 •
edited
Loading

Roles and rolebindings had to be added to all the namespaces too:

Operands:

E0217 13:59:00.708216       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Role: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"roles\" in API group \"rbac.authorization.k8s.io\" in the namespace \"openshift-ingress\"" logger="UnhandledError"
E0217 13:58:56.685918       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: rolebindings.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"rolebindings\" in API group \"rbac.authorization.k8s.io\" in the namespace \"openshift-ingress\"" logger="UnhandledError"

E0217 13:36:54.827324       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Role: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"roles\" in API group \"rbac.authorization.k8s.io\" in the namespace \"openshift-ingress-canary\"" logger="UnhandledError"

Configs:

E0217 14:04:17.341236       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.Role: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"roles\" in API group \"rbac.authorization.k8s.io\" in the namespace \"openshift-config-managed\"" logger="UnhandledError"
E0217 14:04:21.832052       1 reflector.go:158] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:106: Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: rolebindings.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:openshift-ingress-operator:ingress-operator\" cannot list resource \"rolebindings\" in API group \"rbac.authorization.k8s.io\" in the namespace \"openshift-config-managed\"" logger="UnhandledError"

@alebedev87 alebedev87 mentioned this pull request Feb 18, 2025
Closed
@alebedev87
OCPBUGS-31353: Minimize wildcard privileges
e58b7a6 
- Added operand namespaces in the operator payload to be able to create local roles.
- Added local roles and rolebinding in the operand namespaces to manage ingresscontrollers and canary.
- Added a local role for the cluster domain router certificate management in openshift-config-managed namespace.
- Limited cluster permissions on namespace resource to operand namespaces.
- Moved role and rolebinding permissions from cluster level to openshift-config namespace (local role).
- Moved secret and configmap permissions from cluster level to openshift-config and openshift-config-managed namespace (local roles).

TODO: clusterroles/clusterrolebinding, CRDs.
@alebedev87
tmp: add role, rolebindings, deployments, daemonsets resources to all…
17e813b 
… local roles

The question is why we need them in all the local roles.
@alebedev87 alebedev87 force-pushed the least-privilege-operand-namespaces-role-bindings branch from 8cf95df to 17e813b Compare February 18, 2025 08:41
@alebedev87 alebedev87 mentioned this pull request Feb 18, 2025
Closed
@alebedev87
Copy link
Contributor Author

/test e2e-gcp-operator

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Feb 18, 2025

@alebedev87: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-operator 17e813b link true /test e2e-gcp-operator
ci/prow/e2e-aws-operator 8cf95df link true /test e2e-aws-operator

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@melvinjoseph86 melvinjoseph86 Awaiting requested review from melvinjoseph86

Assignees
No one assigned
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alebedev87 @openshift-ci-robot