In this blog, we are going to explore the relationship between OpenShift GitOps (Argo CD) and Red Hat Advanced Cluster Management's (RHACM's) policy framework and how they fit together.
While RHACM is Red Hat’s solution for Kubernetes MultiClusterManagement with a strong focus on Governance, OpenShift GitOps (Argo CD) is a very popular Gitops-Engine which is used by many customers and which just reached CNCF's graduated status.
This blog is not covering a general introduction to policies, for this it is highly recommended to check the list of existing blogs on all details of RHACM's-Governance Framework.
In the following we will list the advantages of deploying RHACM-Policies using Argo CD showing concrete examples.
-
RHACM-Policies can be used to install and configure Argo CD consistently either on the Managing or on Managed-Clusters. See an example here. The Policy is marked as part of the
Baseline-Configurationof theNIST SP 800-53standard and could certainly be part of any other standard you like to implement.kind: Policy metadata: name: example-policy annotations: policy.open-cluster-management.io/standards: NIST SP 800-53 policy.open-cluster-management.io/categories: CM Configuration Management policy.open-cluster-management.io/controls: CM-2 Baseline Configurationas an example you could consistently overwrite the default Gitops-Instance in all your namespaces and environments.
spec: channel: stable config: env: - name: DISABLE_DEFAULT_ARGOCD_INSTANCE value: "true" - name: ARGOCD_CLUSTER_CONFIG_NAMESPACES value: <namespace to deploy to>another example would be to configure SSO for your
fleetof Gitops-Instances.spec: sso: provider: keycloakIt offers you the option to
enforceandmonitorthe settings ofGitops-Operator/ArgoCDregardless if you have a
centralizedordecentralizedapproach. This means you can consistently rollout the configuration to your fleet of clusters avoiding any issues which might come frominconsistenciese.g. regarding RBAC and which are later difficult to troubleshoot.This will be achieved by the following objects. Especially by configuring the
Placement-Objectyou can decide on whichClustersthe Policies should be applied.apiVersion: policy.open-cluster-management.io/v1 kind: PlacementBinding metadata: name: example-binding placementRef: apiGroup: cluster.open-cluster-management.io kind: Placement name: example-placement subjects: - apiGroup: policy.open-cluster-management.io kind: Policy name: example-policy --- apiVersion: cluster.open-cluster-management.io/v1beta1 kind: Placement metadata: name: example-placement spec: predicates: - requiredClusterSelector: labelSelector: matchExpressions: [] --- # this object is generated after the evaluation of the rules apiVersion: cluster.open-cluster-management.io/v1beta1 kind: PlacementDecision metadata: name: example-policy-plr-1 labels: cluster.open-cluster-management.io/placement: destinationcluster -
You get
advanced templating featuresoptimized forMulti-Cluster-Managementwhich includesSecrets-Managementwhere you can securely copy a secret from the Hub to a ManagedCluster like in the example below:objectDefinition: apiVersion: v1 data: city: '{{hub fromSecret "" "hub-secret" "city" hub}}' state: '{{hub fromSecret "" "hub-secret" "state" hub}}' kind: Secret metadata: name: copied-secret namespace: targetThe following example shows you how to dynamically configure a
LimitRangeand aResourceQuota:apiVersion: v1 kind: LimitRange metadata: name: mem-limit-range namespace: '{{hub fromConfigMap "" "app-box-config" (printf "%s-namespace" .ManagedClusterName) hub}}' spec: limits: - default: memory: '{{hub fromConfigMap "" "app-box-config" (printf "%s-memory-limit" .ManagedClusterName) hub}}' defaultRequest: memory: '{{hub fromConfigMap "" "app-box-config" (printf "%s-memory-request" .ManagedClusterName) hub}}' type: Container --- apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources namespace: '{{hub fromConfigMap "" "app-box-config" (printf "%s-namespace" .ManagedClusterName) hub}}' spec: hard: requests.cpu: "1" requests.memory: 1Gi limits.cpu: "2" limits.memory: 2Gi requests.nvidia.com/gpu: 4Benefitsof this approach are among others that there is noduplicationof policies (and thus easier maintenance) as you customize specific elements of a policy over various clusters within the fleet.See here a
real worldexample how to create an ApplicationSet using aTemplatizedPoliciesapproach:apiVersion: policy.open-cluster-management.io/v1 kind: Policy metadata: name: policy-as-bootstrap-gitops annotations: policy.open-cluster-management.io/standards: NIST SP 800-53 policy.open-cluster-management.io/categories: CM Configuration Management policy.open-cluster-management.io/controls: CM-2 Baseline Configuration spec: remediationAction: enforce disabled: false policy-templates: - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: gitops-argocd-applicationset spec: remediationAction: inform # will be overridden by remediationAction in parent policy severity: high object-templates: - complianceType: mustonlyhave objectDefinition: apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: as-bootstrap-acm namespace: openshift-gitops spec: generators: - git: repoURL: https://gitlab.example.com/openshift/argocd-cluster-config.git revision: main files: - path: 'cluster-definitions/{{ fromClusterClaim "name" }}/cluster.json' template: metadata: name: 'ocp-{{ fromClusterClaim "name" }}-bootstrap-acm' spec: project: default source: repoURL: https://gitlab.example.com/openshift/argocd-cluster-config.git targetRevision: main path: "cluster-config/overlays/{{`{{cluster.name}}`}}" destination: server: https://kubernetes.default.svc syncPolicy: automated: prune: true selfHeal: true -
RHACM Policy framework provides the option to generate resources (e.g
Roles,Rolebindings) in one or several namespaces based on namespacenames,labelsorexpressions.In RHACM version 2.6 - as you see below - we enhanced our
namespaceSelectorto chose namespaces also bylabelandexpressionwhich gives you more flexibility on which namespaces you like to operate on:namespaceSelector: matchLabels: name: test2 matchExpressions: key: name operator: In values: ["test1", "test2"]Benefit of this feature is that it reduces significantly the Policies you need to configure, e.g. when you want to copy a
ConfigMapinto several namespace or only into the namespaces with a certain name or label. -
You have the capability to merge or patch resources. This means if a Kubernetes-Object has to contain certain values you specify
musthavein case you can tolerate other fields. Else - if the object must match exactly - you have to specifymustonlyhave.A often requested example for
mustonlyhaveis to disable theself-provisioner rolefrom an existing or newly created OpenShift-Cluster:metadata: name: policy-remove-self-provisioner spec: remediationAction: enforce severity: high object-templates: - complianceType: mustonlyhave objectDefinition: kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: self-provisioners annotations: rbac.authorization.kubernetes.io/autoupdate: 'false' subjects: [] roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: self-provisioner -
We provide the option to just monitor resources instead of creating/patching them (can be configured via
inform, versusenforce). It is possible to monitor the status ofanyKubernetes-Object. In the following case we check for namespaces interminatingstatus.spec: remediationAction: inform severity: low object-templates: - complianceType: mustnothave objectDefinition: apiVersion: v1 kind: Namespace status: phase: Terminating
In this example none of the 4 evaluated Clusters has such a violation.
-
Similar to above we provide the option to
deletecertain objects, you would just set theremediationActiontoenforce.Please check here for more examples regarding the previous points.
Please note that one of the most interesting usecases here is to delete the
kubeadmin-secretfrom the managed-clusters.The capability to delete objects is enhanced by specifying a
prune-object-behaviorso you can decide what should happen with the objects once you delete a Policy. Please review here Prune Object Behavior. -
RHACM's Governance framework provides the option to create group of policies (PolicySets), a feature which has both UI and Gitops-Support.
- See how
PolicySetscan be configured using PolicyGenerator - See an example of a
PolicySetbeing stored in git by checking:
apiVersion: policy.open-cluster-management.io/v1beta1 kind: PolicySet metadata: name: certificates-policyset namespace: cert-manager spec: description: "Grouping policies related to certificate handling" policies: - azure-clusterissuer-policy - cert-manager-csv-policy - certification-expiration-policy- See how they look in the UI (examples taken from our Kyverno-Policysets):
- See how
- See how the OpenShift-Hardening-Policyset looks like. You see some policies are
compliant, some others are not and need investigation.
-
You have the possibility to configure how often checks should be performed considering the current status of an evaluated
Object.spec: evaluationInterval: compliant: 10m noncompliant: 10sor for one time jobs
spec.evaluationInterval.compliant: neverThe above example stops evaluating the policy once it is in compliant state. So it enforces it only once. The feature has mainly the advantage to tune environments with many policies to consume less resources.
-
You can use
PolicyGeneratorwhich also can be used for integration ofKyvernoandGatekeeper. The PolicyGenerator is a Kustomize plugin to wrap Kubernetes manifests in Policies and also generate PolicySets. Generation can be done locally or through GitOps. We restrict generation sources to local subdirectories for security but starting with RHACM 2.7 the path can point to a Kustomize directory, which allows additional levels of customization as well as supplying a remote URL in the Kustomize file.
PolicyGenerator can be used in ArgoCD to transform yaml-resources to Policies at Runtime. The integration works via
Custom Tooling as you see here.
Let's take the following example. You want to deploy the following resources together:
Deployment: define which image to run.Service: component can be reached over the network.Ingress: the outside world can access our Service.ConfigMap: configure the component (often makes sense to make this templatized).Secret: supply credentials to the component (often makes sense to make this templatized).NetworkPolicy: restrict the component's attack surface.
So you could place all yaml-files into a folder, together with the checks like deployment must be running and you can
create a single Policy by just configure:
policies:
# Deployment-Policy - start
- name: policy-deployment
categories:
- System-Configuration
controls:
- ApplicationDeployment
manifests:
- path: input/
In the above all yaml-files would be used to generate a single Policy.
Please note that you can also generate Policies from folders which contain a Kustomization.
Let's check the following example-extract:
policies:
- name: policy-gatekeeperlibrary
manifests:
- path: gatekeeperlibrary
points to
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/open-policy-agent/gatekeeper-library/library
Another nice feature of PolicyGenerator is that it helps you to upgrade from PlacementRules to the new Placement-API.
You see in above file that there is both the option to set a PlacementRule or a Placement. You can either specify a name
(when the object already exists in the Cluster) or a path in the Gitrepo to apply the objects. See:
placementPath,placementName or placementRulePath and placementRuleName in the reference file here.
-
Governance focused UI-support (Governance-Dashboard) which enables you to drill down into errors from every single Policy.
See here for an overall Governance overview in RHACM-UI:
See here example of our
build-in policywhich checks if backup is setup correctly:
-
Option to have less or more fine grained checks by using Configuration-Policies.
This means you can create one
Configuration Policyfor every single Kubernetes-Object or bundle many of them. EachConfiguration Policywill be one unit when it comes to check the status in the UI as you see in the screenshot above. Benefit is that it gives you more flexibility when developing custom checks. -
Monitoring- and Ansible-integration (gives you the option to implement
Automated Governance).Those topics have already been explained in several blogs which can be found here just to summarize:
- you can give Policies a
severitysending alerts viaRHACM's Observabilityframework only from high-prio policies - you can invoke Ansible-Jobs from Policies so you could only invoke Ansible from critical policies
- you could use the previously discussed
evaluationIntervalto trigger an Ansible Job at a regular basis.
- you can give Policies a
-
Policies can be used to check for
expiredCertificates in different namespaces Please review the following example where a violation will be created if a certificate is valid for less than 100 hours:
apiVersion: policy.open-cluster-management.io/v1
kind: CertificatePolicy
metadata:
name: certificate-policy-1
namespace: kube-system
label:
category: "System-Integrity"
spec:
# include are the namespaces you want to watch certificate policies in, while exclude are the namespaces you explicitly do not want to watch
namespaceSelector:
include: ["default", "kube-*"]
exclude: ["kube-system"]
# Can be enforce or inform, however enforce doesn't do anything with regards to this controller
remediationAction: inform
# minimum duration is the least amount of time the certificate is still valid before it is considered non-compliant
minimumDuration: 100h
Starting with ACM 2.7 the best way to setup the example and to provide a nice onboarding experience to to apply those
policies using the new dependency feature:
For quickly installing the below policies just execute:
oc apply -f https://github.com/ch-stark/argocdpoliciesblog/tree/main/setuppolicies/policies
This policy configures ArgoCD with PolicyGenerator and it requires GitopsOperator to be installed.
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
name: openshift-gitops-policygenerator
annotations:
policy.open-cluster-management.io/standards: NIST SP 800-53
policy.open-cluster-management.io/categories: CM Configuration Management
policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
spec:
remediationAction: inform
disabled: false
dependencies:
- apiVersion: policy.open-cluster-management.io/v1
compliance: Compliant
kind: Policy
name: openshift-gitops-installed
policy-templates:
- objectDefinition:
apiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
name: openshift-gitops-policygenerator
spec:
dependencies:
- name: openshift-gitops-policygenerator
apiVersion: policy.open-cluster-management.io/v1
compliance: Compliant
kind: Policy
remediationAction: inform
severity: medium
object-templates:
- complianceType: musthave
objectDefinition:
apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
name: openshift-gitops
namespace: openshift-gitops
spec:
repo:
env:
- name: KUSTOMIZE_PLUGIN_HOME
value: /etc/kustomize/plugin
initContainers:
- args:
- -c
- cp /etc/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator/PolicyGenerator
/policy-generator/PolicyGenerator
command:
- /bin/bash
image: registry.redhat.io/rhacm2/multicluster-operators-subscription-rhel8:v2.7
name: policy-generator-install
volumeMounts:
- mountPath: /policy-generator
name: policy-generator
volumeMounts:
- mountPath: /etc/kustomize/plugin/policy.open-cluster-management.io/v1/policygenerator
name: policy-generator
volumes:
- emptyDir: {}
name: policy-generator
kustomizeBuildOptions: --enable-alpha-plugins
- complianceType: musthave
objectDefinition:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: openshift-gitops-policy-admin
rules:
- verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiGroups:
- policy.open-cluster-management.io
resources:
- policies
- placementbindings
- verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiGroups:
- apps.open-cluster-management.io
resources:
- placementrules
- verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiGroups:
- cluster.open-cluster-management.io
resources:
- placements
- placements/status
- placementdecisions
- placementdecisions/status
- complianceType: musthave
objectDefinition:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: openshift-gitops-policy-admin
subjects:
- kind: ServiceAccount
name: openshift-gitops-argocd-application-controller
namespace: openshift-gitops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: openshift-gitops-policy-admin
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
name: binding-openshift-gitops-policygenerator
placementRef:
name: placement-openshift-gitops-policygenerator
kind: PlacementRule
apiGroup: apps.open-cluster-management.io
subjects:
- name: openshift-gitops-policygenerator
kind: Policy
apiGroup: policy.open-cluster-management.io
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
name: placement-openshift-gitops-policygenerator
spec:
clusterSelector:
matchExpressions:
- {key: name, operator: In, values: ["local-cluster"]}
apiVersion: policy.open-cluster-management.io/v1
This Policy installs an Application, in the following Gatekeeper will be installed
---
kind: Policy
metadata:
name: policy-gatekeeper-application
namespace: default
annotations:
policy.open-cluster-management.io/categories: CM Configuration Management
policy.open-cluster-management.io/standards: NIST SP 800-53
policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
spec:
disabled: false
dependencies:
- apiVersion: policy.open-cluster-management.io/v1
compliance: Compliant
kind: Policy
name: openshift-gitops-policygenerator
policy-templates:
- objectDefinition:
apiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
name: policy-application-gatekeeper
spec:
remediationAction: inform
severity: low
namespaceSelector:
exclude:
- kube-*
include:
- default
object-templates:
- complianceType: musthave
objectDefinition:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: policiesgatekeeper
namespace: openshift-gitops
spec:
destination:
namespace: openshift-gitops
server: https://kubernetes.default.svc
project: default
source:
path: .
repoURL: https://github.com/ch-stark/gatekeeper-examples
targetRevision: HEAD
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: false
prune: true
pruneObjectBehavior: DeleteIfCreated
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
name: gatekeeper-application-placement
namespace: default
placementRef:
name: placement-openshift-gitops-policygenerator
apiGroup: apps.open-cluster-management.io
kind: PlacementRule
subjects:
- name: gatekeeper-application
apiGroup: policy.open-cluster-management.io
kind: Policy
See some of the Policies being synced onto the Hub-Cluster using ArgoCD-Applications in the Governance-View.
A nice example how ArgoCD can be configured to optimize the interaction with RHACM-policies has been the following. It
turned out that as the RHACM-Policy-Controller is copying Policies into a namespace (representing a managed cluster)
therefore ArgoCD-Applications became out-of-sync. This can be fixed by setting the resource tracking method to
label which is
already included in the examples.
For monitoring you can benefit from RHACM's Gatekeeper Integration by using this Gatekeeper-Constraint together with
PolicyGenerator.
---
# Require applications deploy with ArgoProj
# To use this, set Argo CD tracking method to annotation or annotation+label. https://argo-cd.readthedocs.io/en/stable/user- guide/resource_tracking/
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredAnnotations
metadata:
name: must-be-deployed-by-argocd
spec:
enforcementAction: deny
match:
namespaces:
- "*"
kinds:
- apiGroups: [""]
kinds: ["ReplicaSet", "Deployment"]
parameters:
message: "All resources must have a `argocd.argoproj.io/tracking-id` annotation."
annotations:
- key: argocd.argoproj.io/tracking-id
# Matches annotation format
allowedRegex: ^[a-zA-Z0-9-]+:.+$
This overview had the purpose to explain why it is a good idea to use Policies together with GitOpsOperator/ArgoCD. Both approaches can heavily benefit from each other. Certainly it needs to be highlighted that the focus of RHACM-Policies is to
support customers becoming compliant from a technical point of view. It can be even seen as a ArgoCD extension to Governance. You get all the great benefits highlighted above out of the box.
Both sites will provide further features in the future like enhanced Operator Management and we think of
further integration possibilities. We would love to get your feedback and thoughts on this topic.