Skip to content

Latest commit

 

History

History
 
 

monitoring

README.md

Getting started

Files in this folder will deploy:

  • fluentd for APM metrics
  • grafana for visualization
  • prometheus using (prometheus-opearator) with custom rules for metrics collection, storage and evaluation
  • wca as daemonset (on nodes marked with label goal=service) - image build instructions here
  • Kubernetes dashboard for graphic cluster interface
  1. You need to create dedicated namespaces for those applications like this:
kubectl create -f namespaces.yaml
  1. Create hostPath based directory as grafana volume and Prometheus (PV).
# on master node
sudo mkdir /var/lib/grafana
sudo chmod o+rw /var/lib/grafana
sudo mkdir /var/lib/prometheus
sudo chmod o+rw /var/lib/prometheus


kubectl create -f prometheus/persistent_volume.yaml

Grafana is deployed on master and hostPath as volume at accessible /var/lib/grafana

  1. Using kustomize (-k) deploy all applications:
kubectl apply -k .

Note: in case of

unable to recognize ".": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"

warnings, please run kubectl apply -k . once again. This is a problem of invalid order of objects when CRDs are created by kustomize and prometheus-operator.

You can check progress of deployment using kubectl get -k ..

Dashboard

To access Kubernetes Dashboard after deployment token is needed. It can be viewed by using following command:

kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

Kubernetes Dashboard version used in this example is v2.0.0-beta4:

https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml

Following instruction was used to get the token:

https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

Access

Dashboard is exposed at: https://worker-node:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#

Prometheus can be exposed by kubectl proxy kubectl port-forward --namespace prometheus service/prometheus 8090:9090. It will be available at: localhost:8090.

Grafana can be exposed by kubectl proxy kubectl port-forward --namespace grafana deployment/grafana 8091:3000. It will be available at: localhost:8091.

For debug purpose, it might be usefully exposed this service as NodePort. An example is below. It is not recommended for production deployment.

kubectl expose pod prometheus-prometheus-0 --type=NodePort --port=9090 --name=prometheus-nodeport-service --namespace prometheus
kubectl patch service prometheus-nodeport-service --namespace=prometheus --type='json' --patch='[
{"op": "replace", "path": "/spec/ports/0/nodePort", "value":30900}]'

kubectl expose deployment grafana --type=NodePort --port=3000 --name=grafana-nodeport-service --namespace grafana
kubectl patch service grafana-nodeport-service  --namespace=grafana --type='json' --patch='[
{"op": "replace", "path": "/spec/ports/0/nodePort", "value":32135}]'

Log in using default credentials: user: admin, password: admin and change password when prompted.

Note that after deployment you will need to add prometheus source in Grafana.

URL: http://prometheus.prometheus:9090
Access: Server(default)

Grafana configuration (addons)

Grafana boomtable plugin

Required for 2LM contention demo:

https://grafana.com/grafana/plugins/yesoreyeram-boomtable-panel

Grafana is deployed on master and hostPath as volume at accessible /var/lib/grafana

kubectl exec --namespace grafana -ti `kubectl get pod -n grafana -oname --show-kind=false | cut -f 2 -d '/'` bash
grafana-cli plugins install yesoreyeram-boomtable-panel
# restart 
kubectl delete -n grafana `kubectl get pod -n grafana -oname`

Configuring Kubernetes-app for Grafana

When creating Kubernetes cluster in Grafana following parameters should be entered:

After choosing correct checkboxes you need to fill TLS Auth Details:

# CA cert
kubectl config view --raw -ojsonpath="{@.clusters[0].cluster.certificate-authority-data}" | base64 -d
# Client cert
kubectl config view --raw -ojsonpath="{@.users[0].user.client-certificate-data}" | base64 -d
# Client key
kubectl config view --raw -ojsonpath="{@.users[0].user.client-key-data}" | base64 -d

Troubleshooting

Access WCA and fluentd

Both applications are running in host network namespace as daemonsets:

Cleaning up

Warning!: this removes all the objects (excluding CRDs and namespaces), but it will not remove hostPath based data for Grafana and Prometheus.

kubectl delete -f prometheus/persistent_volume.yaml
kubectl delete persistentvolumeclaim/prometheus-prometheus-db-prometheus-prometheus-0 -n prometheus
kubectl delete -k .
kubectl delete -f namespaces.yaml

Remove namespace if stuck in "Terminating" state

Warning!: there might be orphaned resources left after

kubectl proxy &
for NS in fluentd grafana prometheus wca; do
kubectl get namespace $NS -o json | sed '/kubernetes/d' | curl -k -H "Content-Type: application/json" -X PUT --data-binary @- 127.0.0.1:8001/api/v1/namespaces/$NS/finalize
done

Service Monitor configuration troubleshooting

https://github.com/coreos/prometheus-operator/blob/master/Documentation/troubleshooting.md#L38

Missing metrics from node_exporter (e.g. node_cpu)

This setup uses new version of node_exporter (>18.0) and Grafana kubernetes-app is based on older scheme from node_exporter v 0.16 Prometheus rules "16-compatibility-rules-new-to-old" are used to configured new evaluation rules from backward compatibility.

Kubernetes capacity analysis

# First label your namespaces with name of namespace
kubectl  | awk '{print $1}' | xargs -iX kubectl label ns X name=X
# then you can use kube-capacity tool to analysie used/utilized and free node resources:
# with pods 
kube-capacity -n name=default -p -u
# with pods only from service nodes
kube-capacity -n name=default --node-labels goal=service -p -u

Useful links