Fix grafana vm metrics 404 error in multi cluster

[w13915984028]

When accessing vm metrics from Rancher to Harvester to VM, 404 error is return due to wrong appSubUrl

Problem:

When accessing vm metrics from Rancher to Harvester to VM, 404 error is return.

Such a field is returned:

"appSubUrl": "/k8s/clusters/c-m-8zxqjrgg/kubevirt.io.virtualmachine/default/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy",

The expected value is:

"appSubUrl": "/k8s/clusters/c-m-8zxqjrgg/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy",

Solution:

In nginx config, add long match for http referer field, to filter the path like kubevirt.io.virtualmachine/default/vm2

Referer:
https://VIP/dashboard/harvester/c/c-m-8zxqjrgg/kubevirt.io.virtualmachine/default/vm2

is matched by nginx rule:

~.*/dashboard/harvester/c/(c-m-.+)/kubevirt.io.virtualmachine/.*/.* '"appSubUrl":"/k8s/clusters/$1/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy"';

In the Harvester dashboard, the vm metrics is referred as:

Referer:
https://VIP/dashboard/harvester/c/c-m-8zxqjrgg/harvesterhci.io.dashboard

is matched by nginx rule:

~.*/dashboard/harvester/c/(c-m-.+)/.* '"appSubUrl":"/k8s/clusters/$1/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy"';

Related Issue:
harvester/harvester#4343
Test plan:

1. Install a new Harvester cluster with this fix, and enable rancher-monitoring addon
2. Import this new cluster into Rancher
3. Create a new VM in Harvester
4. View VM metrics from Harvester local, from both dashboard and VM page
5. View VM metrics from Rancher->Harvester, from both dashboard and VM page

[w13915984028]

FYI: how to validate the fix quickly.

Quick local validation of this issue, without installing a new cluster

Replace the nginx related configmap, and replace the grafana POD, then check the result

cat > a1.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-nginx-proxy-config
namespace: cattle-monitoring-system
labels:
  app.kubernetes.io/name: grafana
  app.kubernetes.io/instance: rancher-monitoring
data:
nginx.conf: |-
  worker_processes      auto;
  error_log             /dev/stdout warn;
  pid                   /var/cache/nginx/nginx.pid;

  events {
     worker_connections 1024;
  }

  http {
    include       /etc/nginx/mime.types;
    log_format    main '[$time_local - $status] $remote_addr - $remote_user $request ($http_referer)';

    proxy_connect_timeout       10;
    proxy_read_timeout          180;
    proxy_send_timeout          5;
    proxy_buffering             off;
    proxy_cache_path            /var/cache/nginx/cache levels=1:2 keys_zone=my_zone:100m inactive=1d max_size=10g;

    map $http_upgrade $connection_upgrade {
      default upgrade;
      '' close;
    }

    server {
      listen          8080;
      access_log      /dev/stdout;

      gzip            on;
      gzip_min_length 1k;
      gzip_comp_level 2;
      gzip_types      text/plain application/javascript application/x-javascript text/css application/xml text/javascript image/jpeg image/gif image/png;
      gzip_vary       on;
      gzip_disable    "MSIE [1-6]\.";

      proxy_set_header Host $host;

      location /api/dashboards {
        proxy_pass     http://localhost:3000;
      }

      location /api/search {
        proxy_pass     http://localhost:3000;

        sub_filter_types application/json;
        sub_filter_once off;
      }

      location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass http://localhost:3000;
      }

      location / {
        proxy_cache         my_zone;
        proxy_cache_valid   200 302 1d;
        proxy_cache_valid   301 30d;
        proxy_cache_valid   any 5m;
        proxy_cache_bypass  $http_cache_control;
        add_header          X-Proxy-Cache $upstream_cache_status;
        add_header          Cache-Control "public";

        proxy_pass     http://localhost:3000/;

        sub_filter_once off;

        #sub_filter '"appSubUrl":""' '"appSubUrl":"/api/v1/namespaces/{{ template "grafana.namespace" . }}/services/http:{{ template "grafana.fullname" . }}:{{ .Values.service.port }}/proxy"';

        sub_filter '"appSubUrl":""' $final_appSubUrl;

        sub_filter ':"/avatar/' ':"avatar/';

        if ($request_filename ~ .*\.(?:js|css|jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm)$) {
          expires             90d;
        }

        rewrite ^/k8s/clusters/.*/proxy(.*) /$1 break;

      }
    }

    map $http_referer $final_appSubUrl {
      ~.*/k8s/clusters/(c-m-.+)/.*      '"appSubUrl":"/k8s/clusters/$1/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy"';
      ~.*/dashboard/harvester/c/(c-m-.+)/kubevirt.io.virtualmachine/.*/.* '"appSubUrl":"/k8s/clusters/$1/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy"';
      ~.*/dashboard/harvester/c/(c-m-.+)/.*      '"appSubUrl":"/k8s/clusters/$1/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy"';
      default '"appSubUrl":"/k8s/clusters/local/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy"';
    }

  }
EOF

nm=cattle-monitoring-system

kubectl delete configmap -n $cm grafana-nginx-proxy-config

kubectl apply -f a1.yaml

kubectl get pods -n $cm

# delete the grafana POD manually
kubectl delete pod -n $cm rancher-monitoring-grafana-5fb8756b6-xvf6s
kubectl get pods -n $cm

wait until new pod is ready

[w13915984028]

The target of nginx map rule is to filter out multi cluster name.

Seems regex c-m-[a-z,A-Z,0-9]+ can match the exact multi-cluster cluster name, without including the following fields started with /. e.g., the c-m-8zxqjrgg/kubevirt.io.virtualmachine/default/vm2 will be only matched as c-m-8zxqjrgg.

Will check if Rancher multi-cluster naming will include any special chars, when not, we can further optimize&simplify the nginx rule.

[connorkuehl]

Looks great!

[guangbochen]

Looks great on both single cluster and MCM UI after the patch, thanks.

[guangbochen]

@mergify backport v1.2

[mergify]

backport v1.2

✅ Backports have been created

• #556 Fix grafana vm metrics 404 error in multi cluster (backport #555) has been created for branch v1.2