Helm chart: User provided annotations should supersed default ones

[Segflow]

Feature Description

The consul helm chart offers the global.annotations value, but they are being superseded by the default ones because the helm template is as follow:

{{- if .Values.server.annotations }}
  {{- tpl .Values.server.annotations . | nindent 8 }}
{{- end }}
{{- if (and .Values.global.metrics.enabled .Values.global.metrics.enableAgentMetrics) }}
"prometheus.io/scrape": "true"
"prometheus.io/path": "/v1/agent/metrics"
"prometheus.io/port": "8500"
{{- end }}

I believe the user provided annotations should be applied last so they supersede everything else (in this case the prometheus annotations)

Use Case(s)

When metrics.enabled is true, these annotations are added to all pods:

    prometheus.io/path: /v1/agent/metrics
    prometheus.io/port: "8500"
    prometheus.io/scrape: "true"

Which works fine when the scraper is prometheus.

The default format for the metrics endpoint is json, and if we want to have metrics in prometheus format we need to add the query param format=prometheus. Unfortunately there is no "standard" annotation for that.

When using datadog, scraping fails because it does not understand json. Manually updating the pod annotations to:

    prometheus.io/path: /v1/agent/metrics?format=prometheus
    prometheus.io/port: "8500"
    prometheus.io/scrape: "true"

Fixed the issue for me.

Contributions

If you are ok with this, I can implement the change.

[Segflow]

Any update on this?