Mimir Alertmanager failing to load config as a global opsgenie_api_key is not specified

[amrit-canva]

Describe the bug

Since upgrading to Mimir 2.13.0, Mimir Alertmanager is not loading our existing config and is giving this error:

grafana-mimir-alertmanager-0 alertmanager ts=2024-07-15T03:18:41.973420345Z caller=multitenant.go:653 level=warn component=MultiTenantAlertmanager msg="error applying config" err="unable to load fallback configuration for anonymous: no global OpsGenie API
 Key set"

To Reproduce

Steps to reproduce the behavior:

Update the alertmanager fallback config to include an Opsgenie receiver and do not set opsgenie_api_key in the global config.

Alertmanager won't come up and in logs an error will be produced indicating that global opsgenie API key is not set.

Expected behavior

opsgenie_api_key is not a mandatory config parameter. In our setup we use opsgenie_api_key_file and specify this at per receiver level, instead of in the global config.

So I expect that the presence of opsgenie_api_key in global config section must not be manadatory.

Environment

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context

Seems like this behaviour is introduced through this check in grafana/alerting repo: https://github.com/grafana/alerting/blob/main/definition/compat.go#L115-L119

[antonio-tolentino]

I'm facing the same issue with the introduction of this new Grafana definitions validation in Mimir 2.13. From Alertmanager perspective api_key_file field for Opsgenie is still entirely supported.

Questions:

• Why the Opsgenie receiver field api_key is being enforced like this ?
• Is the Opsgenie receiver field api_key_file being deprecated for Mimir Alermanager?
• Should I expect that local storage for Mimir alertmanager to be deprecated too?
• Is there another recommend way to inject api_key to avoid this be flat in the file?

[daanschipper]

Encountered the same issue no global Slack API URL set when using api_url_file in a slack_configs receiver. Workaround is to specify a fake url under global.slack_api_url. The value has to be a valid http url. The global value is ignored as it is only a default value when no url is specified under the receiver. Was too soon with validating, Mimir does start up but the alerts fail to send. It seems api_key needs to be set.

mimirtool alertmanager verify alertmanager.yaml did not give any validation error, the changelog mention anything about these field becoming required nor does upstream alertmanager configuration specify anything about global required field, so this was completely unexpected. Is the validation of the configuration correct?

[anthonynashduco]

Hitting this wile trying to set global.slack_api_url_file. Got a valid webhook url in there but it's just failing to start complaining that global.slack_api_url isn't set. Is there an accepted way of not having to set plain text secrets in the config? One that isn't setting the whole config in a secret...

[daanschipper]

Opened a pr on the downstream repo grafana/alerting#301.