KeyError: 'services' in metrics collector

[zeeZ]

What went wrong?

What happened:

Metrics endpoint at /metrics/ returns 500 internal server error:

2024-06-04 08:16:20 source=engine:app google_trace_id=none logger=django.request Internal Server Error: /metrics/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/app/apps/metrics_exporter/views.py", line 22, in get
    result = generate_latest(application_metrics_registry).decode("utf-8")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prometheus_client/exposition.py", line 198, in generate_latest
    for metric in registry.collect():
  File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 97, in collect
    yield from collector.collect()
  File "/etc/app/apps/metrics_exporter/metrics_collectors.py", line 65, in collect
    alert_groups_total, missing_org_ids_1 = self._get_alert_groups_total_metric(org_ids)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/app/apps/metrics_exporter/metrics_collectors.py", line 104, in _get_alert_groups_total_metric
    for service_name in integration_data["services"]:
                        ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'services'

How do we reproduce it?

Unable to reproduce in dev, but this change removed two checks in old lines 104 and 150: https://github.com/grafana/oncall/pull/4300/files#diff-f52a4f487bce825b13cf5540d5c960096703b3cfb73815791bb77fe1c55778d1L104

Grafana OnCall Version

1.5.4, 1.6.0

Product Area

Other

Grafana OnCall Platform?

Docker

User's Browser?

No response

Anything else to add?

Upgraded from 1.3.117 with working metrics to 1.5.4 and then 1.6.0, both versions returning 500

[joeyorlando]

@Ferril can this be closed now that #4695 is merged?