Issue with Grafana OnCall Integration with Mattermost and API Errors

[MalaGosha]

What went wrong?

I’m facing issues integrating Grafana OnCall with Mattermost and encountering errors when trying to use the API.

Environment:
• My Grafana OnCall running on an EC2 instance via docker-compose (latest version)
• Grafana instance running on a separate EC2 instance via Docker (version 11.4.0)
• Terraform creates the Grafana OnCall instance and connects it to my Redis, RabbitMQ, and PostgreSQL
• Grafana OnCall plugin installed in my instance Grafana
• I want to using webhooks for sending notifications (to Mattermost)

What happened:

1. When I try to create an integration using a webhook I get this information in the container app-engine-1 logs:

2025-02-25 07:58:22 Internal Server Error: /api/internal/v1/alert_receive_channels
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py", line 220, in _get_response
    response = response.render()
               ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/template/response.py", line 114, in render
    self.content = self.rendered_content
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/response.py", line 74, in rendered_content
    ret = renderer.render(self.data, accepted_media_type, context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/renderers.py", line 727, in render
    context = self.get_context(data, accepted_media_type, renderer_context)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/renderers.py", line 703, in get_context
    'extra_actions': self.get_extra_actions(view, response.status_code),
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/renderers.py", line 615, in get_extra_actions
    return view.get_extra_action_url_map()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/viewsets.py", line 204, in get_extra_action_url_map
    url = reverse(url_name, self.args, self.kwargs, request=self.request)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/reverse.py", line 47, in reverse
    url = _reverse(viewname, args, kwargs, request, format, **extra)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/reverse.py", line 62, in _reverse
    return request.build_absolute_uri(url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: PathPrefixedPagePagination.paginate_queryset.<locals>.<lambda>() takes 0 positional arguments but 1 was given
2025-02-25 07:58:22 source=engine:app google_trace_id=none logger=django.request Internal Server Error: /api/internal/v1/alert_receive_channels
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py", line 220, in _get_response
    response = response.render()
               ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/template/response.py", line 114, in render
    self.content = self.rendered_content
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/response.py", line 74, in rendered_content
    ret = renderer.render(self.data, accepted_media_type, context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/renderers.py", line 727, in render
    context = self.get_context(data, accepted_media_type, renderer_context)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/renderers.py", line 703, in get_context
    'extra_actions': self.get_extra_actions(view, response.status_code),
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/renderers.py", line 615, in get_extra_actions
    return view.get_extra_action_url_map()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/viewsets.py", line 204, in get_extra_action_url_map
    url = reverse(url_name, self.args, self.kwargs, request=self.request)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/reverse.py", line 47, in reverse
    url = _reverse(viewname, args, kwargs, request, format, **extra)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rest_framework/reverse.py", line 62, in _reverse
    return request.build_absolute_uri(url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: PathPrefixedPagePagination.paginate_queryset.<locals>.<lambda>() takes 0 positional arguments but 1 was given
2025-02-25 07:58:22 source=engine:app google_trace_id=none logger=root inbound latency=0.245827 status=500 method=GET path=/api/internal/v1/alert_receive_channels user_agent=GrafanaOnCall/1.15.0 content-length=0 slow=0 user_id=1 org_id=1 org_slug=self_hosted_org
2025-02-25 07:58:22 source=engine:uwsgi status=500 method=GET path=/api/internal/v1/alert_receive_channels latency=0.246873 google_trace_id=- protocol=HTTP/1.1 resp_size=157466 req_body_size=0
2025-02-25 07:59:00 source=engine:app google_trace_id=none logger=root inbound latency=0.125394 status=403 method=GET path=/api/internal/v1/alert_receive_channels user_agent=Mozilla/5.0 (Macintosh; Intel Mac OSX 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 content-length=0 slow=0
2025-02-25 07:59:00 Forbidden: /api/internal/v1/alert_receive_channels
2025-02-25 07:59:00 source=engine:app google_trace_id=none logger=django.request Forbidden: /api/internal/v1/alert_receive_channels

2. I created an API token using the Grafana OnCall plugin installed in Grafana, but it seems to be invalid because when I'm trying to access the endpoint GET /api/v1/integrations I receive:

{
  "detail": "Invalid token."
}

The token is generated with two // but when I try to use it with one / I get a timeout

How do we reproduce it?

1.	Install and configure Grafana OnCall using Docker Compose.
2.	Install the Grafana OnCall plugin in Grafana.
3.	Generate an API token from the OnCall plugin in Grafana.
4.	Try to create a webhook integration.
5.	Check logs in app-engine-1 to see error messages related to API authentication and request handling.

Grafana OnCall Version

latest 1.15.0

Product Area

API

Grafana OnCall Platform?

Docker

User's Browser?

No response

Anything else to add?

1.	How should I properly configure webhooks in Grafana OnCall to work with Mattermost?
2.	What are the correct API endpoints for Grafana OnCall? Is api/v1/integrations valid?
3.	What could be causing the TypeError related to pagination? Could it be a Django/DRF version issue?
4.	How should authentication be handled for Grafana OnCall’s API? Is the token generated via the Grafana Oncall plugin valid?