Cannot fetch graph with curl: Operation timded out after 5000 milliseconds

[robertkrenn]

While updating php from 7.6 to 8.1 we had to update a lot of modules, icingadb inclusive, and in parallel we are in process of switching to a dedicated Grafana Server with Influxdb underneath. Before we were on the same machine as icinga2 itself was running, with graphite behind it.

Well, Grafana is working fine, we have imported some Icinga2 dashboards and also created our own dashboards and panels using InfluxQL. We configured the new Grafana Source in the grafana module, but on the services and hosts we only get the error shown of timed out operations. When clicking on the error we get directly to the grafana ui with the correct graphs and everything working as expected. I think it may be only a little config mistake, but i have no idea what detail is causing this. Maybe you have a hint for me.

we use access with api service tokens. On the Grafana management UI I can see, that the service token is used to get the graphs (last used)

Expected Behavior

Showing the graph on the hosts and services detail sites.

Current Behavior

Error: Cannot fetch graph with curl:
Operation Timed out after 5000
milliseconds with 0 bytes received

Your Environment

• Icinga Web 2 Grafana version (System - About): 2.0.3

• Icinga Web 2 version and modules (System - About): 2.12.1

• Version used (icinga2 --version): 2.14.0

• Icingadb used: 1.1.1

• Operating System and version: SLES 15 SP5

• Enabled features (icinga2 feature list):
  Disabled features: debuglog elasticsearch gelf influxdb journald livestatus opentsdb perfdata syslog
  Enabled features: api checker command compatlog graphite icingadb ido-mysql influxdb2 mainlog notification

• Config validation (icinga2 daemon -C):

[2023-12-29 10:53:58 +0100] information/cli: Icinga application loader (version: r2.14.0-1)
[2023-12-29 10:53:58 +0100] information/cli: Loading configuration file(s).
[2023-12-29 10:53:58 +0100] information/ConfigItem: Committing config item(s).
[2023-12-29 10:53:58 +0100] information/ApiListener: My API identity:
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 7 EventCommands.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 481 Comments.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 CompatLogger.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 31 Users.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 5 UserGroups.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 16 TimePeriods.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 13 ServiceGroups.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 439 Zones.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 13184 Services.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 6 NotificationCommands.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 56 Notifications.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 566 Hosts.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 55 HostGroups.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 436 Endpoints.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 273 Downtimes.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 12367 Dependencies.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 5 ApiUsers.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 325 CheckCommands.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 Influxdb2Writer.
[2023-12-29 10:53:59 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2023-12-29 10:53:59 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-12-29 10:53:59 +0100] information/cli: Finished validating the configuration file(s).

The modules debug output is: