Clickhouse Alert Throwing Error - request handler error: plugin unavailable

[gsingh14]

Setup
Grafana : Grafana v8.1.0-pre (unknown-dev)
Clickhouse plugin : 2.3.1

Steps

1. Using Docker to build grafana image
2. In Dockerfile add command to copy vertamedia-clickhouse-datasource folder to var/lib/grafana/plugins
3. Build Docker image
4. Start Docker Container
5. Setup email alert config
6. Create a sample dashboard connecting to clickhouse instance
7. Use the following sample query in dashboard query section

SELECT
    $timeSeries as t,
    count()
FROM $table

WHERE $timeFilter

GROUP BY t

ORDER BY t

8. Update the Alert Tab with sample values to fire an alert
   
9. Click on Test Rule and the following error pop up
   

Can someone help with this issue?

[Slach]

could you open "plugins" in UI and check clickhouse plugin is "signed"?

[gsingh14]

Hi @Slach . Thanks for your reply. The plugin is signed per the following screenshot

[Slach]

could you add to your Dockerfile ENV GF_LOG_LEVEL=debug
and share logs of your custom grafana container?

[gsingh14]

Following is the debug log when i create an alert and click on Test Rule.

t=2021-07-14T17:06:43+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2021-07-14T17:06:44+0000 lvl=dbug msg="Querying for data source via SQL store" logger=datasources id=1 orgId=1
t=2021-07-14T17:06:44+0000 lvl=dbug msg="Applying default URL parsing for this data source type" logger=datasource type=vertamedia-clickhouse-datasource url=https://CH_URL:9443
t=2021-07-14T17:06:53+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2021-07-14T17:06:55+0000 lvl=dbug msg="Extracted alerts from dashboard" logger=alerting.extractor alertCount=1
t=2021-07-14T17:06:55+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=0 name="Panel Title alert" error="request handler error: plugin unavailable" changing state to=alerting
t=2021-07-14T17:07:03+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2021-07-14T17:07:13+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0

Note : I have replaced my internal CH instance url with CH_URL here for security purposes.

[Slach]

@gsingh14 thanks a lot for reporting I will try to reproduce, and tix it ASAP

[zweite]

This problem can be solved by using a Grafana image packaged with Dockerfile.ubuntu

[gsingh14]

Thanks @zweite . I was able to get past the error but now getting @Slach

x509: certificate signed by unknown authority A <nil> []
since my Clickhouse instance is on https.

Would you know where i can install the cert to make it work. I have provided the cert file in Datasource and Grafana is able to make the connection.

Only when i trigger the alert, I'm getting the above error.

[Slach]

Unfortunately, currently it's not easy to provide custom certificate authority in SSL certs inside grafana, both on frontend and on backend side.