Support Basic Auth for datasource which used BasicAuth for alerts

[negbie]

With Grafana 7.1.1 and latest clickhouse-grafana plugin version 2.0.5 every Alert throws following error:

error:"tsdb.HandleRequest() error rpc error: code = Unknown desc = invalid status code. status: 500 Internal Server Error"

For graphs the datasourse is working normally.

[negbie]

Im using Docker with a mounted plugin directory.

[Slach]

could you share your table schema
and grafana dashboard JSON which contains alerts?

also, could you share results for following command docker log <your_grafana_container_id>?

[negbie]

CREATE TABLE cdrmon.call_log
(
    `CallTime` DateTime,
	`FromUser` String,
	`ToUser` String,
    `Prefix` String,
	`IncIP` String,
	`OutIP` String,
	`StatusCode` FixedString(3),
	`CallDuration` UInt32,
	`SpeakDuration` UInt32,
	`Domain` String,
	`Direction` FixedString(8),
    `Switch` FixedString(4),
    `UserAgent` String,
    `CID` String,
	`Custom1` UInt32,
	`Custom2` UInt32,
	`Custom3` UInt32,
	`Custom4` UInt32,
	`Custom5` UInt32,
	`Custom6` UInt32,
	`Custom7` String,
	`Custom8` String,
	`Custom9` String,
	`Custom10` String,
	`Custom11` String,
	`Custom12` String
)
ENGINE = MergeTree()
PARTITION BY (toRelativeHourNum(CallTime))
ORDER BY (CallTime, IncIP, OutIP, StatusCode, Prefix, FromUser, ToUser)
SETTINGS index_granularity = 8192

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 42,
  "links": [],
  "panels": [
    {
      "alert": {
        "alertRuleTags": {},
        "conditions": [
          {
            "evaluator": {
              "params": [
                4000
              ],
              "type": "gt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A",
                "5m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "5m",
        "frequency": "1m",
        "handler": 1,
        "name": "Panel Title alert",
        "noDataState": "no_data",
        "notifications": []
      },
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "ClickHouse",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "percentage": false,
      "pluginVersion": "7.1.1",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "database": "cdrmon",
          "dateColDataType": "",
          "dateLoading": false,
          "dateTimeColDataType": "CallTime",
          "dateTimeType": "DATETIME",
          "datetimeLoading": false,
          "extrapolate": true,
          "format": "time_series",
          "formattedQuery": "SELECT $timeSeries as t, count() FROM $table WHERE $timeFilter GROUP BY t ORDER BY t",
          "intervalFactor": 1,
          "query": "SELECT\n    $timeSeries as t,\n    count()\nFROM $table\nWHERE $timeFilter\nGROUP BY t\nORDER BY t",
          "rawQuery": "SELECT     (intDiv(toUInt32(CallTime), 20) * 20) * 1000 as t,     count() FROM cdrmon.call_log WHERE CallTime >= toDateTime(1596522809) GROUP BY t ORDER BY t",
          "refId": "A",
          "round": "0s",
          "table": "call_log",
          "tableLoading": false
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "fill": true,
          "line": true,
          "op": "gt",
          "value": 4000,
          "yaxis": "left"
        }
      ],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Panel Title",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "schemaVersion": 26,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ]
  },
  "timezone": "",
  "title": "test",
  "uid": "nRtLTe4Mk",
  "version": 2
}

error: code = Unknown desc = invalid status code. status: 500 Internal Server Error" changing state to=alerting
t=2020-08-04T12:35:01+0000 lvl=info msg="New state change" logger=alerting.resultHandler ruleId=9 newState=pending prev state=unknown
t=2020-08-04T12:36:01+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=9 name="Panel Title alert" error="tsdb.HandleRequest() error rpc error: code = Unknown desc = invalid status code. status: 500 Internal Server Error" changing state to=alerting
t=2020-08-04T12:37:01+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalContext ruleId=9 name="Panel Title alert" error="tsdb.HandleRequest() error rpc error: code = Unknown desc = invalid status code. status: 500 Internal Server Error" changing state to=alerting

[negbie]

Don't know if it helps but datasource is using basic auth.

[negbie]

Not sure if this is a heisenbug but I tried it again and this time it worked. Need to investigate it.

[negbie]

Something else I've found along the way. If you select by
toStartOfHour(CallTime) AS t

you will get following error:
error:"tsdb.HandleRequest() error rpc error: code = Unknown desc = unable to parse timestamp with alias=t value=2020-08-04 16:00:00 error=strconv.ParseInt: parsing "2020-08-04 16:00:00": invalid syntax"

[Slach]

toStartOfHour it's expected behavior
please create a separate issue
and use toUInt64(toStartOfHour(...)) AS t as quick workaround

[negbie]

Ah ok. Thanks I will. Currently I use (intDiv(toUInt32(CallTime), 3600) * 3600) * 1000 as t.

[Slach]

maybe "heisenbug" depends of data which returned from your datasource

could you check system.query_log in your clickhouse database and try to find queries related to your alerts

[negbie]

Ok, thanks for the hint, I will see if I can find something useful.

[negbie]

Ok we are lucky and it's not a heisenbug. I can reproduce it. It works now because in the datasource settings I turned the switch on "Use Yandex.Cloud authorization headers" and filled the credentials. Even if you turn of the switch again it will still work.

Looking into the code I found this:
https://github.com/Vertamedia/clickhouse-grafana/blob/master/datasource.go#L104

[negbie]

If you delete the credentials for Yandex.Cloud authorization headers you will see the mentioned error again.

[Slach]

@negbie thanks a lot for investigating,
I think we need to switch to new plugin SDK
@bmanth60 any news about progress with it?

[via-justa]

I'm facing the same issue with self hosted clickhouse and basic auth, any news or workaround on the subject?

[Slach]

@via-justa look to #275
if you able to code on golang, could you help us to switch to new grafana SDK?

[xgxshtc]

Ok we are lucky and it's not a heisenbug. I can reproduce it. It works now because in the datasource settings I turned the switch on "Use Yandex.Cloud authorization headers" and filled the credentials. Even if you turn of the switch again it will still work.

Looking into the code I found this:
https://github.com/Vertamedia/clickhouse-grafana/blob/master/datasource.go#L104

I 've met the same problem. But what is this "Use Yandex.Cloud authorization headers" used for? It seems like account of Yandex.Cloud, why does it can solve this problem? In my test environment, I didn't turn this on, but the Alert is OK.

[Slach]

@xgxshtc it means use authorization method via X-ClickHouse-User and X-ClickHouse-Key headers instead of basic auth
so, just turn on this option in connection properties and use your user \ password credentials

please read https://clickhouse.tech/docs/en/interfaces/http/ for details

[xgxshtc]

@xgxshtc it means use authorization method via and headers instead of basic auth
so, just turn on this option in connection properties and use your user \ password credentialsX-ClickHouse-User``X-ClickHouse-Key

please read https://clickhouse.tech/docs/en/interfaces/http/ for details

Thks a lot, it really helped. Now it works.

[glebsam]

So, my story is:

1. Setup Basic Auth in ClickHouse datasource
2. Build a dashboard, see graphs
3. Create alert, click "Test rule" button, get Error: tsdb.HandleRequest() error rpc error: code = Unknown desc = invalid status code. status: 401 Unauthorized
4. Enable "Yandex.Cloud authorization headers" in datasource settings, fill with basic auth credentials
5. Click "Test rule" button, test is ok
6. Refresh page, see graphs are broken (auth issue, but different)
7. Disable "Use Yandex.Cloud authorization headers" in datasource settings
8. Graphs work, alert works 🎉

[Slach]

@glebsam thanks a lot for workaround ;)

[bmanth60]

This issue should be resolved by #314

[negbie]

@bmanth60 thank you very much for your great work!