Characters supported in tag values

[qqbb18]

We would like to gain more insight into what printable ASCII characters (32-126) are supported in tag values on Grafana/MetricTank.
Currently, our system only directly supports a limited character set: a-zA-Z0-9-_./%, and blocks other characters (or only allows other special characters via encoding them into HEX format).
 However, the tag values in the timeseries data into our system might contain substantially more characters than this list. 
Therefore we plan to support additional ASCII characters in tag values, given that they will not cause issues to any existing Grafana/MetricTank functionality, such as query, computation, etc. 
We have conducted some basic tests on MetricTank and Grafana by putting each character to different positions of a tag value: leading, trailing and somewhere in the middle. 
After combining the test results with other limitations, we have an initial proposal for the characters supported by our system, which is listed below:



0 - 9 allow everywhere in a tag value

a - z allow everywhere in a tag value

A - Z allow everywhere in a tag value




! # % & - / : < = > @ _ . allow everywhere in a tag value


(space) ~ allow only in the middle of a tag value

; Always block in a tag value because this is not supported by MetricTank

' Always block in a tag value because this is causes request error (unexpected character) when used on Grafana

" $ ( ) * + , ? [ ] \ ^ ` { } | Always block in a tag value because they might cause ambiguity when used in regex queries



Please kindly provide suggestions/comments if you feel any of this is not accurate. Moreover, in order to better understand the restrictions on tag values, we would like to confirm with you on the officially supported list of characters. If possible, please provide brief explanations on why a character is not supported. Finally, it is probably worth adding a regression test for this so we can confidently extend the character set that we plan to support in the future. Thanks!

This issue has also been posted to Grafana:
grafana/grafana#15261

[replay]

Generally MT is really permissive regarding tag values. The only character that's not allowed is ;. I wrote a unit test to show that and made sure to include all the special characters you mentioned. In some cases it can be a little tricky to figure out how to query something like \ with a regular expression though. Unfortunately, it seems like GitHub is confused and doesn't show all the characters correctly, but if you do git fetch then it gets copied correctly:

565c5d2f

[Dieterbe]

we should probably align with Graphite on this...

[replay]

I forgot to mention that actually a tag value could start with ~, but then it's going to be hard to query it because =~ is the regular expression operator

[replay]

Graphite is similarly permissive, I'm not sure how well this is tested though: https://github.com/graphite-project/graphite-web/blob/master/webapp/graphite/tags/utils.py#L52

[qqbb18]

@replay This was exactly what I have found from my tests: the only character not accepted by MT is ;. Thanks for adding a unit test to include all the characters mentioned above! I totally agree that it is tricky to figure out how to test with regex queries.

However I found there are cases some other characters that do not work well with Grafana, for example, ', this always gives request error "unexpected character", but this might be a Grafana/Graphite issue ...

[replay]

The issue with ' in the middle of the value is probably related to how Grafana submits the query to Graphite. May I ask why it is necessary for you to use such characters in tag values?

[shalstea]

We aren't asking for it per se. We are suggesting that you document and test what you support/don't support. We would love to have all the characters supported but realize that, that isn't feasible.

For us we are sourcing tag values from systems we don't control (hardware device info, etc). They can have any character, if we don't support them directly then we have to encode the values in some way. We provide teams with a way to do the encoding but it requires a character by character copy, examination. Right now we have nothing officially documented / supported from MetricTank/Grafana and this is to help close the gap.. and then we can work around the corner cases on our side.

[replay]

As an update to the Graphite (and MT is Graphite compatible) documentation, we got the character support documented in the official docs now: graphite-project/graphite-web#2426

[Dieterbe]

@replay status of this?

[replay]

On Metrictank side i think this can be considered done with the merging of the following PRs:
#1371
#1348
raintank/schema#28

In Graphite & Carbon there are still related changes which have not been merged yet, but that doesn't really concern this issue:
graphite-project/graphite-web#2458
graphite-project/carbon#858

[robert-milan]

On Metrictank side i think this can be considered done with the merging of the following PRs:
#1371
#1348
raintank/schema#28

^^ Those are all merged

In Graphite & Carbon there are still related changes which have not been merged yet, but that doesn't really concern this issue:
graphite-project/graphite-web#2458
graphite-project/carbon#858

There are ongoing discussions and work related to the graphite-project changes.

[shalstea]

Can this be closed ?

[replay]

I'm waiting for the two PRs in the graphite-project to be closed. let me ping the involved people about those