[inputs.statsd] Tags mangled with tags from other StatsD metrics

[gnjack]

After upgrading from Telegraf 1.16.3 to 1.17.0, we've noticed that tags from StatsD metrics are being mangled with unrelated metrics.

Relevant telegraf.conf:

This may require Graphite templates to be set on the statsd input as the thread safe issue appears to be partially in the GraphiteParser

[[inputs.statsd]]
  templates = ["example.* measurement.host"]

Steps to reproduce:

1. Configure a StatsD input
2. Send a large number of metrics with Influx StatsD style tags (issue is worse at high concurrency when reading StatsD input). For example, send these two metrics a large number of times:

test,a=b:1|c
foobar,foo=bar:1|c

Expected behavior:

Something similar to these two metrics are written to the output:

test,a=b count=100
foobar,foo=bar count=100

Actual behavior:

Something similar to these metrics are written to the output, where sometimes the tags are swapped:

test,a=b count=93
test,foo=bar count=7
foobar,foo=bar count=89
foobar,a=b count=11

Additional info:

Appears to be caused by a locking change in #8509, as I attempted to explain here #8509 (comment)

[ivorybilled]

I was able to replicate this, it's definitely caused by the cause you identified and take a look at the above PR if you'd like. Thank you!