Add influx push endpoint to mimir #10153

alexgreenbank · 2024-12-06T09:46:14Z

What this PR does

Add influx endpoint to Mimir.

Based on original code from #1971 but brought up to date for modern Distributor framework.

MVP. Experimantal/hidden feature so no documentation yet.

Checklist

Tests updated.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
about-versioning.md updated with experimental features.
Documentation.

Signed-off-by: alexgreenbank <[email protected]>

CLAassistant · 2024-12-06T09:46:21Z

All committers have signed the CLA.

Signed-off-by: alexgreenbank <[email protected]>

colega · 2024-12-17T12:55:39Z

pkg/api/api.go

+	// TODO(alexg): hidden behind a featureflag or experimental config option?
+	a.RegisterRoute(InfluxPushEndpoint, distributor.InfluxHandler(


I don't think a feature flag for this is needed. We can just state in the docs (about-versioning.md) that the endpoint is experimental.

colega · 2024-12-17T12:58:25Z

pkg/distributor/influx.go

+	level.Debug(spanLogger).Log(
+		"msg", "decodeAndConvert complete",
+		"bytesRead", bytesRead,
+	)


Opinionated style nit: I don't think we need 4 lines for this debug log.

Fixed in next push.

colega · 2024-12-17T13:00:14Z

pkg/distributor/influx.go

+		"bytesRead", bytesRead,
+	)
+	if err != nil {
+		level.Error(logger).Log("err", err.Error())


Does this logger have all the required context? Can you also add some context here about what was going on when this happened? I'm scared of finding this log:

ts=2024-12-17 err="unexpected EOF"

Also, nit, .Error() call is not needed, just pass err.

Agreed, fixed in next push.

colega · 2024-12-17T13:02:11Z

pkg/distributor/influx.go

+	"github.com/grafana/dskit/grpcutil"
+	"github.com/grafana/dskit/httpgrpc"
+	"github.com/grafana/dskit/middleware"
+	io2 "github.com/influxdata/influxdb/v2/kit/io"


Suggested change

io2 "github.com/influxdata/influxdb/v2/kit/io"

influxio "github.com/influxdata/influxdb/v2/kit/io"

Agreed, fixed in next push.

colega · 2024-12-17T13:03:31Z

pkg/distributor/influx.go

+	"github.com/grafana/mimir/pkg/util/spanlogger"
+)
+
+func parser(ctx context.Context, r *http.Request, maxSize int, _ *util.RequestBuffers, req *mimirpb.PreallocWriteRequest, logger log.Logger) error {


This function lives in pkg/distributor, I would say it's a little bit pretentious to take the name parser for this :D

How about influxRequestParser?

Agreed, fixed in next push.

colega · 2024-12-17T13:04:45Z

pkg/distributor/influx.go

+				// TODO(alexg): Do we even need httpgrpc here?
+				// Check for httpgrpc error, default to client error if parsing failed


I don't see any httpgrpc errors being returned by parser

Oops, should have got rid of that one. Will resolve once I work out best way to wrap existing error in the StatusBadRequest.

Ah, httpgprc is used to smuggle both the http status code and the error message out of the supplier() function. I've removed the misleading comment.

colega · 2024-12-17T13:23:22Z

pkg/distributor/influxpush/parser.go

+			(charInt >= 48 && charInt <= 57) || // 0-9
+			charInt == 95) { // _
+
+			*in = (*in)[:charIndex] + "_" + (*in)[charIndex+1:]


This allocates a new string (hopefully just one, compiler should be clever) for each one of the invalid characters. How about you modify the bytes slice before transforming it to a string?

Agreed. Done in latest commit.

pkg/distributor/influxpush/parser.go

colega · 2024-12-17T13:25:22Z

pkg/distributor/influxpush/parser.go

+func replaceInvalidChars(in *string) {
+	for charIndex, char := range *in {
+		charInt := int(char)
+		if !((charInt >= 97 && charInt <= 122) || // a-z


If you use 'a' instead of 97 (is it correct?) you won't need the comment and the code will be less prone to have bugs.

Moved to literal chars instead of ASCII codes.

pkg/distributor/influxpush/parser.go

colega · 2024-12-17T13:31:50Z

pkg/distributor/influxpush/parser.go

+			key := string(tag.Key)
+			if key == "__name__" || key == internalLabel {
+				continue
+			}
+			replaceInvalidChars(&key)
+			lbls = append(lbls, mimirpb.LabelAdapter{
+				Name:  key,
+				Value: string(tag.Value),


We can do yoloString here instead of string(), as the push code is crafted to avoid keeping references to the strings from LabelAdapter

Interesting! Fixed as per suggestion. I take it no other strings are guaranteed not to be referenced and so it can't be used anywhere else?

Signed-off-by: alexgreenbank <[email protected]>

alexgreenbank · 2024-12-19T13:17:04Z

OK, comments left to deal with

end2end test for happy path
TryUnwrap
work on byteslice rather than chars
yoloString

Signed-off-by: alexgreenbank <[email protected]>

…into alexg/influx-push-handler

tacole02

Looks good!

pkg/distributor/influxpush/parser.go

Co-authored-by: Oleg Zaytsev <[email protected]>

Signed-off-by: alexgreenbank <[email protected]>

colega · 2025-01-16T16:39:01Z

pkg/distributor/influx.go

+				// TODO(alexg): One thing we have seen in the past is that telegraf clients send a batch of data
+				// if it is too big they should respond to the 413 below, but if a client doesn't understand this
+				// it just sends the next batch that is even bigger. In the past this has had to be dealt with by
+				// adding rate limits to drop the payloads.
+				level.Warn(logger).Log("msg", "request too large", "err", err, "bytesRead", bytesRead, "maxMsgSize", maxRecvMsgSize)


Is this really a TODO? What is pending to be done here?

Removed the TODO. This is more an operational issue, something that I plan on taking a look at during the dev/ops testing and will add it to the epic.

When the telegraf sends a payload that is too big it just tries it again a bit later with even more data, and so the problem never seems to fix itself.

We need to documented method/runbook for dealing with this situation (which is easy to trigger given we can control the maximum payload size we accept) otherwise it never resolves itself. We need to test how the telegraf agent responds to different return codes, or come up with another plan.

For now the code does the right thing and logs the error and rejects the payload.

Removed the TODO and comment for now.

colega · 2025-01-16T16:40:33Z

CHANGELOG.md

@@ -2,6 +2,8 @@

 ## main / unreleased

+* [FEATURE] Distributor: Add experimental Influx handler. #10153


Please move this to the Grafana Mimir section below.

Signed-off-by: alexgreenbank <[email protected]>

alexgreenbank added 2 commits December 6, 2024 09:14

olegs base commits from #1971

444d34f

Signed-off-by: alexgreenbank <[email protected]>

move top level influx files

adb376a

Signed-off-by: alexgreenbank <[email protected]>

alexgreenbank added 11 commits December 9, 2024 22:08

latest wip

3db179f

Signed-off-by: alexgreenbank <[email protected]>

still WIP but better, still need to move to parserFunc() style

295fa8d

Signed-off-by: alexgreenbank <[email protected]>

it builds!

d2897e3

Signed-off-by: alexgreenbank <[email protected]>

tweaks and add span logging

c2fb679

Signed-off-by: alexgreenbank <[email protected]>

more todo

593b2e2

Signed-off-by: alexgreenbank <[email protected]>

further tweaks

31bf23d

Signed-off-by: alexgreenbank <[email protected]>

some fixes to tests

2487a88

Signed-off-by: alexgreenbank <[email protected]>

rejigged error handling, tests passing

03e8e3c

Signed-off-by: alexgreenbank <[email protected]>

add vendored influxdb code

0bd8da8

Signed-off-by: alexgreenbank <[email protected]>

lint

4a76a11

Signed-off-by: alexgreenbank <[email protected]>

go mod sum vendor/modules.txt

066c009

Signed-off-by: alexgreenbank <[email protected]>

alexgreenbank requested a review from colega December 16, 2024 15:17

colega reviewed Dec 17, 2024

View reviewed changes

alexgreenbank added 12 commits December 17, 2024 15:21

add a metric, add tenant info, other tweaks

bec3a26

Signed-off-by: alexgreenbank <[email protected]>

various rework, still WIP

ac51def

Signed-off-by: alexgreenbank <[email protected]>

propagate bytesRead down to caller and log and histogram

3a57dc6

Signed-off-by: alexgreenbank <[email protected]>

remove comment now dealt with

92379e4

Signed-off-by: alexgreenbank <[email protected]>

add defaults in error handling

d44c71d

Signed-off-by: alexgreenbank <[email protected]>

Add note to docs about experimental Influx flag

591389e

Signed-off-by: alexgreenbank <[email protected]>

Note influx endpoint as experimental too

afbc357

Signed-off-by: alexgreenbank <[email protected]>

test for specific errors received

847bcb9

Signed-off-by: alexgreenbank <[email protected]>

bolster parser tests

320c467

Signed-off-by: alexgreenbank <[email protected]>

Use literal chars rather than ascii codes

730a7c3

Signed-off-by: alexgreenbank <[email protected]>

remove unnecessary cast to int()

de27d4b

Signed-off-by: alexgreenbank <[email protected]>

use mimirpb.PreallocTimeseries in influx parser

af3def1

Signed-off-by: alexgreenbank <[email protected]>

alexgreenbank marked this pull request as ready for review January 10, 2025 23:03

alexgreenbank requested review from stevesg, grafanabot, a team and tacole02 as code owners January 10, 2025 23:03

alexgreenbank added 5 commits January 10, 2025 23:07

CHANGELOG

537fa37

Signed-off-by: alexgreenbank <[email protected]>

about-versioning.md

14bae20

Signed-off-by: alexgreenbank <[email protected]>

Merge branch 'main' into alexg/influx-push-handler

b951127

merge main

9d035f5

Signed-off-by: alexgreenbank <[email protected]>

Merge branch 'alexg/influx-push-handler' of github.com:grafana/mimir …

c31c191

…into alexg/influx-push-handler

tacole02 approved these changes Jan 13, 2025

View reviewed changes

colega reviewed Jan 13, 2025

View reviewed changes

pkg/distributor/influxpush/parser.go Outdated Show resolved Hide resolved

colega reviewed Jan 13, 2025

View reviewed changes

pkg/distributor/influxpush/parser.go Outdated Show resolved Hide resolved

colega reviewed Jan 13, 2025

View reviewed changes

pkg/distributor/influxpush/parser.go Outdated Show resolved Hide resolved

alexgreenbank and others added 5 commits January 14, 2025 15:05

Update pkg/distributor/influxpush/parser.go

0872e6a

Co-authored-by: Oleg Zaytsev <[email protected]>

Update pkg/distributor/influxpush/parser.go

de92ac5

Co-authored-by: Oleg Zaytsev <[email protected]>

Update pkg/distributor/influxpush/parser.go

0e6cea6

Co-authored-by: Oleg Zaytsev <[email protected]>

fix parsing string replacing code

8afdd9b

Signed-off-by: alexgreenbank <[email protected]>

fix merge conflicts

7018f72

Signed-off-by: alexgreenbank <[email protected]>

alexgreenbank requested a review from colega January 16, 2025 09:10

colega reviewed Jan 16, 2025

View reviewed changes

alexgreenbank added 2 commits January 17, 2025 11:07

fix nits

d81ac52

Signed-off-by: alexgreenbank <[email protected]>

Merge branch 'main' into alexg/influx-push-handler

332576e

colega approved these changes Jan 17, 2025

View reviewed changes

colega changed the title ~~add influx push endpoint to mimir~~ Add influx push endpoint to mimir Jan 17, 2025

colega merged commit 1836ff1 into main Jan 17, 2025
31 checks passed

colega deleted the alexg/influx-push-handler branch January 17, 2025 12:49

alexgreenbank mentioned this pull request Jan 23, 2025

influx: fix endpoint path to match existing proxy #10505

Merged

alexgreenbank mentioned this pull request Jan 30, 2025

influx: rename extra label to match existing series #10545

Draft

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add influx push endpoint to mimir #10153

Add influx push endpoint to mimir #10153

alexgreenbank commented Dec 6, 2024 •

edited

Loading

CLAassistant commented Dec 6, 2024 •

edited

Loading

colega Dec 17, 2024

alexgreenbank Dec 17, 2024

colega Dec 17, 2024

alexgreenbank Dec 17, 2024

colega Dec 17, 2024

alexgreenbank Dec 17, 2024

colega Dec 17, 2024

alexgreenbank Dec 17, 2024

colega Dec 17, 2024

alexgreenbank Dec 17, 2024

colega Dec 17, 2024

alexgreenbank Dec 17, 2024 •

edited

Loading

alexgreenbank Dec 17, 2024

colega Dec 17, 2024

alexgreenbank Dec 19, 2024

colega Dec 17, 2024

alexgreenbank Dec 17, 2024

colega Dec 17, 2024

alexgreenbank Dec 19, 2024

alexgreenbank commented Dec 19, 2024 •

edited

Loading

tacole02 left a comment

colega Jan 16, 2025

alexgreenbank Jan 17, 2025 •

edited

Loading

colega Jan 16, 2025

alexgreenbank Jan 17, 2025

		// TODO(alexg): hidden behind a featureflag or experimental config option?
		a.RegisterRoute(InfluxPushEndpoint, distributor.InfluxHandler(

	io2 "github.com/influxdata/influxdb/v2/kit/io"
	influxio "github.com/influxdata/influxdb/v2/kit/io"

		// TODO(alexg): Do we even need httpgrpc here?
		// Check for httpgrpc error, default to client error if parsing failed

		@@ -2,6 +2,8 @@

		## main / unreleased

		* [FEATURE] Distributor: Add experimental Influx handler. #10153

Add influx push endpoint to mimir #10153

Add influx push endpoint to mimir #10153

Conversation

alexgreenbank commented Dec 6, 2024 • edited Loading

What this PR does

Checklist

CLAassistant commented Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexgreenbank Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexgreenbank commented Dec 19, 2024 • edited Loading

tacole02 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexgreenbank Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexgreenbank commented Dec 6, 2024 •

edited

Loading

CLAassistant commented Dec 6, 2024 •

edited

Loading

alexgreenbank Dec 17, 2024 •

edited

Loading

alexgreenbank commented Dec 19, 2024 •

edited

Loading

alexgreenbank Jan 17, 2025 •

edited

Loading