Dynamic pipelines - a new foreach block

[ptodev]

PR Description

This is a proposal for starting up Alloy pipelines dynamically based on data in transit. For example, if a discovery component comes up with 10 targets, Alloy can start 10 sub-pipelines, each dealing with a different target.

The PR consists of a a design doc in docs/design/ and an experimental implementation. The user will have to have the "experimental" cmd flag turned on.

Which issue(s) this PR fixes

Fixes #1443

[captncraig]

I really like the idea of it being component based. I am comparing to terraform's for_each, where you make a "template" resource and have an argument in there which tells it how to expand it into multiple components:

resource "azurerm_resource_group" "rg" {
  for_each = tomap({
    a_group       = "eastus"
    another_group = "westus2"
  })
  name     = each.key
  location = each.value
}

• for_each can be any expression that evaluates to something iterable (in our case limiting to an array would probably be fine, but maybe we could do maps in a similar way).
• The each keyword is valid in other arguments to reference the "current" item for iteration.

I really like the notion that this is a single component that would be "expanded" based on the evaluation of some list. It could be a built-in component or a previously imported dynamic component.

I am not sure how naming of the expanded components would work, but we could figure something out. I'm also not sure if it is possible to reference a dynamically created subcomponent from elsewhere in the config.

[github-actions]

This PR has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If you do not have enough time to follow up on this PR or you think it's no longer relevant, consider closing it.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your PR will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[ptodev]

It looks likely that we are going to go with the foreach option. I updated the PR to add some draft code that I've been working on. As a first step, it'd be good to have a unit tests that doesn't use the proposed var argument. The test could look something like what's proposed in internal/runtime/testdata/foreach/foreach_1.txtar.

As a second step, we will need to figure out how to refer to the values that are currently iterated on. Should we actually have a var argument? I think it might be unnecessary - having a predefined argument (e.g. val) which refers to the current thing being iterated on may be enough. I'm not yet sure how to add this reserved word into the Alloy syntax though.

By the way, apparently the Collector already has something like dynamic pipelines. There are observer extensions which you can hook up to a recover creator component. For example, you can have a k8s observer which crates a receiver for each discovered pod.

[github-actions]

💻 Deploy preview available: https://deploy-preview-alloy-1480-zb444pucvq-vp.a.run.app/docs/alloy/latest/

[mattdurham]

Fantastic work, gave it a first review and will go over it again. Added some comments on tests.

[mattdurham]

Why are we using two different hash functions? sha256 and fnv, though they dont collide in usage, it feels odd.

[wildum]

The runner pkg only needs a 64 bits hash but for the objects in the collection we use a 256 bits hash. I would be ok to use 64 bits for the items in the collection. A collision at this level could end up in missing metrics and duplicated metrics in unlucky scenarios but with 1000 items in the collection (which would be a lot for Alloy) the probability is 2.71×10-14. A collision in the runner pkg would be even worse so I'm not sure that the extra 256 hash security is needed but I also don't mind it so much because it should not matter much in terms of performance. @ptodev what do you think?

[ptodev]

IIUC, for items in the collection the hash is 256 bits since there could be lots of things in each item in a collection, and it gives us extra protection against collisions. For the foreach ID we use a 64 bit hash since it's just a string. IMO it's ok to use different hashes, but we do need to document why each situation uses a different one.

[wildum]

we use the hash of the items in the collection for the foreachID and the foreachID as a key: https://github.com/grafana/alloy/pull/1480/files#diff-9cebda46e5a40368c4b76027ff2bda36114d2737124b5390aa35fa6435d79127R193
The 64 bits hash is used for the runner that wraps around the foreach to run it.
What's in an item in a collection should not matter. Whether the object has 3 or 30 fields, it should not have any influence on the collision I believe. The only difference would be the number of items in the collection and we would need billions of items for a collision to be likely.
I'm ok to keep it as it is and add a comment that we keep the 256 bits hash for extra security and that if one day it becomes a performance bottleneck it would be ok to use a 64 bits hash instead

[mattdurham]

Small comments but overall fantastic!