feature request: dynamic stacks discovery / wildcard paths

[sanzoghenzo]

To avoid having to fiddle with SwarmCD configuration every time a new stack needs to be added, I wish to use something similar (but much simpler/simplistic) to what Argo CD provides via ApplicationSet.

My use case would be to have a single repository with multiple folders, one for each stack, and let SwarmCD detect new folders that matches a pattern, for example stacks/**/docker-compose.yaml or a more regex-que stacks/.*/docker-compose.yaml.

Another approach could be a to mimic something like Argo CD "Application of Applications" pattern: be able to specify a stacks_file, instead of compose_file, inside the stacks config, to point to another stacks configuration file that would then be merged (recursively).

[m-adawi]

Yes that's a nice feature to have and I was planning to do something similar soon

[sanzoghenzo]

Hi there, I'm in the process of designing a solution for this issue, at least the wildcard-based part.

This is what I have in mind

• let the user use wildcards compatible with path/filepath.Glob in the compose_file field (I see that Glob doesn't support the double **, so we could borrow this workaround)
• for each result of the glob search, create a swarmStack object with the name of stack declared in the stacks.yaml as base and the glob match (in the case above, the subfolder of the stacks folder) as suffix for the stack name - and obviously the matched path as compose_file
• business as usual

There are some changes to be done in the architecture, though:

• initStacks should be moved inside the Run loop to constantly monitor the repositories and update the stacks list.
• since we need access to the repo during the initStacks, we should ensure it to be already cloned, updated and checked-out on the right branch before doing the glob
• speaking of branches, I propose to group the stacks (not only the dynamic ones, but all of them) by repository and branch to improve the I/O operations a bit and checkout+pull only once in a update cycle
• at this point I'm wondering if the stack discovery should call the updateStack directly instead of storing the swarmStack list, so that the repository is already in the desired state to read the compose file.
• bonus: if we store the list of stack names from the previous cycle, be it in-memory or in any kind of storage (json file, redis, sqlite), we could find out which stacks have been removed from git and issue a docker stack rm on them)

Any thoughts?

[m-adawi]

I like your approach.

speaking of branches, I propose to group the stacks (not only the dynamic ones, but all of them) by repository and branch to improve the I/O operations a bit and checkout+pull only once in a update cycle

Yeah this would actually be much more efficient. We can also improve threading and have more stacks updated in parallel cuz currently stacks of the same repo are locked using a mutex to prevent git inconsistencies. But if we do this we might need to checkout different branches of the same repo in different directories, maybe we can reuse shallow clone in this case to reduce space, we won't have to worry about #33 because each git directory will be used for a single branch.

at this point I'm wondering if the stack discovery should call the updateStack directly instead of storing the swarmStack list, so that the repository is already in the desired state to read the compose file.

This would work but I think we better keep a list of stacks object anyway. The UI for example needs an object that contains the state of all stacks, so to keep this object updated we might need the list. We might also add more features in the future that could make use of this list

bonus: if we store the list of stack names from the previous cycle, be it in-memory or in any kind of storage (json file, redis, sqlite), we could find out which stacks have been removed from git and issue a docker stack rm on them)

This is similar to application pruning in ArgoCD. It can be useful and dangerous at the same time so maybe we can make it an optional behaviour. ArgoCD stores its applications as kubernetes objects so it doesn't need a special storage, so it might be more complicated in our case, especially if we need the state of stacks to be persistent if something happens to SwarmCD. I think we should have this in a separate issue/PR.

[kevinholtkamp]

Just to add my two cents, not sure if I understood everything correctly, maybe you already mentioned this:

The naming for the compose files should not be fixed. For example, I have a folder Compose where I have all my stack files, adguard_home.yaml, grafana.yaml etc.

[sanzoghenzo]

Hi @kevinholtkamp,
great point, indeed this feature should be as versatile as possible.
I didn't clearly state it, but I took it for granted; in fact, the Glob function I was planning to use can cover your use case without any problem.

Not much time to work on it right now, though 😅

[kevinholtkamp]

@sanzoghenzo thanks for your reply! I would love to help, but unfortunately I am not familiar with Go :(

If there is any way I could help (e.g. Testing) let me know though!