Support --detach flag in docker stack deploy / rm

[dedalusj]

The docker stack deploy command should support a --detach flag much like the docker service create and docker service update commands.

[dnephin]

When --detach=false is used, what should we show as output? I assume progress bars, should we have a progress bar per task (like we have with service commands), or should it be a progress bar per service?

cc @thaJeztah @vdemeester

[thaJeztah]

Perhaps the same (set of) progress bars as for service create/update - grouped per service (although I can see that becoming too verbose with large stacks containing lots of services

Should we have someone from the UX / design team craft a design?

[dnephin]

That might be overkill. We need to keep it consistent with the service commands, so I think there are only really two options. Either progress of tasks (grouped if possible as you've suggested) or progress of services.

[dedalusj]

Progress of services seems a good choice to me.

When I'm deploying a stack of a few services the number of underlying tasks can be large and progress can be overwhelming. I would quickly tune out of it.

[thaJeztah]

IIRC, the progress bars for services already have a threshold; if more than X tasks for a service, then a single progress bar is shown, instead of per task (I should check the code)

[akalipetis]

I'd say that anything returning 0 if all the updates are successful and something else otherwise (like the total number of failed services?) should be more than enough as a start.

Progress bars are definitely a nice addition, but it's more important IMO to be able to know if your deployment succeeded or not.

Currently, you cannot reliably deploy using compose file from a CI service.

Given that waitOnService is already implemented, we could simply use the exact same logic for stacks and revisit if needed. Otherwise I believe that just waiting and giving an exit code is more than enough.

[dnephin]

I just noticed the origin issue (moby/moby#32367) suggested the progress bars would be one per service, instead of per task.

[sirlatrom]

@dnephin It is my impression that that is how docker service scale --detach works too. Am I right?

[dnephin]

No, I'm pretty sure all the docker service commands all show progress bars for each task, not for each service.

[sirlatrom]

@dnephin You are correct, of course. I suppose any command that is supposed to show the status of multiple services interactively should provide a similar user experience? Which would that be, one progress bar per task, or one per service?

[dustin-decker]

detach=false on a service changes behavior when you exceed the maxProgressBars constant for both replicated and global services:
d8ab384#diff-5bd6b950bc904a376a90157b719668a0R255

Based on that, a similar approach should probably be used for stack deploy, showing tasks if tasks<maxProgressBars, and services if it exceeds that constant.

[shouze]

I'm very interested about that one. I don't know about you guys but typically we use to run our e2e tests on a true docker (mono host) swarm... so very handy to easily know when we can launch our e2e test suite against the deployed infra.

I'm ok to work on this one but 1st I have to understand well on what I can rely. I've listed the current/desired states returned by docker stack ps:

• Current state:
  • pending
  • starting
  • complete
  • running
  • failed
• Desired state:
  • running
  • shutdown
  • accepted

Also the Error field can be used I guess in case of desired state shutdown?

Note: Since few weeks we also have some services deployed with a restart policy condition to none, this is a good illustration of a short lived / run once service I guess that the --detach=false should support too and should be illustrated in tests.

[shouze]

Ok, after some further investigation in fact it's pretty straight to the point as docker stack deploy implementation call under the hood some service update/create that already support --detach=false option.

[shouze]

I have created this project until this feature is available in docker cli: https://github.com/karibbu/docker-php

[nullobject]

Until this feature is implemented, how do I tell if a stack was updated successfully?

[mattmattmatt]

Any update on this, or on nullobject's question above?
This tool tries to help with that but native support is certainly preferred.

[AdrianSkierniewski]

Is there any news when this could be implemented? I'd like to use docker stack deploy together with ansible but don't know how to monitor if that deployment went successful.

[ushuz]

@AdrianSkierniewski

For deploying stack with Ansible, I modified docker_stack module (ansible/ansible#24588) to deploy stacks and mimic docker service's behaviour to wait for services to be converged.

You can find the modified module in this gist, hope it's useful.

[AdrianSkierniewski]

@ushuz I didn't know about that module. Thank you!

[yangm97]

Updates?

[aselvan]

Is there a target release date for this enhancement?

[thaJeztah]

I don't think someone is working on this currently, but contributions welcome if someone wants to work on this

[drozzy]

Without this feature, it is not clear whether we should proceed to the next step in CI/CD or not.

Also, it is not clear, what happens if we run docker stack deploy and then exit the shell session that executed that command. Is the deploy only partially done? Or completely done?

[sorenhansendk]

@drozzy: We have the same problem, but we solved it with few lines of Powershell.

Run this right after your docker stack deploy command:

Do
{
    $Services = $(docker stack services [YOUR-STACK-NAME] -q);
    $Inspects = (docker service inspect $Services) | ConvertFrom-Json

    $Updating = @();
    $Completed = @();
    $Rolledback = @();

    ForEach ($Service in $Inspects | Where-Object { $_.UpdateStatus -ne $null })
    {
        if ($Service.UpdateStatus.State -eq 'completed') {
            $Completed += $Service
        } elseif ($Service.UpdateStatus.State -like 'rollback*') {
            $Rolledback += $Service
        }

        $Updating += $Service;
    }

    "Waiting for {0} service(s) to update ({1} completed / {2} rollback)" -f $Updating.Count, $Completed.Count, $Rolledback.Count
    Start-Sleep 5

} Until ($Updating.Count -eq ($Completed.Count + $Rolledback.Count))

[drozzy]

Thanks, but my understanding was that swarm accepts the command right away and you’re free to logout of the shell?

Are you saying you need to hang around until it’s all done? What if there is an error and rollback or failure with an unhealthy service?

[kinghuang]

@drozzy Once the deploy command returns, there's no need to wait around. You can freely exit the shell.

That doesn't mean that the deployment is done, in the sense that all services are running. But, the CLI's job is done, and it's up to the swarm to reach the desired state.

[sorenhansendk]

@drozzy: Yes. We wait on all our updates is completely done - so we can warn the developers about a rollback action, if anything has failed in the deployment. We have some extra code to mark a deployment as failed. But I don't copied that, into the code snippet above :-)

[drozzy]

Thanks guys! Is there any way to know that a rollback was done (if I set my docker swarm to rollback automatically)?
Otherwise, it basically looks like I'm just waiting and waiting for a result, thinking that my deployment is not done (when I'm using CI/CD) - but in reality rollback has been performed!

Perhaps some sort of event is fired?

If not, I guess I would have to do something along the lines as @sorenhansendk suggested. But I think this is something docker swarm should provide out of the box (unless I'm missing something?)

[sorenhansendk]

@drozzy: Yes - please change this line:

($Service.UpdateStatus.State -like 'rollback*')

to this one:

($Service.UpdateStatus.State -eq 'rollback_completed')

[sudo-bmitch]

Adding to the list of external implementations, here's a script to run after a docker stack deploy to wait until the deploy completes.
https://github.com/sudo-bmitch/docker-stack-wait

[vitalets]

I'd like to share my script waiting for docker stack deploy to complete.
https://github.com/vitalets/docker-stack-wait-deploy

Compared to existing solutions it shows update progress per instance of each service.
Example output:

STATUS (nodejs_service): updating
ID                  NAME                   IMAGE                     NODE                DESIRED STATE       CURRENT STATE             ERROR               PORTS
h1isbgdgmwv1        nodejs_service.1       nodejs_service_prod:1.5   swarm-manager       Running             Preparing 2 seconds ago                       
8zu8v3waxyjx         \_ nodejs_service.1   nodejs_service_prod:1.4   swarm-manager       Running             Running 6 minutes ago                         
b45471xoq9d4        nodejs_service.2       nodejs_service_prod:1.4   swarm-manager       Running             Running 6 minutes ago                         
xyagddf48560        nodejs_service.3       nodejs_service_prod:1.5   swarm-manager       Running             Running 3 seconds ago                         

[decentral1se]

If someone would like to discuss being funded to work on this issue, please gimme a shout [email protected]. We need this as part of a project which we're using swarm as the underlying orchestrator and having to write custom workarounds for the fact that --detach is missing is not ideal. There are so many +1s on this issue, clearly a lot of people need this to get done.

[robertjgtoth]

It's been almost 2 years since the last activity on this ticket... makes me think the devs don't really actively monitor these tickets, which is unfortunate for such a widely used product. If they really wanted to compete with k8s, they'd prioritize quality-of-life features like this.

[gmargaritis]

Hey everyone 👋

I got to work on this long-standing issue with the help of @withlogicco! I hope to close this one soon!

In the meantime, you can test the binary (2023.05.03-master.withlogicco.detach) that includes implementations of the —detach flag for both stack deploy and stack rm.

Happy deployments!

[decentral1se]

Yessss @gmargaritis! Let us know how we can support you! This feature is so so so needed. In our downstream project, we implemented a work-around that kept things going for the time bring but I'd love to have this upstream functionality. More in https://git.coopcloud.tech/coop-cloud/organising/issues/209

[thaJeztah]

Closing, as this has been implemented for the upcoming v26.0 release in;

• Add support for --detach flag in stack deploy #4258
• Add support for --detach flag in stack rm #4259

With the above PRs merged, this feature is functional, but the UX is not perfect yet, so a tracking-ticket is created to look at future improvements;

• Follow-up improvements for docker stack --detach options #4907

[gmargaritis]

👋 @thaJeztah,

Thank you for your comments and co-ordination! Also, I would like to thank @withlogicco for supporting me in this one!

I’ll be more than happy to continue with a follow-up, and make it even more user-friendly.

[decentral1se]

@gmargaritis can i just say you're an absolute hero for fixing this. some heroes do not wear capes. you deserve a medal! this makes the entire ecosystems deployments and undeployments more stable. it's really wild how much impact this change has... already enjoying the benefits!

[gmargaritis]

@decentral1se Super happy to hear that it's already making an impact 🎉

Thank you for the kind words ✌️