Is it available for Docker Swarm?

[laferte-tech]

Hello,

first of all, thanks a lot for your work, seems very nice!
i would like to know if your image could work for a docker swarm architecture?

In my case, i have only one node (one vps) and only one replica of each container (php, mysql,...) but instead of starting/stopping independent containers i need to start/stop a service.
and i tried manually to stop one container (docker stop) but the service restarts it automatically just after (so i can't backup right now with your solution).
Do i need to change something in my docker compose or is it possible to modify your script to stop and start a whole service?

Thanks a lot

[m90]

Thanks for raising this issue. I was tangentially aware of it, but as I rarely use swarm myself I kind of forgot about it.

From what I understand, stopping a container in swarm will make swarm mode deploy a replacement instantly as you describe, so that won't work.

How would a command that start/stops a service look like when running in swarm mode? I think it might be possible to issue a different command in case the setup is running in swarm mode, so we might be able to work around this.

[laferte-tech]

Hi,
thanks a lot for your answer.
For my use case, i use this command to start the service:

docker stack deploy -c docker-compose.prod.yaml --with-registry-auth service_name --prune

i specify some options that are not mandatory (like the docker-compose file) but useful. So maybe it would be very nice to be able to choose the service name (mandatory) and provide some options to start the service (optional) through the shell. Unfortunately my skills are not enough good to help to translate this command in shell.

and to stop:
docker stack rm service_name

thanks!

[m90]

I looked into this a bit further and one question came up: what happens (in case you can) if you set your service's restart_policy to on-failure? (https://docs.docker.com/compose/compose-file/compose-file-v3/#restart_policy)

Maybe this is all you need to do to stop swarm from recreating the container instantly?

[laferte-tech]

Hi,
thanks for this hint :) actually yes you are right, it's working for this point but another problem came up.
I tried with restart-policy:on-failure and i can stop the container with docker stop and it's not restarting. So it's all good for this.

But actually i cannot start it again with docker start... since we are in the swarm mode, i need to stop /start services and not individual containers (since, from my understand, a service can have several containers)

So to make it work, i think there are 2 options:
start/stop the whole stack (with docker stack deploy and docker stack rm )
start/stop individual services (with docker service update (https://stackoverflow.com/questions/44811886/restart-one-service-in-docker-swarm-stack) and docker service rm)

Maybe, would it be possible to enable an option for swarm mode and then instead of taking the containers id, we would take the services id and then start/stop with the appropriate commands?

[m90]

I think the following approach should work for us here:

• using a on-failure restart policy is mandatory if you want to use this tool when running in swarm mode
• stopping the labeled containers will work as it does right now
• when restarting, instead of just calling docker start the script needs to check whether the stopped container was part of a swarm service
  • in case no, we just restart it again
  • in case yes, we remove the old container and update --force the service so that it starts back up (ideally doing this only once for each service in case it was running more than one container)

Do you think that should work?

[laferte-tech]

Yes that would be perfect, thanks a lot!

Since i'm here, i'm experiencing a new problem while trying to test it in production. Maybe do you have any idea how to solve this?
It was working fine in dev without swarm but when i try to put the service in production inside my stack with Swarm,i have the following error:

mc: <ERROR> Unable to initialize new alias from the provided credentials. The AWS Access Key Id you provided does not exist in our records.
 (5) alias-set.go:319 cmd.mainAliasSet(..) Tags: [backup-target, https://s3.amazonaws.com, "access_key", "scret_key"]
 (4) alias-set.go:243 cmd.BuildS3Config(..) Tags: [https://s3.amazonaws.com, "access_key", "secret_key", , auto]
 (3) alias-set.go:217 cmd.probeS3Signature(..) Tags: [s3v4, s3v2]
 (2) alias-set.go:209 cmd.probeS3Signature.func1(..) Tags: [s3v2]
 (1) client-s3.go:1419 cmd.(*S3Client).Stat(..) Tags: [probe-bucket-sign-usuhca9zfb0j]
 (0) client-s3.go:1985 cmd.(*S3Client).bucketStat(..)
 Release-Tag:RELEASE.2021-06-13T17-48-22Z | Commit:DEVELOPMENT. | Host:b4cc6fc803ed | OS:linux | Arch:amd64 | Lang:go1.16.5 | Mem:3.4 MB/73 MB | Heap:3.4 MB/67 MB.

it seems related to this: minio/mc#3421 that the MinIO server is not available yet when starting the container.

[m90]

that the MinIO server is not available yet when starting the container.

Is the MinIO server part of the same stack?

[laferte-tech]

i'm sorry i'm not an expert in this, but i'm just using your image and no other container is related to MinIO.
Reading your dockerfile, i guess that the "run mc" command comes to late compared to the entrypoint where the mc command is used.
It's just a guess of course but maybe before calling the mc command in the entrypoint, shouldn't we check that mc is available?

[m90]

The problem is not related to the mc command in the container being unavailable, but it seems it cannot probe the backup location with the configuration you provide (i.e. the issue is remote, not local).

• Do you backup against AWS S3 or a MinIO server?
• Did you set AWS_ENDPOINT in case you do not use S3?

[m90]

mc: Unable to initialize new alias from the provided credentials. The AWS Access Key Id you provided does not exist in our records.

Could it be that your Docker version is slightly outdated and you are running into this issue: docker/compose#2854 which would mean your access keys are inadvertently quoted? You can test this easily by defining it like this:

AWS_ACCESS_KEY_ID=SOMETHINGSOMETHING

instead of:

AWS_ACCESS_KEY_ID="SOMETHINGSOMETHING"

[m90]

Swarm is now supported in v1.6.0, thank you for input on this ✌️

If you need help on that other issue (mc not picking up credentials correctly) feel free to open another issue.

[laferte-tech]

mc: Unable to initialize new alias from the provided credentials. The AWS Access Key Id you provided does not exist in our records.

Could it be that your Docker version is slightly outdated and you are running into this issue: docker/compose#2854 which would mean your access keys are inadvertently quoted? You can test this easily by defining it like this:

AWS_ACCESS_KEY_ID=SOMETHINGSOMETHING

instead of:

AWS_ACCESS_KEY_ID="SOMETHINGSOMETHING"

Hi, yes very good hint, it was totally this! Actually i have the same and the last version of Docker both on my computer and on my vps but it's not working with "" on VPS... and after checking it's not working for all env vars defined in en .env file...weird, but anyway, i have to check this; must be something else.

For Swarm, thanks a lot for your quick release, really nice. I just experienced an issue when it was time to restart service.
Here the log of backup.sh

[INFO] Stopping containers

8b2b39820fec
d457af9d324c
c5ae82e26d97
57e6fe92c1fe
ca14d59e1182

[INFO] Creating backup

backup/db/binlog.000013
...

[INFO] Starting containers/services back up

Removing 8b2b39820fec
Removing d457af9d324c
Removing c5ae82e26d97
Removing 57e6fe92c1fe
Removing ca14d59e1182
"docker service update" requires exactly 1 argument.
See 'docker service update --help'.

Usage:  docker service update [OPTIONS] SERVICE

Update a service

[INFO] Uploading backup to remote storage

From what i see in the error of service update, the error seems to come from this line:
for SERVICE_NAME in "$(echo -n "$SERVICES_REQUIRING_UPDATE" | tr ' ' '\n' | sort -u)"; do
maybe it's taking more than a service name at a time?

Thanks again for your excellent work!

[m90]

Yeah it was indeed taking more than one service name at a time because the expression is quoted (I only tested against a setup with one service with multiple replicas where it makes no difference...). This is fixed in v1.6.1 https://github.com/offen/docker-volume-backup/releases/tag/v1.6.1

[laferte-tech]

thanks a lot, it works!