Complete adding podman connection to synchronize #230

sshnaidm · 2021-07-25T10:32:11Z

SUMMARY

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

synchronize

ADDITIONAL INFORMATION

Continue of #229
For fixing ansible-community/molecule-plugins#81

sshnaidm · 2021-07-25T10:32:33Z

@tadeboro please take a look if it's OK.

plugins/action/synchronize.py

tadeboro

I am not an expert at this module, but I would expect that the podman needs a similar special treatment just like the docker and buildah connection plugins have at

ansible.posix/plugins/action/synchronize.py

Lines 416 to 424 in 1ebacfb

    
           if self._remote_transport in ['docker', 'community.general.docker', 'community.docker.docker']: 
        
               if become and self._play_context.become_user: 
        
                   _tmp_args['rsync_opts'].append("--rsh=%s exec -u %s -i" % (self._docker_cmd, self._play_context.become_user)) 
        
               elif user is not None: 
        
                   _tmp_args['rsync_opts'].append("--rsh=%s exec -u %s -i" % (self._docker_cmd, user)) 
        
               else: 
        
                   _tmp_args['rsync_opts'].append("--rsh=%s exec -i" % self._docker_cmd) 
        
           elif self._remote_transport in ['buildah', 'containers.podman.buildah']: 
        
               _tmp_args['rsync_opts'].append("--rsh=buildah run --")

I do not have access to my work machine with the podman installed right now, which means that I cannot confirm if any special treatment is actually needed here. I will try to test this out in about 12 hours.

sshnaidm · 2021-07-25T19:38:47Z

Yeah, probably it requires more work, moving to draft for now

Akasurde

Also, add a changelog entry.

plugins/action/synchronize.py

codecov · 2021-07-26T10:10:40Z

Codecov Report

Merging #230 (1ebacfb) into main (1ebacfb) will not change coverage.
The diff coverage is n/a.

❗ Current head 1ebacfb differs from pull request most recent head 133ffe5. Consider uploading reports for the commit 133ffe5 to get more accurate results

@@           Coverage Diff           @@
##             main     #230   +/-   ##
=======================================
  Coverage   25.65%   25.65%           
=======================================
  Files          41       41           
  Lines        3918     3918           
  Branches      770      770           
=======================================
  Hits         1005     1005           
  Misses       2716     2716           
  Partials      197      197

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ebacfb...133ffe5. Read the comment docs.

ssbarnea · 2021-07-26T11:26:12Z

May I suggest disabling codecov comments? It requires only one line https://github.com/ansible-community/ansible-compat/blob/main/codecov.yml#L1 and it does not disable codecov a run and ability to browse the full reports if you press on the job result. Still, no more spam in PR conversations.

tadeboro · 2021-08-03T15:11:34Z

I just tested the code from this PR and things still failed. I am not sure what is the main reason, but I just wanted to show what I see.

Playbook:

---
- hosts: tmp

  tasks:
    - name: Sync
      ansible.posix.synchronize:
        src: collections
        dest: /tmp/collections

Inventory:

---
all:
  hosts:
    tmp:
      ansible_connection: containers.podman.podman
      ansible_host: 109fbc505407

Output:

$ ansible-playbook -vvvv -i inv.yaml play.yaml 
ansible-playbook [core 2.11.3] 
  config file = None
  configured module search path = ['/home/tadej/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/tadej/tmp/venv/lib64/python3.9/site-packages/ansible
  ansible collection location = /home/tadej/tmp/collections
  executable location = /home/tadej/tmp/venv/bin/ansible-playbook
  python version = 3.9.6 (default, Jul 16 2021, 00:00:00) [GCC 11.1.1 20210531 (Red Hat 11.1.1-3)]
  jinja version = 3.0.1
  libyaml = True
No config file found; using defaults
setting up inventory plugins
host_list declined parsing /home/tadej/tmp/inv.yaml as it did not pass its verify_file() method
script declined parsing /home/tadej/tmp/inv.yaml as it did not pass its verify_file() method
Parsed /home/tadej/tmp/inv.yaml inventory source with yaml plugin
Loading collection ansible.posix from /home/tadej/tmp/collections/ansible_collections/ansible/posix
Loading callback plugin default of type stdout, v2.0 from /home/tadej/tmp/venv/lib64/python3.9/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: play.yaml *******************************************************************************************************************************************
Positional arguments: play.yaml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/tadej/tmp/inv.yaml',)
forks: 5
1 plays in play.yaml

PLAY [tmp] ****************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************
task path: /home/tadej/tmp/play.yaml:2
Loading collection containers.podman from /home/tadej/tmp/collections/ansible_collections/containers/podman
Using podman connection from collection
<109fbc505407> RUN [b'/usr/bin/podman', b'mount', b'109fbc505407']
Failed to mount container 109fbc505407: b'Error: cannot run command "podman mount" in rootless mode, must execute `podman unshare` first'
<109fbc505407> RUN [b'/usr/bin/podman', b'exec', b'109fbc505407', b'/bin/sh', b'-c', b'echo ~ && sleep 0']
<109fbc505407> RUN [b'/usr/bin/podman', b'exec', b'109fbc505407', b'/bin/sh', b'-c', b'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505 `" && echo ansible-tmp-1628002539.775511-120351-11322244557505="` echo /root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505 `" ) && sleep 0']
<tmp> Attempting python interpreter discovery
<109fbc505407> RUN [b'/usr/bin/podman', b'exec', b'109fbc505407', b'/bin/sh', b'-c', b"echo PLATFORM; uname; echo FOUND; command -v '/usr/bin/python'; command -v 'python3.9'; command -v 'python3.8'; command -v 'python3.7'; command -v 'python3.6'; command -v 'python3.5'; command -v 'python2.7'; command -v 'python2.6'; command -v '/usr/libexec/platform-python'; command -v '/usr/bin/python3'; command -v 'python'; echo ENDFOUND && sleep 0"]
<tmp> Python interpreter discovery fallback (pipelining support required for extended interpreter discovery)
Using module file /home/tadej/tmp/venv/lib64/python3.9/site-packages/ansible/modules/setup.py
<109fbc505407> PUT /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/tmpedbzo1ql TO /root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505/AnsiballZ_setup.py
<109fbc505407> RUN [b'/usr/bin/podman', b'cp', b'/home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/tmpedbzo1ql', b'109fbc505407:/root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505/AnsiballZ_setup.py']
<109fbc505407> RUN [b'/usr/bin/podman', b'exec', b'109fbc505407', b'/bin/sh', b'-c', b'chmod u+x /root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505/ /root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505/AnsiballZ_setup.py && sleep 0']
<109fbc505407> RUN [b'/usr/bin/podman', b'exec', b'109fbc505407', b'/bin/sh', b'-c', b'/usr/bin/python3.7 /root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505/AnsiballZ_setup.py && sleep 0']
<109fbc505407> RUN [b'/usr/bin/podman', b'exec', b'109fbc505407', b'/bin/sh', b'-c', b'rm -f -r /root/.ansible/tmp/ansible-tmp-1628002539.775511-120351-11322244557505/ > /dev/null 2>&1 && sleep 0']
[WARNING]: Platform linux on host tmp is using the discovered Python interpreter at /usr/bin/python3.7, but future installation of another Python interpreter
could change the meaning of that path. See https://docs.ansible.com/ansible/2.11/reference_appendices/interpreter_discovery.html for more information.
ok: [tmp]
META: ran handlers

TASK [Sync] ***************************************************************************************************************************************************
task path: /home/tadej/tmp/play.yaml:5
Loading collection containers.podman from /home/tadej/tmp/collections/ansible_collections/containers/podman
Using podman connection from collection
<109fbc505407> ESTABLISH LOCAL CONNECTION FOR USER: tadej
<109fbc505407> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg `"&& mkdir "` echo /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/ansible-tmp-1628002540.953294-120694-160254713204166 `" && echo ansible-tmp-1628002540.953294-120694-160254713204166="` echo /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/ansible-tmp-1628002540.953294-120694-160254713204166 `" ) && sleep 0'
Using module file /home/tadej/tmp/collections/ansible_collections/ansible/posix/plugins/modules/synchronize.py
<109fbc505407> PUT /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/tmphzwks4vt TO /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/ansible-tmp-1628002540.953294-120694-160254713204166/AnsiballZ_synchronize.py
<109fbc505407> EXEC /bin/sh -c 'chmod u+x /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/ansible-tmp-1628002540.953294-120694-160254713204166/ /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/ansible-tmp-1628002540.953294-120694-160254713204166/AnsiballZ_synchronize.py && sleep 0'
<109fbc505407> EXEC /bin/sh -c '/home/tadej/tmp/venv/bin/python3 /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/ansible-tmp-1628002540.953294-120694-160254713204166/AnsiballZ_synchronize.py && sleep 0'
<109fbc505407> EXEC /bin/sh -c 'rm -f -r /home/tadej/.ansible/tmp/ansible-local-1203473j2_86vg/ansible-tmp-1628002540.953294-120694-160254713204166/ > /dev/null 2>&1 && sleep 0'
fatal: [tmp]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --blocking-io --rsh=podman exec -i '--out-format=<<CHANGED>>%i %n%L' /home/tadej/tmp/collections 109fbc505407:/tmp/collections",
    "invocation": {
        "module_args": {
            "_local_rsync_password": null,
            "_local_rsync_path": "rsync",
            "_substitute_controller": false,
            "archive": true,
            "checksum": false,
            "compress": true,
            "copy_links": false,
            "delay_updates": true,
            "delete": false,
            "dest": "109fbc505407:/tmp/collections",
            "dest_port": null,
            "dirs": false,
            "existing_only": false,
            "group": null,
            "link_dest": null,
            "links": null,
            "mode": "push",
            "owner": null,
            "partial": false,
            "perms": null,
            "private_key": null,
            "recursive": null,
            "rsync_opts": [
                "--blocking-io",
                "--rsh=podman exec -i"
            ],
            "rsync_path": null,
            "rsync_timeout": 0,
            "set_remote_user": true,
            "src": "/home/tadej/tmp/collections",
            "ssh_args": null,
            "ssh_connection_multiplexing": false,
            "times": null,
            "verify_host": false
        }
    },
    "msg": "Error: unknown flag: --server\nrsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: remote shell killed (code 125) at io.c(228) [sender=3.2.3]\n",
    "rc": 125
}

PLAY RECAP ****************************************************************************************************************************************************
tmp                        : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

sshnaidm · 2021-08-03T15:19:01Z

@tadeboro how did you prepare your container?

podman run -dit --rm --name podmancont python:3.8-alpine3.13 sleep 1h
podman exec -it podmancont apk update
podman exec -it podmancont apk add rsync

and running your playbook pass.

sshnaidm · 2021-08-03T15:46:21Z

recheck

tadeboro · 2021-08-03T16:02:55Z

@tadeboro how did you prepare your container?

I used https://hub.docker.com/_/nginx/ as a base, used apt to install python3 and rsync, and then executed the playbook.

podman run -d nginx
podman exec testpod apt update
podman exec testpod apt install python3 rync

sshnaidm · 2021-08-03T16:41:30Z

@tadeboro I ran your config and still works fine, please take a look maybe some differences:

$ podman run -d --name testpod nginx
53b6505754431f6789f3cc99cebd3f094337ff6a3cd27f72864bc4504e88dc69
$ podman exec testpod apt update
....
$ podman exec testpod apt install python3 rsync -y
....

$ cat <<EOF>/tmp/inventory
all:
  hosts:
    tmp:
      ansible_connection: containers.podman.podman
      ansible_host: testpod
EOF
$ cat <<EOF>/tmp/play.yml
---
- hosts: tmp

  tasks:
    - name: Sync
      ansible.posix.synchronize:
        src: /tmp/collections
        dest: /tmp/collections2

EOF     
      
$ ansible-playbook -vv /tmp/play.yml -i /tmp/inventory
...
PLAY RECAP ***********************************************************************************************************************************************************
tmp                        : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0  


...

$ ansible-playbook -vv /tmp/play.yml -i /tmp/inventory
...

TASK [Sync] **********************************************************************************************************************************************************
task path: /tmp/play.yml:5
ok: [tmp] => {
    "changed": false,
    "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --blocking-io --rsh=podman exec -i --out-format=<<CHANGED>>%i %n%L /tmp/collections testpod:/tmp/collections2",
    "rc": 0
}

sshnaidm · 2021-08-03T16:42:24Z

/rebuild_failed

tadeboro · 2021-08-03T16:54:15Z

I have no further ideas about where the difference would come from. The host system I use is a fresh Fedora 34 system and as far as I can tell, I made no changes to the default podman configuration.

sshnaidm · 2021-08-03T17:06:52Z

As I see the error is "Error: unknown flag: --server, I don't know where exactly this flag is passed in your setup, but there is no such option for rsync for sure. I use f33 with current packages:

$ rpm -qa | grep rsync
rsync-3.2.3-3.fc33.x86_64
$ podman --version
podman version 3.2.3
$ ansible --version
ansible [core 2.11.3] 
...

tadeboro · 2021-08-03T17:28:01Z

This seems to be an issue with my setup, so feel free to merge this PR. I can try to determine what is going awry without making a mess out of this PR.

Thanks for the help anyhow, @sshnaidm.

sshnaidm · 2021-08-03T17:30:43Z

@tadeboro thanks for test anyway, probably worth someone else to check the same. So we can be sure it's not something with my setup 😄

tadeboro · 2021-08-03T18:20:01Z

I managed to find what is causing errors on my machine. The culprit is the lack of proper quoting around some rsync arguments. The command that fails is

/usr/bin/rsync \
  --delay-updates \
  -F \
  --compress \
  --archive \
  --blocking-io \
  --rsh=podman exec -i \
  '--out-format=<<CHANGED>>%i %n%L' \
  /home/tadej/tmp/collections \
  109fbc505407:/tmp/collections

When I patched the action plugin a bit to produce the following command, things started working (the difference is in the --rsh parameter):

/usr/bin/rsync \
  --delay-updates \
  -F \
  --compress \
  --archive \
  --blocking-io \
  --rsh='podman exec -i' \
  '--out-format=<<CHANGED>>%i %n%L' \
  /home/tadej/tmp/collections \
  109fbc505407:/tmp/collections

But this has nothing to do with this PR, so I think current changes are ready to ship once AZP stops messing around. I will file a bug for my issue so that it does not get lost.

sshnaidm · 2021-08-04T09:24:27Z

/rebuild_failed

Akasurde

LGTM

ansible-zuul

LGTM!

Akasurde · 2021-08-04T12:07:04Z

@sshnaidm Thanks for the contribution. @ssbarnea @tadeboro Thanks for the reviews and comments.

ssbarnea reviewed Jul 25, 2021

View reviewed changes

plugins/action/synchronize.py Outdated Show resolved Hide resolved

tadeboro reviewed Jul 25, 2021

View reviewed changes

sshnaidm marked this pull request as draft July 25, 2021 19:39

Akasurde requested changes Jul 26, 2021

View reviewed changes

plugins/action/synchronize.py Outdated Show resolved Hide resolved

sshnaidm mentioned this pull request Jul 26, 2021

Release plan #149

Open

sshnaidm force-pushed the podmanfix2 branch from 467b0a1 to 8d7d1e2 Compare August 3, 2021 12:41

sshnaidm marked this pull request as ready for review August 3, 2021 12:42

Complete adding podman connection to synchronize

133ffe5

sshnaidm force-pushed the podmanfix2 branch from 8d7d1e2 to 133ffe5 Compare August 3, 2021 12:44

sshnaidm closed this Aug 4, 2021

sshnaidm reopened this Aug 4, 2021

tadeboro approved these changes Aug 4, 2021

View reviewed changes

Akasurde approved these changes Aug 4, 2021

View reviewed changes

Akasurde added the gate Gate PR in Zuul CI (Obsolete: Please set "mergeit" instead of "gate") label Aug 4, 2021

ansible-zuul bot approved these changes Aug 4, 2021

View reviewed changes

ansible-zuul bot merged commit 5282ece into ansible-collections:main Aug 4, 2021

sshnaidm deleted the podmanfix2 branch August 4, 2021 11:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Complete adding podman connection to synchronize #230

Complete adding podman connection to synchronize #230

sshnaidm commented Jul 25, 2021 •

edited

Loading

sshnaidm commented Jul 25, 2021 •

edited

Loading

tadeboro left a comment

sshnaidm commented Jul 25, 2021

Akasurde left a comment

codecov bot commented Jul 26, 2021 •

edited

Loading

ssbarnea commented Jul 26, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 4, 2021

Akasurde left a comment

ansible-zuul bot left a comment

Akasurde commented Aug 4, 2021

	if self._remote_transport in ['docker', 'community.general.docker', 'community.docker.docker']:
	if become and self._play_context.become_user:
	_tmp_args['rsync_opts'].append("--rsh=%s exec -u %s -i" % (self._docker_cmd, self._play_context.become_user))
	elif user is not None:
	_tmp_args['rsync_opts'].append("--rsh=%s exec -u %s -i" % (self._docker_cmd, user))
	else:
	_tmp_args['rsync_opts'].append("--rsh=%s exec -i" % self._docker_cmd)
	elif self._remote_transport in ['buildah', 'containers.podman.buildah']:
	_tmp_args['rsync_opts'].append("--rsh=buildah run --")

Complete adding podman connection to synchronize #230

Complete adding podman connection to synchronize #230

Conversation

sshnaidm commented Jul 25, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

sshnaidm commented Jul 25, 2021 • edited Loading

tadeboro left a comment

Choose a reason for hiding this comment

sshnaidm commented Jul 25, 2021

Akasurde left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 26, 2021 • edited Loading

Codecov Report

ssbarnea commented Jul 26, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 3, 2021

tadeboro commented Aug 3, 2021

sshnaidm commented Aug 4, 2021

Akasurde left a comment

Choose a reason for hiding this comment

ansible-zuul bot left a comment

Choose a reason for hiding this comment

Akasurde commented Aug 4, 2021

sshnaidm commented Jul 25, 2021 •

edited

Loading

sshnaidm commented Jul 25, 2021 •

edited

Loading

codecov bot commented Jul 26, 2021 •

edited

Loading