List of ports that should be opened for plumbum/zerodeploy to work.

[msvbhat]

I was trying to connect to one docker container from another docker container. But when both the containers are in same host, the zerodeploy connection works. But when they are in different host the zerodeploy (plumbum.SshMachine) actually doesn't work. I get EOFError (refer to my last topic in this group)

The cluster is setup using etcd/flannel. And I can ssh from one container to another container in another host.

So I think it might be a problem with ports bot being accessible or plumbum trying to use non-default port.

Can anybody tell, if this an plumbum issue or my setup issue?

Thanks,
MS

[henryiii]

I believe, from a quick perusal of the code, that it is randomly selecting an open port, using something like socket.bind((host, 0)). I'm assuming this is RPyC zero deploy. Plumbum itself uses the default port (should be the same one the ssh command uses), but has the option for a different port, which zero deploy seems to use. I assume it may make multiple connections?

[msvbhat]

Thanks for the quick response henry.

The weird thing is, I can ping and ssh to the machine from the bash. But when I try to connect using SshMachine(), I get following exception.

from plumbum import SshMachine
rem = SshMachine("10.0.11.3")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/plumbum/machines/ssh_machine.py", line 106, in init
new_session = new_session)
File "/usr/lib/python2.7/site-packages/plumbum/machines/remote.py", line 150, in init
self._session = self.session(new_session = new_session)
File "/usr/lib/python2.7/site-packages/plumbum/machines/ssh_machine.py", line 150, in session
self.encoding, isatty, self.connect_timeout)
File "/usr/lib/python2.7/site-packages/plumbum/machines/session.py", line 141, in init
self.run("")
File "/usr/lib/python2.7/site-packages/plumbum/machines/session.py", line 224, in run
return run_proc(self.popen(cmd), retcode)
File "/usr/lib/python2.7/site-packages/plumbum/commands/processes.py", line 244, in run_proc
stdout, stderr = proc.communicate()
File "/usr/lib/python2.7/site-packages/plumbum/machines/session.py", line 92, in communicate
line = pipe.readline()
File "/usr/lib/python2.7/site-packages/plumbum/machines/session.py", line 42, in readline
raise EOFError()
EOFError

Does a single SshMachine() makes multiple connections? Since this happens even before calling any RPyC functions, I don't think its any issue with RPyC (yet)

Also, is there any way we can specify a range? So that in case of containers (where every port is not open across bridge), it has some workaround functions?

Thanks again.

MS

[henryiii]

You can get extended logging using:

import logging
logging.basicConfig(level=logging.DEBUG)

before you run that command. What does that show? I get the command ssh -T localhost, indicating that it is not adding a specific port (in my case).

[msvbhat]

Thanks again Henry,

I get the below output. This looks more and more weird to me.

rem = SshMachine("10.0.11.3")
DEBUG:plumbum.local:Running ['/usr/bin/ssh', '-T', '10.0.11.3']
DEBUG:plumbum.shell:Running "true ; echo $? ; echo '--.END60452207.245.--' ; echo '--.END60452207.245.--' 1>&2"
Traceback (most recent call last):
File "", line 1, in

What does EOFError exception indicate? I thought it means the other end did not reply. Maybe it is because of some capabilities (like selinux, cap sets) mismatch in docker containers?

Thanks

MS

[henryiii]

The EOFError is a bit general, since it means that the reply was not all that was expected. Plumbum works by the command you see above, it sends a request (like echo $?), followed by two echo commands with tags (The END things above), one to stdout, and one to stderr (Using 1>&2). If your default remote shell does not support the redirection to stderr that you see here, you will get an EOFError. (Plumbum 1.6 will have a slightly better error message for this). Do your remote shells support redirection in exactly this form (i.e., are they bash?)

[msvbhat]

Thanks again Henry,

The remote container is running rhel, so is running bash. But since plumbum can connect to same container running in local system and have trouble connecting only to container running remotely, I think it must be something to do with either ports (which is kind of ruled out with above discussion) or some selinux/capsets which rejects the incoming request in remote machine.

The other problem might be the etcd/flannel connection which might be dropping the packets from this node/process.

I do not know where else to look for errors. Any other ideas with container?

Thanks

MS

[henryiii]

Have you tried manually calling ssh from the remote then seeing if echo "hi" 1>&2 works?

[msvbhat]

Thanks. I just tried the same.

[root@6aa709be5247 ~]# echo "Hello" 1>&2
Hello

[root@6aa709be5247 ~]# echo "Hello" 2>/dev/null 1>&2
[root@6aa709be5247 ~]#

The redirection seems to be working just fine. Anything else plumbum might be looking for? Like some encoding capabilities or anything? Still can't figure where the problem might be.

Thanks

MS

[henryiii]

I'm not sure why it's not working now, then. Figured that was worth a shot, anyway. I would start adding debug outputs (ie, print(...)) directly in the source. You could even drop a pdb breakpoint in. Remember, you can grab the git source then install with pip install -e .

I can't, unfortunately, suggest much more without a broken example to fix. I don't think it does more than pipes through the standard ssh command ( @tomerfiliba might know? ) for the example you have.

[msvbhat]

Thanks @henryiii for you help so far. I will try to check the exact difference between connecting to container in same host and different host.

@tomerfiliba Any thoughts why it might be behaving the way it is?

Thanks...

MS