Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vnet-manager installation fails on Ubuntu 20.04 due to old pyopenssl #60

Closed
Southparkfan opened this issue Nov 7, 2023 · 4 comments
Closed

vnet-manager installation fails on Ubuntu 20.04 due to old pyopenssl #60

Southparkfan opened this issue Nov 7, 2023 · 4 comments

Comments

@Southparkfan
Copy link

Southparkfan commented Nov 7, 2023 •
edited
Loading

Hi Erik,

Nice to meet you. You're an alumnus of OS3, I'm a new student 👋

We're using vnet-manager in our labs, in Ubuntu 20.04 VMs. Unfortunately, I've had to deal with a bug during initial setup of vnet-manager. TL;DR there is a workaround (upgrading pyopenssl).

I have installed a clean VM:

root@guest-02:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

(+ https://github.com/Erik-Lamers1/vnet-manager#install-the-required-packages)

Running pip3 install vnet-manager:

root@guest-02:~/vnet-manager# pip3 install vnet-manager
Collecting vnet-manager
  Downloading vnet_manager-1.1.0-py3-none-any.whl (38 kB)
Collecting PyYAML>=6.0
  Downloading PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (736 kB)
     |████████████████████████████████| 736 kB 35.9 MB/s 
Collecting pylxd>=2.3.1
  Downloading pylxd-2.3.1.tar.gz (77 kB)
     |████████████████████████████████| 77 kB 6.0 MB/s 
Collecting distro>=1.7.0
  Downloading distro-1.8.0-py3-none-any.whl (20 kB)
Collecting tabulate>=0.8.9
  Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
Collecting psutil>=5.9.0
  Downloading psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
     |████████████████████████████████| 283 kB 46.8 MB/s 
Collecting pyroute2==0.7.3
  Downloading pyroute2-0.7.3-py3-none-any.whl (445 kB)
     |████████████████████████████████| 445 kB 46.5 MB/s 
Collecting colorama>=0.4.4
  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting cryptography>=3.2
  Downloading cryptography-41.0.5-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB)
     |████████████████████████████████| 4.4 MB 45.1 MB/s 
Collecting python-dateutil>=2.4.2
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB 69.1 MB/s 
Collecting requests-toolbelt>=0.8.0
  Downloading requests_toolbelt-1.0.0-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 2.5 MB/s 
Requirement already satisfied: requests-unixsocket>=0.1.5 in /usr/lib/python3/dist-packages (from pylxd>=2.3.1->vnet-manager) (0.2.0)
Requirement already satisfied: requests>=2.20.0 in /usr/lib/python3/dist-packages (from pylxd>=2.3.1->vnet-manager) (2.22.0)
Collecting ws4py!=0.3.5,>=0.3.4
  Downloading ws4py-0.5.1.tar.gz (51 kB)
     |████████████████████████████████| 51 kB 146 kB/s 
Collecting cffi>=1.12
  Downloading cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (444 kB)
     |████████████████████████████████| 444 kB 59.3 MB/s 
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.4.2->pylxd>=2.3.1->vnet-manager) (1.14.0)
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     |████████████████████████████████| 118 kB 80.9 MB/s 
Building wheels for collected packages: pylxd, ws4py
  Building wheel for pylxd (setup.py) ... done
  Created wheel for pylxd: filename=pylxd-2.3.1-py3-none-any.whl size=121866 sha256=39172683c793b6425815f9b35b3cb85375cde988a5b66040888c72f68f7df5f3
  Stored in directory: /root/.cache/pip/wheels/1b/ba/db/f0e675c17be723606096131e6f5059e5e05e7a9904f30bb4dd
  Building wheel for ws4py (setup.py) ... done
  Created wheel for ws4py: filename=ws4py-0.5.1-py3-none-any.whl size=45215 sha256=8fd5a8e22548cf9f7e483749c1da41f7450b1bf994c2bce09f2bf3e49e64f8ae
  Stored in directory: /root/.cache/pip/wheels/ea/f9/a1/34e2943cce3cf7daca304bfc35e91280694ced9194a487ce2f
Successfully built pylxd ws4py
Installing collected packages: PyYAML, pycparser, cffi, cryptography, python-dateutil, requests-toolbelt, ws4py, pylxd, distro, tabulate, psutil, pyroute2, colorama, vnet-manager
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 5.3.1
    Not uninstalling pyyaml at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'PyYAML'. No files were found to uninstall.
  Attempting uninstall: cryptography
    Found existing installation: cryptography 2.8
    Not uninstalling cryptography at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'cryptography'. No files were found to uninstall.
  Attempting uninstall: distro
    Found existing installation: distro 1.4.0
    Not uninstalling distro at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'distro'. No files were found to uninstall.
  Attempting uninstall: colorama
    Found existing installation: colorama 0.4.3
    Not uninstalling colorama at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'colorama'. No files were found to uninstall.
Successfully installed PyYAML-6.0.1 cffi-1.16.0 colorama-0.4.6 cryptography-41.0.5 distro-1.8.0 psutil-5.9.6 pycparser-2.21 pylxd-2.3.1 pyroute2-0.7.3 python-dateutil-2.8.2 requests-toolbelt-1.0.0 tabulate-0.9.0 vnet-manager-1.1.0 ws4py-0.5.1

vnet-manager cannot be executed:

root@guest-02:~/vnet-manager# vnet-manager 
Traceback (most recent call last):
  File "/usr/local/bin/vnet-manager", line 5, in <module>
    from vnet_manager.vnet_manager import main
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager/vnet_manager.py", line 8, in <module>
    from vnet_manager.actions.manager import ActionManager
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager/actions/manager.py", line 6, in <module>
    import vnet_manager.operations.machine as machine_op
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager/operations/machine.py", line 8, in <module>
    from pylxd.exceptions import NotFound, LXDAPIException
  File "/usr/local/lib/python3.8/dist-packages/pylxd/__init__.py", line 17, in <module>
    from pylxd.client import Client, EventType
  File "/usr/local/lib/python3.8/dist-packages/pylxd/client.py", line 21, in <module>
    import requests
  File "/usr/lib/python3/dist-packages/requests/__init__.py", line 95, in <module>
    from urllib3.contrib import pyopenssl
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 12, in <module>
    import os, glob, subprocess, os.path, time, pwd, sys, requests_unixsocket
  File "/usr/lib/python3/dist-packages/requests_unixsocket/__init__.py", line 1, in <module>
    import requests
  File "/usr/lib/python3/dist-packages/requests/__init__.py", line 95, in <module>
    from urllib3.contrib import pyopenssl
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/bin/vnet-manager", line 5, in <module>
    from vnet_manager.vnet_manager import main
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager/vnet_manager.py", line 8, in <module>
    from vnet_manager.actions.manager import ActionManager
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager/actions/manager.py", line 6, in <module>
    import vnet_manager.operations.machine as machine_op
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager/operations/machine.py", line 8, in <module>
    from pylxd.exceptions import NotFound, LXDAPIException
  File "/usr/local/lib/python3.8/dist-packages/pylxd/__init__.py", line 17, in <module>
    from pylxd.client import Client, EventType
  File "/usr/local/lib/python3.8/dist-packages/pylxd/client.py", line 21, in <module>
    import requests
  File "/usr/lib/python3/dist-packages/requests/__init__.py", line 95, in <module>
    from urllib3.contrib import pyopenssl
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

This error was fixed by reinstalling pyopenssl (cf. https://stackoverflow.com/a/75295873). In yet another clean VM I have verified running pip3 install pyopenssl --upgrade (upgrade to v23.3.0) before installing vnet-manager also fixes the issue. focal's python3-openssl package is based on v19.0.0, which is too old.

On top of this, I had encountered another error during topology creation (TypeError: request() got an unexpected keyword argument 'chunked'), which I fixed by installing request==2.28.0, but I couldn't reproduce this bug... probably interesting for later.
@Southparkfan
Copy link
Author

Southparkfan commented Nov 7, 2023 •
edited
Loading

Output from pipdeptree:

[snip]
pyOpenSSL==23.3.0
└── cryptography [required: >=41.0.5,<42, installed: 41.0.5]
    └── cffi [required: >=1.12, installed: 1.16.0]
        └── pycparser [required: Any, installed: 2.21]
[snip]
vnet-manager==1.1.0
├── colorama [required: >=0.4.4, installed: 0.4.6]
├── distro [required: >=1.7.0, installed: 1.8.0]
├── psutil [required: >=5.9.0, installed: 5.9.6]
├── pylxd [required: >=2.3.1, installed: 2.3.1]
│   ├── cryptography [required: >=3.2, installed: 41.0.5]
│   │   └── cffi [required: >=1.12, installed: 1.16.0]
│   │       └── pycparser [required: Any, installed: 2.21]
│   ├── python-dateutil [required: >=2.4.2, installed: 2.8.2]
│   │   └── six [required: >=1.5, installed: 1.14.0]
│   ├── requests [required: >=2.20.0, installed: 2.22.0]
│   ├── requests-toolbelt [required: >=0.8.0, installed: 1.0.0]
│   │   └── requests [required: >=2.0.1,<3.0.0, installed: 2.22.0]
│   ├── requests-unixsocket [required: >=0.1.5, installed: 0.2.0]
│   └── ws4py [required: >=0.3.4,!=0.3.5, installed: 0.5.1]
├── pyroute2 [required: ==0.7.3, installed: 0.7.3]
├── PyYAML [required: >=6.0, installed: 6.0.1]
└── tabulate [required: >=0.8.9, installed: 0.9.0]

I wouldn't consider myself to be a Python expert, but I think all of this boils down to a compatibility issue between cryptography and pyOpenSSL. The developers of pylxd could pin higher versions for their dependencies. The lab teachers have tested the INR lab on 22.04, which has python3-openssl v21.0.0, so I guess that OS works just fine.

@Erik-Lamers1
Copy link
Owner

Erik-Lamers1 commented Nov 7, 2023 •
edited
Loading

The requests issue should be fixed in release 1.1.1 with PR #61

I cannot reproduce the openssl issue. I tried to make a fresh install on Ubuntu 20.04 with Py3.8 and the latest packages.

  • system OpenSSL 1.1.1f-1ubuntu2.20
  • python cryptography==41.0.5

The pyopenssl package should not be needed by vnet-manager

@Southparkfan
Copy link
Author

Southparkfan commented Nov 8, 2023 •
edited
Loading

Thanks for your reply! I've used this image: https://cloud-images.ubuntu.com/focal/current/ -> focal-server-cloudimg-amd64-disk-kvm.img

On a fresh install (you might have used a different image!), these packages are installed:

ii  python3-cryptography           2.8-3ubuntu0.1                    amd64        Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-openssl                19.0.0-1build1                    all          Python 3 wrapper around the OpenSSL library

Installing python3-pip did NOT cause an upgrade for these packages. After purging python3-openssl using apt, I installed vnet-manager using pip3. This trigger a library upgrade for cryptography: cryptography-41.0.5-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (instead of 2.8 as provided by Ubuntu!), and pyOpenSSL will not be installed. vnet-manager seems to work just fine.

Now... after installing python3-openssl, pip3 install vnet-manager refuses to install vnet-manager: AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK. So yeah, either you need to upgrade pyOpenSSL before installing vnet-manager, or you need to purge python3-openssl before installing. vnet-manager and the older version pyOpenSSL cannot co-exist. If you executed your test without python3-openssl, I'm not surprised you weren't able to reproduce this bug.

Co-installing cryptography==35.0 and pyOpenSSL==22.0.0 (per https://pypi.org/project/pyOpenSSL/22.0.0/) also works, by the way, so it looks like these packages have something to do with each other. Vincent and I are aware pyOpenSSL is not a dependency for vnet-manager, cryptography is however (for requests, required by pylxd), and pyOpenSSL also requires cryptography (but not vice versa). Talking about rabbit holes....

Apparently, we should not use pyOpenSSL if possible, see https://pypi.org/project/pyOpenSSL/. Even while pyOpenSSL may not be a hard dependency in vnet-manager and/or pylxd, requests certainly is a hard dependency. And if pyOpenSSL exists on the system (which it does by default!), it will be used in requests (see the traceback in the error). Work is underway to remove any usage of pyOpenSSL inrequests, see psf/requests#5443, a commit that is included in v2.24.0.

You have added requests as a requirement, but the version constraints will not prevent a requests version older than '2.24.0' from being installed. After re-installing my focal guest and installing vnet-manager from source, vnet-manager still throws the pyOpenSSL-related error. I was curious if changing the constraint to `>=2.24.0,<2.30.0' would work.

Re-installed the guest yet another time. After applying this change:

diff --git a/requirements/base.txt b/requirements/base.txt
index def0384..2bb889c 100644
--- a/requirements/base.txt
+++ b/requirements/base.txt
@@ -5,4 +5,4 @@ pylxd>=2.3.1
 pyroute2==0.7.3
 psutil>=5.9.0
 distro>=1.7.0
-requests<2.30.0
+requests>=2.24.0,<2.30.0

requests v2.29.0 was installed during installation of vnet-manager. Guess what? It's not requests, but ws4py (another dependency of pylxd) relying on pyOpenSSL:

Traceback (most recent call last):
  File "/usr/local/bin/vnet-manager", line 11, in <module>
    load_entry_point('vnet-manager==1.1.1', 'console_scripts', 'vnet-manager')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2445, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2451, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager-1.1.1-py3.8.egg/vnet_manager/vnet_manager.py", line 8, in <module>
    from vnet_manager.actions.manager import ActionManager
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager-1.1.1-py3.8.egg/vnet_manager/actions/manager.py", line 6, in <module>
    import vnet_manager.operations.machine as machine_op
  File "/usr/local/lib/python3.8/dist-packages/vnet_manager-1.1.1-py3.8.egg/vnet_manager/operations/machine.py", line 8, in <module>
    from pylxd.exceptions import NotFound, LXDAPIException
  File "/usr/local/lib/python3.8/dist-packages/pylxd-2.3.1-py3.8.egg/pylxd/__init__.py", line 17, in <module>
    from pylxd.client import Client, EventType
  File "/usr/local/lib/python3.8/dist-packages/pylxd-2.3.1-py3.8.egg/pylxd/client.py", line 25, in <module>
    from ws4py.client import WebSocketBaseClient
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.8/dist-packages/ws4py-0.5.1-py3.8.egg/ws4py/client/__init__.py", line 10, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.8/dist-packages/ws4py-0.5.1-py3.8.egg/ws4py/websocket.py", line 11, in <module>
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

ws4py is yet another piece of software not requiring pyOpenSSL, but it'll definitely use it if available. Okay, looks like for lots of packages, pyOpenSSL is considered to be a soft dependency. After some more fiddling, I've found out installing pyOpenSSL v21.0.0 (the oldest version including the commit where the reference to X509_V_FLAG_CB_ISSUER_CHECK was removed ) also works.

Conclusion

  • PyOpenSSL is installed by default on (at least) certain focal images
  • PyOpenSSL is not a dependency for pylxd/vnet-manager, but various dependencies of pylxd (at least requests and ws4py) will use it if installed
  • The python3-openssl package is too old for the installed version of cryptography
  • Removing PyOpenSSL before installing vnet-manager fixes the issue, but I haven't tested if something else would break w/o PyOpenSSL
  • Adding PyOpenSSL >=21.0.0 to base/requirements.txt master HEAD also fixes the issue

Either we remove support for focal, we ask people to uninstall (using apt) or upgrade (using pip3) python3-openssl/PyOpenSSL on focal systems before installing vnet-manager on focal, we introduce a hard dependency on PyOpenSSL with a proper version constraint, or we ask the pylxd developers to incorporate a fix in their requirements file. What do you think?

Erik-Lamers1 added a commit that referenced this issue Nov 11, 2023
@Erik-Lamers1
Update docs
234fd4d 
Based on issues seen in #60
Recomment to use venv
@Erik-Lamers1
Copy link
Owner

Thanks for the explanation @Southparkfan, I see now that this is an issue with system packages interfering with the setup.
I was able to reproduce this on my test 20.04 system as well using the default system packages and trying to install vnet-manager. I think this mostly comes down to a documentation issue. As the README recommends to install vnet-manager in the global system env, these issues are bound the happen on older OS'es. Its not only vnet-manager that suffers from this, every package that uses the pyopenssl lib is broken in this case, even PIP itself.

I updated the README to recommend people to install vnet-manager in a virtualenv now (at least on older OS'es).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Erik-Lamers1 @Southparkfan