Meta-ticket: SSL issues #30556

slel · 2020-09-11T18:11:02Z

Building Sage with its own Python 3 can easily produce
a Python 3 missing its _ssl module.

Python's _ssl module is only built if OpenSSL (with headers)
is available, either via the system or via the openssl spkg.

The result is reduced functionality:

it hinders installing pip packages
it hinders working with Jupyter

See also:

many reports on Ask Sage and sage-support
sage-release: cannot start Jupyter due to missing _ssl module

Tickets:

Add spkg-configure.m4 for openssl #30557: Add spkg-configure.m4 for openssl
openssl: Make build more robust by clearing some environment variables #31094 openssl: Make build more robust by clearing some environment variables
Patch tornado so it can be imported in Python without an ssl module #30674: Patch tornado so it can be imported in Python without an ssl module
Upgrade: OpenSSL 3.0, make it a standard package #29555: Upgrade: OpenSSL 3.0, make it a standard package
Check for required extension module "ssl" in python3 build and spkg-configure #29291: Check for more strictly required extension module "ssl" in Python build
Inclusion of OpenSSL, stage 1 #24107: Inclusion of OpenSSL, stage 1
Upgrade: OpenSSL 3.0.0.beta2 #32311: Upgrade: OpenSSL 3.0.0.beta2
make openssl a dependency for pip packages #23893: make openssl a dependency for pip packages
Document that SSL is optional #22620: Document that SSL is optional
Remove outdated SSL section in the "install from source code" documentation #17690: A SSL section in the "install from source code" documentation
sage-download-file: Proper initialization of SSL certificates #30950: sage-download-file: Proper initialization of SSL certificates
Remove pip package "pyopenssl" #32374 Remove pip package pyopenssl

References:

https://www.python.org/dev/peps/pep-0644/ - Require OpenSSL 1.1 or newer

Depends on #30557
Depends on #30560
Depends on #30383

CC: @mwageringel @mkoeppe @slel @williamstein @dimpase

Component: packages: standard

Keywords: openssl, python

Issue created by migration from https://trac.sagemath.org/ticket/30556

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-09-11T18:22:41Z

Dependencies: #30557

slel · 2020-09-11T19:10:29Z

comment:3

Many builds of the python3 spkg without its _ssl module
likely result from an oversight.

Could there be a configure flag to explicitly require
building python3 without its _ssl module?

Without that flag, consider openssl a dependency of python3,
and let the build fail if it's not there.

slel · 2020-09-11T19:23:22Z

comment:4

Even when configuring with --enable-openssl,
I think openssl may be built after python3.

So currently one really has to separate into two steps:

$ make openssl
$ make

or later have to repair with

$ sage -i openssl
$ sage -f python3

mkoeppe · 2020-09-11T19:27:36Z

comment:5

Replying to @slel:

Even when configuring with --enable-openssl,
I think openssl may be built after python3.

Yes, this is a problem. We do not have a mechanism to order "optional dependencies".

mkoeppe · 2020-09-11T19:31:27Z

comment:6

See #21700 - Packages with "optional" dependencies (a package manager's suggested/recommended packages)

mkoeppe · 2020-09-11T19:33:09Z

comment:7

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

mkoeppe · 2020-09-11T21:47:16Z

comment:8

Replying to @mkoeppe:

We could add openssl to TOOLCHAIN_DEPS as a workaround. Then it would be built at the very beginning.

This is now #30560

mkoeppe · 2020-09-13T19:36:01Z

Changed dependencies from #30557 to #30557, #30560

mkoeppe · 2020-09-13T19:54:32Z

Changed dependencies from #30557, #30560 to #30557, #30560, #30383

mkoeppe · 2020-09-13T19:54:32Z

comment:10

If we are building python3 and openssl is not available as a system package and not enabled, then we should disable the packages that depend on openssl -- Jupyter notebook (and dependencies) and all source=pip packages. The latter are already optional packages by definition. But we would need give the Jupyter notebook packages a new package type: optional-enabled-by-default (#30383).

mkoeppe · 2020-09-13T19:58:30Z

comment:11

Replying to @slel:

Could there be a configure flag to explicitly require
building python3 without its _ssl module?

This would effectively make openssl a standard package. I don't think we should do that because of license reasons. Warnings about the disabled features are better.

mwageringel · 2020-09-19T08:32:35Z

comment:12

I would like to mention that the SSL requirement for Jupyter seems to be a recent change – presumably the upgrade of tornado or other Jupyter packages in 9.2.beta9. This was not clear to me and part of why I was confused on the mailing list. I was actually using the Python 3 SPKG without SSL all along and until now I have not experienced any problem with this. In particular, this is not related to the Python 3.8 upgrade as I had assumed at first.

mkoeppe · 2020-09-19T13:14:46Z

comment:13

This might be worth investigating more.

mwageringel · 2020-09-27T10:58:50Z

comment:14

Replying to @mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in this commit.

mkoeppe · 2020-09-27T18:45:49Z

comment:15

Replying to @mwageringel:

Replying to @mkoeppe:

This might be worth investigating more.

As suspected, this is a consequence of the Jupyter upgrades in #26919, merged in 9.2.beta9. The dependency on ssl in tornado/httpserver.py was added in this commit.

Looks like this would be easy to patch out.

mkoeppe · 2020-09-27T21:49:45Z

comment:16

This is now #30674

mkoeppe · 2020-12-15T04:28:28Z

comment:18

Building a Python without ssl module is happening on ubuntu-trusty (https://github.com/sagemath/sage/runs/1553902103) in 9.3.beta4

Our configure finds system openssl; system python3 is too old, so python 3.8 is built from source; but apparently it fails to build the ssl module. (This should be investigated!)

The symptom is that pytest (a pip package that is a check-only dependency of some standard package) fails to install:

[pytest] installing. Log file: /sage/logs/pkgs/pytest.log
  [pytest] error installing, exit status 1. End of log file:
  [pytest]   WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
  [pytest]   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pytest/

It is likely that this problem has actually been around for a while and was only masked by the faulty SAGE_CHECK logic fixed in #31020.

mkoeppe · 2020-12-15T04:38:14Z

comment:19

Same in ubuntu-xenial-standard.

Also in various -minimal builds, of course, where the SSL headers are not available. For example ubuntu-hirsute-minimal (https://github.com/sagemath/sage/runs/1553902375)

mkoeppe · 2020-12-16T19:59:12Z

comment:20

#31062 "tox / GH Actions: Disable testsuites of packages depending on pip packages (pytest, ...) if there is no ssl" proposes a workaround

mkoeppe · 2021-03-26T21:01:57Z

comment:24

Moving it to 9.4; we seem to have an OK solution for now

mkoeppe · 2023-02-10T22:52:27Z

I think we can close this. Haven't seen any issues with SSL in a long time.

slel added this to the sage-9.2 milestone Sep 11, 2020

slel added c: packages: standard labels Sep 11, 2020