SSL error when attempting to use data.nextstrain.org aligned files (north-america)

[jacaravas]

When attempting to include a remote input in the builds.yaml file, an SSL certificate error occurs.

It is not clear to me whether the error is due to a code bug, error in my builds.yaml file, configuration error on my end, an actual SSL certificate, firewall issue, etc... Can you verify that this is working as intended and give some troubleshooting tips if it is?

Builds.yaml inputs block (simplified):

`
inputs:

• name: north-america
  aligned: https://data.nextstrain.org/files/ncov/open/north-america/aligned.fasta.xz
  metadata: https://data.nextstrain.org/files/ncov/open/north-america/metadata.tsv.xz
  `

When this input is included, the analysis fails with

requests.exceptions.SSLError: HTTPSConnectionPool(host='data.nextstrain.org', port=443): Max retries exceeded with url: /files/ncov/open/north-america/aligned.fasta.xz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))
When that input and associated subsampling references are removed, the builds.yaml file works as intended.

Full error log below.

`>Building DAG of jobs...
WARNING: No valid subsampling scheme is defined for build 'basic'. Skipping subsampling and using all available samples.
Traceback (most recent call last):
File "home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in validate_conn
conn.connect()
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/connection.py", line 416, in connect
self.sock = ssl_wrap_socket(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/my_conda_envs/nextstrain/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/my_conda_envs/nextstrain/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/my_conda_envs/nextstrain/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='data.nextstrain.org', port=443): Max retries exceeded with url: /files/ncov/open/north-america/aligned.fasta.xz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/init.py", line 699, in snakemake
success = workflow.execute(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/workflow.py", line 763, in execute
dag.init()
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 179, in init
job = self.update([job], progress=progress, create_inventory=True)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 874, in update_
selected_job = self.update(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 759, in update
self.update_(
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/dag.py", line 863, in update_
if not res.file.exists:
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/io.py", line 453, in exists
return self.exists_remote
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/io.py", line 247, in wrapper
v = func(self, *args, **kwargs)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/io.py", line 474, in exists_remote
return self.remote_object.exists()
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/remote/HTTP.py", line 159, in exists
with self.httpr(verb="HEAD") as httpr:
File "/home/my_conda_envs/nextstrain/lib/python3.9/contextlib.py", line 119, in enter
return next(self.gen)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/snakemake/remote/HTTP.py", line 152, in httpr
r = requests.head(url, *args_to_use, **kwargs_to_use)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/requests/api.py", line 102, in head
return request('head', url, **kwargs)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/my_conda_envs/nextstrain/lib/python3.9/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='data.nextstrain.org', port=443): Max retries exceeded with url: /files/ncov/open/north-america/aligned.fasta.xz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))`

[tsibley]

Thanks for the report. I can confirm that the certificates for https://data.nextstrain.org are ok.

There's two things I suspect to be the issue. Either:

1. An out-of-date local CA trust store. Since you're using Conda, it's likely the CA trust store in use is from Conda packages instead of your base OS's trust store. Could you try running conda list -n nextstrain | grep certif and reporting back the output? I'm expecting two packages to match.

2. An intervening HTTP proxy that's man-in-the-middling (MITM) your connection. This is unfortunately not uncommon in some network environments. MITM proxies are also used in development to inspect traffic, but you'd likely know about it in that case. I suspect possible MITM because there isn't a self-signed cert in the cert chain from data.nextstrain.org, yet the error message says self signed certificate in certificate chain, and MITM proxies often use self-signed certs. In network environments with MITM proxies, the MITM self-signed CA is usually added to device trust stores by IT so that things Just Work. It wouldn't get added to your Conda environment, however, leading to an error like this one. You might be able to spot it in the output of this command openssl s_client -connect data.nextstrain.org:443 < /dev/null.

[jacaravas]

Good morning @tsibley,
Thank you for the response. The results of the grep you suggested are:
ca-certificates 2021.10.8 ha878542_0 conda-forge
certifi 2021.10.8 py39hf3d152e_1 conda-forge

[tsibley]

@jacaravas Well those are certainly new enough, so it's likely not situation 1.

Can you run

openssl s_client -connect data.nextstrain.org:443 < /dev/null

and report the output?

[jacaravas]

@tsibley After a long pause, I got:

CONNECTED(00000003)
140391746942864:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1636393176
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

[jacaravas]

@tsibley I just tried the same command on a different machine and got a different response. This may be a machine config issue.

CONNECTED(00000003)
depth=1 C = US, ST = Georgia, L = Atlanta, O = Centers for Disease Control and Prevention, OU = OCISO, CN = CDC PaloAlto, emailAddress = [email protected]
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 C = US, ST = Georgia, L = Atlanta, O = Centers for Disease Control and Prevention, OU = OCISO, CN = CDC PaloAlto, emailAddress = [email protected]
verify return:1
depth=0 CN = data.nextstrain.org
verify return:1
---
Certificate chain
 0 s:CN = data.nextstrain.org
   i:C = US, ST = Georgia, L = Atlanta, O = Centers for Disease Control and Prevention, OU = OCISO, CN = CDC PaloAlto, emailAddress = [email protected]
 1 s:C = US, ST = Georgia, L = Atlanta, O = Centers for Disease Control and Prevention, OU = OCISO, CN = CDC PaloAlto, emailAddress = [email protected]
   i:C = US, ST = Georgia, L = Atlanta, O = Centers for Disease Control and Prevention, OU = OCISO, CN = CDC PaloAlto, emailAddress = [email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID7TCCAtWgAwIBAgIQYM1WRt/KVCwXN6wd0wKPTjANBgkqhkiG9w0BAQsFADCB
rDELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0dlb3JnaWExEDAOBgNVBAcTB0F0bGFu
dGExMzAxBgNVBAoTKkNlbnRlcnMgZm9yIERpc2Vhc2UgQ29udHJvbCBhbmQgUHJl
dmVudGlvbjEOMAwGA1UECxMFT0NJU08xFTATBgNVBAMTDENEQyBQYWxvQWx0bzEd
MBsGCSqGSIb3DQEJARYOZnd0ZWFtQGNkYy5nb3YwHhcNMjAxMjMwMDAwMDAwWhcN
MjIwMTI4MjM1OTU5WjAeMRwwGgYDVQQDExNkYXRhLm5leHRzdHJhaW4ub3JnMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwT+rrReMhRBX9e+d2oPoBHOO
MCsipMDCFYmP5U+oDRLh1FmWL61a6uGfViYs8ubiAqLzZsDJNTbRn0mNAGgEQQ6w
NcVDMhyIL0Rh+aR2LoIrjtDBr9GgoQSePUIVgHAL39+nFnZIbCK4fPcMf0m93UVV
Eff+4YCYHHgTH1ZC1NEjS71/TLTomZT51RFB+XvBedV9yyxGL2SryS41nCwB5DDC
ng8Qpu9OY+1qnpTslFY5vLt5lsHwJZrs5F+nWka8zgi5/cdT5VyjmybnuY7SG6mf
XdMiPtMSmWta4HZwd977miAs0peFl7Dg4nQhPxC5/lcZUBG4BoYZbVjzrfStXwID
AQABo4GXMIGUMB0GA1UdDgQWBBSBej+HsRG5Lk6s0Ki4K2duYN+f8TA2BgNVHREE
LzAtghNkYXRhLm5leHRzdHJhaW4ub3JnghZzdGFnaW5nLm5leHRzdHJhaW4ub3Jn
MA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
DAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAQEAIMQUlb4n42a1QAk6zxt0
I2o8Cqgc8laJ4x1Gpp81sGLDOvjK1Pai+ITbSvSTjXMKV5HfBJH+GouskEBKJ72H
PD+m5aJ17ibmJfNB6onTDXoFT2Dq+DXeUANMc38/3iAIIW+R7isbXY9ws9ApFWv5
Ccw9TuAMpVredvEoAxvf4xoRX//7j9SqwteDZ6X1CwZI11J0BRGrb33w+nTyl1+6
dZvs7f1WfSAFU/hKjWwChndspty1mfE2RpaQqQf3BHiLszsfTbAA+3SMikZht08t
uk6imxmEiSpb+P5v31UAu1G/R/ezLA4YWXkRjT31CdbuSmFu0i3Zt8xHmS2qA1eQ
8A==
-----END CERTIFICATE-----
subject=CN = data.nextstrain.org

issuer=C = US, ST = Georgia, L = Atlanta, O = Centers for Disease Control and Prevention, OU = OCISO, CN = CDC PaloAlto, emailAddress = [email protected]

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2522 bytes and written 447 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: E1701148A39AA01BEA13DE600D0423E46BE87EFFF1AFBE8674D9319AD13C26C2
    Session-ID-ctx:
    Master-Key: 99D9C467DB8A0F3DAC511EB2FA28F54A16DB1812A3E8AD0F8E9DC3A685E0FC78034E3B815CF2F66CAD481A75BEB29674
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1636393457
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
---
DONE

[tsibley]

@jacaravas Thank you for troubleshooting. Yes, this is a machine/network configuration issue. Your second response shows that your network operations team at CDC is MITMing connections (ostensibly for security). Note that the certificate reported is:

Certificate chain
 0 s:CN = data.nextstrain.org
   i:C = US, ST = Georgia, L = Atlanta, O = Centers for Disease Control and Prevention, OU = OCISO, CN = CDC PaloAlto, emailAddress = [email protected]

s: means "subject" and i: means "issuer". The real certificate is:

 0 s:CN = data.nextstrain.org
   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

You'll need to contact the network team ([email protected] is listed on the cert) or internal IT support to resolve this.

[jacaravas]

@tsibley Thank you very much for your help!

[jacaravas]

Updating with resolution if anyone else encounters this error.
Non-standard certs needed to be specified in the environment with

export REQUESTS_CA_BUNDLE=/path/to/converted/certificate.pem

See:
https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/non-standard-certs.html