Support custom CA bundles. #10757

benjyw · 2020-09-11T00:00:45Z

Supports fetching binaries etc. from behind a proxy.

Slightly refactors fs_test.py to separate the download-related
tests from the filesystem-related tests, as that single test class was
getting unwieldy.

Removes the old https test, that had to hit a live server over the internet,
since we now have a proper local test with its own test PKI.

Includes a script to generate that PKI, so future modifiers don't have
to struggle with openssl too much.

src/python/pants/engine/fs_test.py

benjyw · 2020-09-11T01:39:59Z

Reviewers, please pay extra attention to the Rust code, as Rust is not my forte...

Eric-Arellano

Epic! Thanks, Benjy!

src/python/pants/engine/fs_test.py

src/python/pants/engine/internals/tls_testing/generate_certs.sh

src/rust/engine/src/context.rs

[ci skip-build-wheels] [ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

src/rust/engine/src/context.rs

coveralls · 2020-09-11T21:52:37Z

Coverage remained the same at 0.0% when pulling f256ffc on benjyw:custom_root_ca into 62810c8 on pantsbuild:master.

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

src/python/pants/option/global_options.py

Eric-Arellano

Btw, do we need to pass these cert files to each Process somehow?

benjyw · 2020-09-11T23:31:37Z

Btw, do we need to pass these cert files to each Process somehow?

Not right now. Processes might take it as an env var such as REQUESTS_CA_BUNDLE for requests.

matze999 · 2020-09-21T12:48:16Z

Hi @benjyw,
thanks for your work. I still have issues making it work.

I specified:
pants_version = "2.0.0b0"
I added the github certificate to my certificate store
I run as:
export PYTHON=/usr/bin/python3.8; export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt; ./pants binary src/python:: --print-exception-stacktrace -ldebug

I get:
14:24:20.06 [ERROR] 1 Exception encountered:

Engine traceback:
in select
in binary goal
in pants.backend.python.goals.create_python_binary.create_python_binary
in pants.backend.python.util_rules.pex.two_step_create_pex
in pants.backend.python.util_rules.pex.create_pex
in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
in pants.core.util_rules.external_tool.download_external_tool
in Downloading: DownloadFile
Traceback (no traceback):

Exception: Error downloading file: error sending request for url (https://github.com/pantsbuild/pex/releases/download/v2.1.16/pex): error trying to connect: invalid certificate: UnknownIssuer

The following works:
openssl s_client -showcerts -connect github.com:443

Can you direct me on what i need to specify?

Thanks,
Matt

thamenato · 2020-09-21T13:16:15Z

Hey @matze999 I managed to get this to work by setting the new flag --ca-certs-path. Eg:
./pants test tests/libs:: --ca-certs-path=/path/to/ca_cert.crt

You can also add that ca-certs-path inside your pants.toml (if I'm not mistaken it's under [source]) you can check that using ./pants help-advanced

benjyw · 2020-09-21T20:24:28Z

Thanks @thamenato!

This is actually a global option, so in pants.toml it would be:

[GLOBAL]
...
ca_certs_path = "/path/to/certs_file"
...

benjyw · 2020-09-21T22:18:00Z

Documented in https://www.pantsbuild.org/docs/troubleshooting#using-pants-behind-a-proxy

matze999 · 2020-09-22T16:01:07Z

Hi @benjyw ,
first of all thank for your work which got me further in the process.
In my [GLOBAL] section i set:
ca_certs_path = "/etc/ssl/certs/ca-certificates.crt"

I read what you wrote in the doc.
I also tried to pass some environment variables to point to my certificate store.
I further added the https://pypi.org certificate to my strust store.

However, i am still stuck. It appears to me that it is trying to setup a python virtual environment and trying to get the package from pypi.org. I could be wrong though.

Here is what happens:

`05:47:43.65 [ERROR] 1 Exception encountered:

Engine traceback:
in select
in binary goal
in pants.backend.python.goals.create_python_binary.create_python_binary
in pants.backend.python.util_rules.pex.two_step_create_pex
in pants.backend.python.util_rules.pex.create_pex
in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
File "/home/mafunk/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py38/lib/python3.8/site-packages/pants/engine/process.py", line 235, in fallible_to_exec_result_or_raise
raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Resolving 15 requirements: en-core-web-md@ https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz, hdbscan==0.8.26, motor==2.1.0, nltk==3.2.5, numpy==1.18.4, pymongo==3.10.1, python-dotenv==0.13.0, scikit-learn==0.22.2.post1, scipy==1.4.1, sentence-transformers==0.2.6.1, spacy==2.2.4, textblob==0.15.3, torch==1.5.0, transformers==2.11.0, umap-learn==0.4.4' failed with exit code 1.
stdout:

stderr:
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'))': /explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'))': /explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'))': /explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'))': /explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'))': /explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz
ERROR: Exception:
Traceback (most recent call last):
File "/home/mafunk/.cache/pants/named_caches/pex_root/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/pip/pip/_vendor/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/home/mafunk/.cache/pants/named_caches/pex_root/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/pip/pip/_vendor/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/home/mafunk/.cache/pants/named_caches/pex_root/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/pip/pip/_vendor/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/home/mafunk/.cache/pants/named_caches/pex_root/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/pip/pip/_vendor/urllib3/connection.py", line 386, in connect
self.sock = ssl_wrap_socket(
File "/home/mafunk/.cache/pants/named_caches/pex_root/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/pip/pip/vendor/urllib3/util/ssl.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)
`

Any idea of what else i can do here? I am stuck ...

benjyw · 2020-09-22T19:18:03Z

Oh! We should also be passing that cert setting into Pip (via Pex). And possibly other processes. Will get a fix up today.

@thamenato I'm guessing this didn't impact you because you're using your own PyPI mirror?

thamenato · 2020-09-22T23:18:54Z

That's puzzling that it didn't work.

@benjyw actually I run it locally by accessing normal PyPI mirror (I still need the ca-certs-path so my proxy works) and inside my CI I have my own PyPI mirror/wheels cache.
I was having the same issue that @matze999 mentioned while trying to bootstrap pants v2: it would try to get the .pex binary from Github but the SSL would fail with invalid certificate: UnknownIssuer.

In my case my case I have to use a self-signed certificate to make SSL work properly using my company's proxy so everything needs to be set: REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, etc ... but adding it to ca_certs_path did the trick for me, it bootstrapped and installed all external packages.

benjyw · 2020-09-23T00:45:36Z

It may have worked because you had the PyPI artifacts cached on the local pip cache, so it never needed to access the network.

benjyw · 2020-09-23T02:47:25Z

Fixed in #10837

matze999 · 2020-09-24T06:36:08Z

Hi @benjyw ,
thanks again ... :)

Is there any way to already try out the fix you made? I am still specifying version 2.0.0b0 as i did not see any other version. I was also looking for nightly builds but did not find anything? Else, can you tell when and how it will be available?

Thanks,
Matt

Eric-Arellano · 2020-09-24T06:39:12Z

Hey Matt, I'm about 20 minutes away from releasing 2.0.0b1 :) As soon as our "Build Wheels" jobs go green: https://travis-ci.com/github/pantsbuild/pants/jobs/390257318

Also see the note on https://github.com/pantsbuild/setup/blob/c95cde54860587e798c64c3960beeb271b897f0d/pants#L24-L27 for how to consume nightly builds.

Let us know if we can help with anything else, too! We're usually most responsive via Slack: https://www.pantsbuild.org/docs/community, but GitHub issues work too.

Eric-Arellano · 2020-09-24T07:14:46Z

Released: https://pypi.org/project/pantsbuild.pants/2.0.0b1/

Change pants_version in your pants.toml to "2.0.0b1".

matze999 · 2020-09-24T09:52:49Z

It worked. Thank you guys very much! Congratulations on having good people working for your team ... 👍

benjyw · 2020-09-24T18:47:58Z

Glad to help! Let us know if you hit any further issues.

Eric-Arellano · 2020-10-23T20:43:40Z

Hey @matze999, we added a new page dedicated to using Pants with proxies. We'd love any feedback you have! https://www.pantsbuild.org/docs/proxies

benjyw force-pushed the custom_root_ca branch 3 times, most recently from 1278125 to 8b55762 Compare September 11, 2020 01:31

benjyw changed the title ~~WIP not for review~~ Support custom CA bundles. Sep 11, 2020

benjyw requested review from stuhood and Eric-Arellano September 11, 2020 01:33

benjyw commented Sep 11, 2020

View reviewed changes

src/python/pants/engine/fs_test.py Show resolved Hide resolved

benjyw mentioned this pull request Sep 11, 2020

Support setting a custom certificate bundle #10572

Closed

Eric-Arellano approved these changes Sep 11, 2020

View reviewed changes

src/python/pants/engine/fs_test.py Show resolved Hide resolved

src/python/pants/engine/internals/tls_testing/generate_certs.sh Show resolved Hide resolved

Eric-Arellano requested a review from tdyas September 11, 2020 01:42

Eric-Arellano reviewed Sep 11, 2020

View reviewed changes

src/rust/engine/src/context.rs Outdated Show resolved Hide resolved

gshuflin reviewed Sep 11, 2020

View reviewed changes

src/rust/engine/src/context.rs Show resolved Hide resolved

benjyw added 3 commits September 11, 2020 13:51

Support custom CA bundles.

1b0a11f

[ci skip-build-wheels] [ci skip-rust]

Simplify file reading

60e3c42

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Fix test and address code review comment

790e5c4

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

benjyw force-pushed the custom_root_ca branch from 48e3963 to 790e5c4 Compare September 11, 2020 21:01

gshuflin reviewed Sep 11, 2020

View reviewed changes

src/rust/engine/src/context.rs Outdated Show resolved Hide resolved

gshuflin reviewed Sep 11, 2020

View reviewed changes

src/rust/engine/src/context.rs Outdated Show resolved Hide resolved

gshuflin approved these changes Sep 11, 2020

View reviewed changes

Use path.display()

f256ffc

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano reviewed Sep 11, 2020

View reviewed changes

src/python/pants/option/global_options.py Show resolved Hide resolved

Eric-Arellano reviewed Sep 11, 2020

View reviewed changes

benjyw merged commit a9eabf5 into pantsbuild:master Sep 11, 2020

benjyw deleted the custom_root_ca branch September 11, 2020 23:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support custom CA bundles. #10757

Support custom CA bundles. #10757

benjyw commented Sep 11, 2020 •

edited

Loading

benjyw commented Sep 11, 2020

Eric-Arellano left a comment

coveralls commented Sep 11, 2020 •

edited

Loading

Eric-Arellano left a comment

benjyw commented Sep 11, 2020

matze999 commented Sep 21, 2020 •

edited

Loading

thamenato commented Sep 21, 2020

benjyw commented Sep 21, 2020

benjyw commented Sep 21, 2020

matze999 commented Sep 22, 2020 •

edited

Loading

benjyw commented Sep 22, 2020

thamenato commented Sep 22, 2020

benjyw commented Sep 23, 2020

benjyw commented Sep 23, 2020

matze999 commented Sep 24, 2020

Eric-Arellano commented Sep 24, 2020

Eric-Arellano commented Sep 24, 2020

matze999 commented Sep 24, 2020

benjyw commented Sep 24, 2020

Eric-Arellano commented Oct 23, 2020

Support custom CA bundles. #10757

Support custom CA bundles. #10757

Conversation

benjyw commented Sep 11, 2020 • edited Loading

benjyw commented Sep 11, 2020

Eric-Arellano left a comment

Choose a reason for hiding this comment

coveralls commented Sep 11, 2020 • edited Loading

Eric-Arellano left a comment

Choose a reason for hiding this comment

benjyw commented Sep 11, 2020

matze999 commented Sep 21, 2020 • edited Loading

thamenato commented Sep 21, 2020

benjyw commented Sep 21, 2020

benjyw commented Sep 21, 2020

matze999 commented Sep 22, 2020 • edited Loading

benjyw commented Sep 22, 2020

thamenato commented Sep 22, 2020

benjyw commented Sep 23, 2020

benjyw commented Sep 23, 2020

matze999 commented Sep 24, 2020

Eric-Arellano commented Sep 24, 2020

Eric-Arellano commented Sep 24, 2020

matze999 commented Sep 24, 2020

benjyw commented Sep 24, 2020

Eric-Arellano commented Oct 23, 2020

benjyw commented Sep 11, 2020 •

edited

Loading

coveralls commented Sep 11, 2020 •

edited

Loading

matze999 commented Sep 21, 2020 •

edited

Loading

matze999 commented Sep 22, 2020 •

edited

Loading