Pip 25.0 regression: truststore is not used for installing build dependencies

[BeefaloKing]

Description

I am behind a SSL terminating proxy, and have installed my own root CA.
I have a python source project with a standard pyproject.toml which includes dependencies

dependencies = [
    ...
]

After updating from pip 24.3.1 to 25.0, if I perform a pip install ., I see SSLErrors

In the same environment, pip install <dependency> continues to work without any SSL errors, so I'm confident I have installed my root CA correctly. Problem appears isolated to installing dependencies read from a pyproject.toml file.

While pip helpfully tells me note: This error originates from a subprocess, and is likely not a problem with pip., the only change made to my environment is the pip version, and reverting back to 24.3.1 fixes the issue, so I do believe it is a problem with pip.

Expected behavior

Pip should use my system installed root CA for SSL connections while installing dependencies listed in a pyproject.toml.

pip version

25.0

Python version

3.11.11

OS

RHEL 9

How to Reproduce

1. Be behind a proxy that terminates SSL connections and presents a new cert resigned with its own root CA.
2. Verify you've correctly installed your own root CA, and that pip will use it when installing packages (e.g., pip install --upgrade setuptools should return no error).
3. Attempt to build/install a python package with a pyproject.toml that lists dependencies (e.g., pip install .).

Output

$ pip install .
Processing
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Collecting setuptools>=61.0
        WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
      ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Thanks for reporting, can you also please provide your configuration, e.g. pip config list

[BeefaloKing]

Just in case it helps, and to clarify, I am using setuptools as my build system.

[build-system]                         
requires = ["setuptools>=61.0"]        
build-backend = "setuptools.build_meta"

In both environments where I can and cannot reproduce the issue, the actual installed version of setuptools was unchanged (75.8.0).

Thanks for reporting, can you also please provide your configuration, e.g. pip config list

Returns no output.

$ pip config list

[notatallshaw]

Thanks, it may take a while to debug, do you know how to downgrade pip as a workaround for now?

@ichard26 any thoughts? I assume this is related to passing the cert config to the subprocess, could there be an issue when there is an empty config?

[BeefaloKing]

Yes, I can downgrade to 24.3.1 as a workaround for now.

Thanks for taking the time to look into this.

[ichard26]

Good morning, how are you passing certificate and proxy configuration to pip? Via environment variables? Or have you done nothing special to tell pip about the system certificate and proxy. I'd like to know as pip has different logic when a cert/proxy is explicitly given (--cert, --proxy, or via the corresponding envvars) or implicitly used.

[BeefaloKing]

My proxy settings are set via environment variables (e.g., HTTPS_PROXY, HTTP_PROXY). I'm not doing anything special to tell pip about the cert, rather it's installed via the system wide trust store.

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/securing_networks/using-shared-system-certificates_securing-networks#the-system-wide-trust-store_using-shared-system-certificates

[ichard26]

Hmm, #13063 changed pip to always pass --cert to the build dependency install subprocess, either to pip's CA bundle, or to whatever the user had set. I wonder if that disables truststore (which is what enables system CAs to work with zero-config) in the build environment.

I unfortunately don't have a system CA that I can easily test with, could you try this patch?

diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
index e820dc3d5..9b7582c42 100644
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -246,8 +246,8 @@ class BuildEnvironment:
             # target from config file or env var should be ignored
             "--target",
             "",
-            "--cert",
-            finder.custom_cert or where(),
+            # "--cert",
+            # finder.custom_cert or where(),
         ]
         if logger.getEffectiveLevel() <= logging.DEBUG:
             args.append("-vv")

You can install the modified pip via pip install https://github.com/ichard26/pip/archive/truststore-hotfix.zip.

[BeefaloKing]

Yep, that patch fixes it.

Processing
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

[ichard26]

Coolio. I wonder if we even need to still pass --cert when we're simply using the pip's own CA bundle. I'll take a look at a patch some time this week when I have more time.

I'm glad we could figure out the root issue quickly. Thank you for being so cooperative @BeefaloKing! Sorry about the regression. (It's funny because the changes were supposed to make the build environment respect the user's cert/proxy configuration in all situations.)

[sbidoul]

@BeefaloKing we have merged the fix. If by chance you could test again with pip main that would be very helpful.

[BeefaloKing]

It will be a few days before I can test again, but I'll drop a comment then.

[notatallshaw]

Hey @BeefaloKing I was trying to reproduce this error to better understand it, and I couldn't, even in an environment that relies on truststore (the feature in question).

It would be really helpful if could run some commands on pip 24.3.1 and let me know the output so we can better understand the conditions under which the error occurs:

pip install --dry-run --no-cache --use-deprecated=legacy-certs requests

python -c "from pip._vendor.certifi import where; print(where())"

pip install --dry-run --no-cache --cert "{PATH OF ABOVE OUTPUT}" requests

[BeefaloKing]

@BeefaloKing we have merged the fix. If by chance you could test again with pip main that would be very helpful.

Confirmed today, pip install . worked today with pip installed from git+https://github.com/pypa/pip.git@main.

Hey @BeefaloKing I was trying to reproduce this error to better understand it, and I couldn't, even in an environment that relies on truststore (the feature in question).
[...]

$ which pip
/<venv>/bin/pip
$ pip --version
pip 25.0 from /<venv>/lib64/python3.11/site-packages/pip (python 3.11)
$ python -c "from pip._vendor.certifi import where; print(where())"
/<venv>/lib64/python3.11/site-packages/pip/_vendor/certifi/cacert.pem
$ pip install --dry-run --no-cache --cert '/<venv>/lib64/python3.11/site-packages/pip/_vendor/certifi/cacert.pem' requests
Collecting requests                                                                                                                                                                                                                            
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl.metadata
[...]

(I see the same certificate errors when I pass --use-deprecated=legacy-certs as well.)

Fwiw, if I'm not in a venv certifi spits out the location of the truststore ca-bundle (though RHELs packaged version of pip is older).

$ which python3
/usr/bin/python3
$ which pip3
/usr/bin/pip3
$ pip3 --version
pip 22.3.1 from /usr/lib/python3.11/site-packages/pip (python 3.11)
$ python3 -c "from pip._vendor.certifi import where; print(where())"
/etc/pki/tls/certs/ca-bundle.crt

[BeefaloKing]

Fwiw, if I'm not in a venv certifi spits out the location of the truststore ca-bundle (though RHELs packaged version of pip is older).

Er, but ofc it does.
https://src.fedoraproject.org/rpms/python-pip/blob/rawhide/f/dummy-certifi.patch

[notatallshaw]

Thanks, can you please confirm those commands I sent you on 24.3.1 give the same errors as 25.0, particularly passing the cert.

[BeefaloKing]

Those commands give me identical output with pip 24.3.1 (Same SSL errors, and where() returns the same path to the bundled cacert.pem).

[notatallshaw]

Thanks! I'm still confused why I can't trigger this bug, but your output makes logical sense to the regression you saw.