1.3 Poetry Install: Isolated PIP build dependency installation fails when behind proxy

[rschneider98]

• Poetry version: 1.3.0 and 1.3.1
• Python version: 3.10.7
• OS version and name: Windows 10
• pyproject.toml:

[tool.poetry]
name = "new-project"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
packages = [{include = "new_project"}]

[[tool.poetry.source]]
name = "jfrog"
url = "https://jfrog.io/artifactory/api/pypi/pypi-virtual/simple"
default = true

[tool.poetry.dependencies]
python = "^3.9"
protobuf3-to-dict = "^0.1.5"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

With the poetry configuration: poetry config http-basic.jfrog $USER $API_TOKEN we are able to pass authentication for poetry's installation for a different python repository. We also set similar settings for the pip.conf.

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Installation of dependencies via python3 -m poetry install fails with PIP subprocess error "subprocess-exited-with-error" due to failure to install build dependencies for protobuf3-to-dict (there are other examples, but this is the simplest).

The subprocess issue is within the pip installation command but is not an issue with pip. In the change from poetry 1.2.2 to 1.3.0, there was the additional flag added for poetry's subprocess call to pip such that pip is executed in an --isolated environment (meaning pip will not look for environment variables or configuration file).

• Relevant Pull Request.
• Source Line.

This prevents a corrupt pip file from preventing poetry from installing dependencies correctly, but in the event pip wants to install dependencies that poetry has not managed to install, then pip defaults to the pythonhosted.org repository (and fails to see any proxy variables). Both of these are issues for my use-case where I am using a separate python repository (JFrog) and am behind a proxy.

When running the install for protobuf3-to-dict, pip goes to install setuptools>=40.8.0 which is a build dependency. The command that poetry runs with pip:

/home/builder/.cache/pypoetry/virtualenvs/src-7MtTr2Jo-py3.8/bin/python -m pip install 
--use-pep517
--disable-pip-version-check
--isolated
--no-input 
--prefix /home/builder/.cache/pypoetry/virtualenvs/src-7MtTr2Jo-py3.8
--no-deps
'/home/builder/.cache/pypoetry/artifacts/2c/c7/ea/157415a270cbad8a95d4275c95328be285912400ab3f825705bc85c4d9/protobuf3-to-dict-0.1.5.tar.gz'

I was able to fix my installation of protobuf3-to-dict by either removing the --isolated flag (which then allows PIP to install the build dependencies based on my pip.conf) or the addition of the --no-build-isolation flag allows PIP to pass the install build dependency step since setuptools==63.6.3 is already installed.

I would like for there to be an additional setting to turn off PIP isolation, this seems like the easiest way to fix this issue.

To avoid PIP installing build dependencies separately, if that is something the poetry community desires, then the --no-build-isolation flag could be added to allow PIP to reference installed build dependencies instead of repeating that work.

Error logs:

Installing dependencies from lock file

Package operations: 98 installs, 0 updates, 0 removals

  • Installing atomicwrites (1.4.1)

  CalledProcessError

  Command 'C:\Users\USER\Python_Projects\visualization-1\.venv\Scripts\python.exe -m pip install --use-pep517 --disable-pip-version-check --isolated --no-input --prefix C:\Users\USER\Python_Projects\visualization-1\.venv --no-deps C:\Users\USER\AppData\Local\pypoetry\Cache\artifacts\ef\c1\a0\4bf107db8d2e7ed8bc65b758fc37582e74a031fdf41a13e152e81a089a\atomicwrites-1.4.1.tar.gz' returned non-zero exit status 1.

  at ~\AppData\Local\Programs\Python\Python39\lib\subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│
       532│

The following error occurred when trying to handle this error:


  EnvCommandError

  Command C:\Users\vadt8y\Python_Projects\visualization-1\.venv\Scripts\python.exe -m pip install --use-pep517 --disable-pip-version-check --isolated --no-input --prefix C:\Users\USER\Python_Projects\visualization-1\.venv --no-deps C:\Users\USER\AppData\Local\pypoetry\Cache\artifacts\ef\c1\a0\4bf107db8d2e7ed8bc65b758fc37582e74a031fdf41a13e152e81a089a\atomicwrites-1.4.1.tar.gz errored with the following return code 1, and output:
  Processing c:\users\USER\appdata\local\pypoetry\cache\artifacts\ef\c1\a0\4bf107db8d2e7ed8bc65b758fc37582e74a031fdf41a13e152e81a089a\atomicwrites-1.4.1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    error: subprocess-exited-with-error

    × pip subprocess to install build dependencies did not run successfully.
    │ exit code: 1
    ╰─> [6 lines of output]
        Collecting setuptools>=40.8.0
          ERROR: HTTP error 403 while getting https://files.pythonhosted.org/packages/ef/e3/29d6e1a07e8d90ace4a522d9689d03e833b67b50d1588e693eec15f26251/setuptools-65.6.3-py3-none-any.whl (from https://pypi.org/simple/setuptools/) (requires-python:>=3.7)
        ERROR: Could not install requirement setuptools>=40.8.0 from https://files.pythonhosted.org/packages/ef/e3/29d6e1a07e8d90ace4a522d9689d03e833b67b50d1588e693eec15f26251/setuptools-65.6.3-py3-none-any.whl because of HTTP error 403 Client Error: Forbidden for url: https://files.pythonhosted.org/packages/ef/e3/29d6e1a07e8d90ace4a522d9689d03e833b67b50d1588e693eec15f26251/setuptools-65.6.3-py3-none-any.whl for URL https://files.pythonhosted.org/packages/ef/e3/29d6e1a07e8d90ace4a522d9689d03e833b67b50d1588e693eec15f26251/setuptools-65.6.3-py3-none-any.whl (from https://pypi.org/simple/setuptools/) (requires-python:>=3.7)

        [notice] A new release of pip available: 22.3 -> 22.3.1
        [notice] To update, run: python.exe -m pip install --upgrade pip
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.


  at ~\AppData\Local\Programs\Python\Python39\lib\site-packages\poetry\utils\env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│
      1542│         return decode(output)
      1543│
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install C:/Users/USER/AppData/Local/pypoetry/Cache/artifacts/ef/c1/a0/4bf107db8d2e7ed8bc65b758fc37582e74a031fdf41a13e152e81a089a/atomicwrites-1.4.1.tar.gz

  at ~\AppData\Local\Programs\Python\Python39\lib\site-packages\poetry\utils\pip.py:58 in pip_install
       54│
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       59│

[dimbleby]

the request to control pip behaviour duplicates #7182, #3249, and probably others.

[rschneider98]

This is a duplicate, sorry I did not see those when looking through the issues board. I think I will leave our versions pinned to 1.2.2 until #6205 removes pip as the package installer or use the system configuration for PIP settings if the need to update occurs. Thank you.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.