install --download fails if development library dependencies are missing

[akaihola]

I have a server where outbound HTTP traffic is intentionally prohibited. When deploying my web app, I'm downloading Python package dependencies on another host and rsyncing them up to the server.

If the dependencies include packages which have C extensions, and the required development libraries aren't present on the host, pip refuses to download any remaining packages. The --no-deps option has no effect on this behavior.

Downloading packages on a different machine is probably a common use case for --download. It would make sense for it to work even if development packages aren't installed on the dowloading machine.

As an example, let's assume a requirements.txt:

psycopg2
python-dateutil

If I now run:

$ pip install --download . -r requirements.txt

I get this:

Downloading/unpacking psycopg2 (from -r requirements.txt (line 1))
Saved ./psycopg2-2.4.5.tar.gz
Running setup.py egg_info for package psycopg2
  Error: You need to install postgresql-server-dev-X.Y for building a server-side extension or libpq-dev for building a client-side application.
  Complete output from command python setup.py egg_info:
  running egg_info
creating pip-egg-info/psycopg2.egg-info
writing pip-egg-info/psycopg2.egg-info/PKG-INFO
writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
Error: You need to install postgresql-server-dev-X.Y for building a server-side extension or libpq-dev for building a client-side application.
----------------------------------------
Command python setup.py egg_info failed with error code 1 in build/psycopg2
Storing complete log in /home/akaihola/.pip/pip.log

In this case, the psycopg2 package is downloaded, but python-dateutils is not:

$ ls
psycopg2-2.4.5.tar.gz  requirements.txt

[carljm]

The root problem here is that Python packaging does not yet have static metadata, and thus requires running setup.py to even acquire metadata about a package; to make sure it is actually the right project name and version, to find out what dependencies it has, etc. And projects like psycopg2 or scipy that have build dependencies are not careful to make it possible to get that metadata out of setup.py if you don't have all the build dependencies present on your system, because they primarily think of setup.py as intended for actual installation.

In general, it's intentional that pip runs egg_info first, and errors out quickly if that fails, so that if one of your packages is broken you don't waste time downloading all the rest of them. Arguably that's less ideal in the case of --download, so one workaround in pip would be to modify that behavior in the case of --download so that errors in egg_info are not fatal, and it continues with downloading the remainder.

Another possibility might be to avoid running egg_info entirely, but we can only do that in the specific case of --download --no-deps, because we have to run egg_info to get dependency information.

[pnasrat]

I've worked similar places with firewalling on prod servers. We used --download during the continuous integration phase, which was allowed to speak to the internet and published to an internal repository and used --download with a output dir then in prod we used --find-links and --index-url to pull locally.

[akaihola]

@carljm Thanks for the clarification! I reported this on the psycopg2 bug tracker, so let's see if they have an interest to fix setup.py egg_info in their library.

[dvarrazzo]

Hello from the psycopg2 bug tracker. Feel free to provide a patch. Thank you.

[akaihola]

I looked into this a bit. It looks like the problem isn't in the pip+psycopg2 combination, but setuptools+psycopg2 since pip uses setuptools under the hood.

I was able to reproduce the same error message by adding an

import setuptools

at the top of psycopg2/setup.py and running:

$ export DISTUTILS_DEBUG=1
$ python setup.py egg_info
options (after parsing config files):
options (after parsing command line):
option dict for 'aliases' command:
  {}
option dict for 'build_ext' command:
  {'define': ('setup.cfg',
              'PSYCOPG_EXTENSIONS,PSYCOPG_NEW_BOOLEAN,HAVE_PQFREEMEM'),
   'have_ssl': ('setup.cfg', '0'),
   'use_pydatetime': ('setup.cfg', '1')}
option dict for 'egg_info' command:
  {}
running egg_info
Distribution.get_command_obj(): creating 'egg_info' command object
writing psycopg2.egg-info/PKG-INFO
writing top-level names to psycopg2.egg-info/top_level.txt
writing dependency_links to psycopg2.egg-info/dependency_links.txt
Distribution.get_command_obj(): creating 'build_py' command object
Distribution.get_command_obj(): creating 'build' command object
Distribution.get_command_obj(): creating 'build_ext' command object
  setting options for 'psycopg_build_ext' command:
    use_pydatetime = 1 (from setup.cfg)
    have_ssl = 0 (from setup.cfg)
    define = PSYCOPG_EXTENSIONS,PSYCOPG_NEW_BOOLEAN,HAVE_PQFREEMEM (from setup.cfg)
Error: You need to install postgresql-server-dev-X.Y for building a server-side extension or libpq-dev for building a client-side application.

Any ideas where to go from here? Should I raise this on the Distribute bug tracker, or is Setuptools/Distribute doing the right thing when it always creates all command objects? Should the psycopg_build_ext command object instead disable extension checks in this case? Or is this going to be difficult to solve because psycopg2 uses distutils and not setuptools?

[dstufft]

I'm going to close this, the issue is going to be with the psycogp2 library which requires libpq to be intalled to run the setup.py at all.