pip-compile: handle zip/tar.gz requirements

[blueyed]

I am getting the following error when trying to compile a requirements file, which contains a tar.gz entry:

Traceback (most recent call last):
  File "venv/bin/pip-compile", line 8, in <module>
    execfile(__file__)
  File "/home/user/Vcs/pip-tools/bin/pip-compile", line 182, in <module>
    main()
  File "/home/user/Vcs/pip-tools/bin/pip-compile", line 173, in main
    compile_specs(src_files, include_sources=args.include_sources, dry_run=args.dry_run)
  File "/home/user/Vcs/pip-tools/bin/pip-compile", line 102, in compile_specs
    top_level_specs = list(collect_source_specs(source_files))
  File "/home/user/Vcs/pip-tools/bin/pip-compile", line 96, in collect_source_specs
    for spec in walk_specfile(filename):
  File "/home/user/Vcs/pip-tools/bin/pip-compile", line 87, in walk_specfile
    spec = Spec.from_line(line, source='{0}:{1}'.format(filename, lineno))
  File "/home/user/Vcs/pip-tools/piptools/datastructures.py", line 122, in from_line
    req = Requirement.parse(line)
  File "venv/python2.7/site-packages/pkg_resources.py", line 2914, in parse
    reqs = list(parse_requirements(s))
  File "venv/python2.7/site-packages/pkg_resources.py", line 2839, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "venv/python2.7/site-packages/pkg_resources.py", line 2807, in scan_list
    raise ValueError("Expected "+item_name+" in",line,"at",line[p:])
ValueError: ('Expected version spec in', 'https://github.com/blueyed/envdir/archive/0.6.1.1.zip', 'at', '://github.com/blueyed/envdir/archive/0.6.1.1.zip')

[blueyed]

This happens for normal editables, like -e /home/user/src/django-master, too.

[stefanfoulis]

@blueyed normal editables seem to work now.
I'm curious: is this a hard feature to implement?

[stefanfoulis]

In pip-tools==1.1.4 the error message is now:

pip-compile does not support URLs as packages, unless they are editable. Perhaps add -e option?

[blueyed]

@stefanfoulis
Can you try if #169 fixes this one, too?

[stefanfoulis]

@blueyed I tested with #169

When I put https://example.com/my-package-0.1.tar.gz in requirements.in:

pip-compile requirements.in
pip-compile does not support artifact URLs as packages (but you can use VCS URLs). Constraint: https://example.com/my-package-0.1.tar.gz (from -r requirements.in (line 1)).

When I put -e https://example.com/my-package-0.1.tar.gz in requirements.in:

pip-compile requirements.in
Traceback (most recent call last):
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/bin/pip-compile", line 9, in <module>
    load_entry_point('pip-tools==1.0dev1', 'console_scripts', 'pip-compile')()
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/Users/stefanfoulis/tmp/pip-tools/pip-tools-169/piptools/scripts/compile.py", line 76, in cli
    for line in parse_requirements(src_file, finder=repository.finder, session=repository.session):
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/pip/req/req_file.py", line 89, in parse_requirements
    for req in req_iter:
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/pip/req/req_file.py", line 147, in process_line
    wheel_cache=wheel_cache
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/pip/req/req_install.py", line 118, in from_editable
    editable_req, default_vcs)
  File "/Users/stefanfoulis/.virtualenvs/tmp-pip-tools-169/lib/python2.7/site-packages/pip/req/req_install.py", line 1113, in parse_editable
    editable_req
pip.exceptions.InstallationError: https://example.com/my-package-0.1.tar.gz should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

[davidfraser]

I have attached my extremely awkward wrapper of piptools/scripts/compile.py.
This will find a comment with # fork: original_spec url
It will then run pip-compile with original_spec, and then substitute the url into the resulting output
This works for us but was a horrible workaround :)

pip_compile_with_forks.txt

[davidovich]

Is this still something relevant in the latest release (1.8.1 as I write this) and/or workflow ?

[ashwinvis]

This issue should be closed. Specifying the path to tarball as following works

-f  https://example.com/my-package-0.1.tar.gz

[blueyed]

@ashwinvis

It does not work with -f https://github.com/lock8/django-rest-framework/archive/d9ee7d68178a6b50b55caacdb50a531b2cc0eaf6.tar.gz#egg=djangorestframework for me: no error and no output..?!

pip-compile 3.1.0, pip 18.1.

[blueyed]

AFAIK -f refers to --find-links, i.e. it allows to use a different index, doesn't it?

[ashwinvis]

Just end the link with .tar.gz. I managed to get it working with:

-f https://software.ecmwf.int/wiki/download/attachments/56664858/ecmwf-api-client-python.tgz 

[blueyed]

@ashwinvis
What is your output with -v?
Given your example I only get:

Using indexes:
  https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file test.txt test.in
#

[ashwinvis]

@blueyed You are right. It ignores the line. I had several packages in the requirements.in, and when I saw no errors I assumed it installed ecmwf-api-client. It did not :-/

[cjerdonek]

As a simpler case, have you tried a zip / tar.gz with a local file while using the file scheme (and with no flags)? For example:file:///<abs_path>/0.6.1.1.zip#egg=<PackageName>

[jcushman]

Fixed by #807