pip-compile ignores existing output file if --output-file is specified

[mikepqr]

What's the problem this feature will solve?

This may be a supported workflow, in which case I'd be happy to submit a PR to add it to the README if someone can help me understand what the intended steps are!

Suppose I have:

# requirements.in
foo==1.0
bar==2.0

foo depends on baz>=3.0 and bar depends on qux>=4.0 so that when I run pip-compile requirements.in I get:

# requirements.txt
foo==1.0  # via requirements.in
bar==2.0  # via requirements.in
baz==3.0  # via foo
qux==4.0  # via bar

But then there's a CVE against bar==3.0 and I want to upgrade it to 3.1. So I run pip-compile --upgrade-package baz==3.1, which yields:

# requirements.txt
foo==1.0  # via requirements.in
bar==2.0  # via requirements.in
baz==3.1  # via foo
qux==4.2  # via baz

baz has been upgraded as requested, but foo and bar remain pinned as in the requirements file. That's all great. But qux has been upgraded too, because bar follows best practices by not overpinning its qux dependency, and qux happens to have released a new version. This is not what I want in this case. I want to upgrade only baz.

Describe the solution you'd like

I guess I'm asking for an --upgrade-only-package option that does what I want, but this seems like such a common use case that I assume I'm missing something!

Alternative Solutions

Right now I simply edit the baz line in requirements.txt by hand in cases where it doesn't contain hashes.

When requirements.txt does contain hashes I do an insane set of manual steps that involves:

1. Copying requirements.txt over requirements.in
2. Deleting all the hashes from the file.
3. Pinning baz==3.1 in the new requirements.in
4. Rerunning pip-compile to generate the hashed requirements.txt file for this fully pinned set of requirements.
5. Restore the old requirements.in file

[AndydeCleyre]

Thanks for the report!

I think you mixed up bar and baz a number of times here, but I'm following.

But the behavior you want is what I usually see. Can you provide a testable example?

reqs.in:

httpx

$ pip-compile --no-header reqs.in

reqs.txt:

certifi==2020.12.5
    # via httpx
h11==0.12.0
    # via httpcore
httpcore==0.12.3
    # via httpx
httpx==0.17.1
    # via -r reqs.in
idna==3.1
    # via rfc3986
rfc3986[idna2008]==1.4.0
    # via httpx
sniffio==1.2.0
    # via
    #   httpcore
    #   httpx

$ $EDITOR reqs.in

reqs.in:

httpx<0.17.1
sniffio<1.2.0

$ pip-compile --no-header reqs.in

reqs.txt:

certifi==2020.12.5
    # via httpx
h11==0.12.0
    # via httpcore
httpcore==0.12.3
    # via httpx
httpx==0.17.0
    # via -r reqs.in
idna==3.1
    # via rfc3986
rfc3986[idna2008]==1.4.0
    # via httpx
sniffio==1.1.0
    # via
    #   -r reqs.in
    #   httpcore
    #   httpx

$ echo httpx >reqs.in
$ pip-compile --no-header -P sniffio reqs.in

reqs.txt:

certifi==2020.12.5
    # via httpx
h11==0.12.0
    # via httpcore
httpcore==0.12.3
    # via httpx
httpx==0.17.0
    # via -r reqs.in
idna==3.1
    # via rfc3986
rfc3986[idna2008]==1.4.0
    # via httpx
sniffio==1.2.0
    # via
    #   httpcore
    #   httpx

Are you sure your package requirements are as loose as you expect?

[AndydeCleyre]

In your workaround steps at the end of your report, you shouldn't need to do numbers 1, 2, or 5.

[mikepqr]

Apologies for the extremely buggy original post. I think I've tracked down some of the errors, but please let me know if it needs more tidying to make sense for posterity (or feel free to edit if you have permissions!)

Thank you very much for 1) confirming the expected behavior 2) providing a minimal example. That's extremely helpful, and has allowed me to narrow it down to behavior caused by --output-file.

If I run your example I get the same results. But if I change the last command to specify --output-file then httpx is upgraded:

$ pip-compile --no-header -P sniffio reqs.in --output-file reqs0.txt
certifi==2020.12.5
    # via httpx
h11==0.12.0
    # via httpcore
httpcore==0.12.3
    # via httpx
httpx==0.17.1
    # via -r reqs.in
idna==3.1
    # via rfc3986
rfc3986[idna2008]==1.4.0
    # via httpx
sniffio==1.2.0
    # via
    #   httpcore
    #   httpx

Is this a bug? Or is --output-file intentionally synonymous with "don't read the existing output file, if it exists". Is there any way to avoid this behavior and get the more usual behavior while retaining use of --output-file?

[mikepqr]

I think I'm generally pretty confused about the role of any existing requirements.txt file when compiling a requirements.in. There may or may not be bugs here, but it's not obvious from the docs IMO.

[atugushev]

Is this a bug? Or is --output-file intentionally synonymous with "don't read the existing output file, if it exists". Is there any way to avoid this behavior and get the more usual behavior while retaining use of --output-file?

Via --output-file option pip-tools finds a lock file (requirements.txt). If you pass a different file - it would generate a new lock-file and ignore pinnings from the previous one. It's not a bug, this is how pip-tools works.

[mikepqr]

Ah, got it.

Is there any way to create a lockfile, while respecting the contents of an existing lockfile, without overwriting that existing lockfile? pip-compile requirements.txt -P foo --output-file requirements0.txt doesn't work, in the sense that foo does not get upgraded.

[AndydeCleyre]

@mikepqr

Can you provide a testable example?

[mikepqr]

Sure. Starting from this in reqs.in:

httpx

and this in reqs.txt

certifi==2020.12.5
    # via httpx
h11==0.12.0
    # via httpcore
httpcore==0.12.3
    # via httpx
httpx==0.17.1
    # via -r reqs.in
idna==3.1
    # via rfc3986
rfc3986[idna2008]==1.4.0
    # via httpx
sniffio==1.2.0
    # via
    #   httpcore
    #   httpx

This command does what I want (i.e. upgrades sniffio, and leaves everything else unchanged):

pip-compile reqs.in -P sniffio

This command also does what I want:

pip-compile reqs.in -P sniffio --output-file reqs.txt

But they both overwrite the existing lockfile.

This command writes a reqs0.txt in which everything has been upgraded (which is httpx and sniffio at the time of writing):

pip-compile reqs.in -P sniffio --output-file reqs0.txt

And this command upgrades nothing (but changes the via lines to say -r reqs.txt)

pip-compile reqs.txt -P sniffio --output-file reqs0.txt

So, my question is: is there any way to create a lockfile, while respecting the contents of an existing lockfile, without overwriting that existing lockfile?

I don't think it's relevant, but just in case you think I'm being pedantic, here's why I want to do this. My requirements are mounted into a docker container. If I overwrite inside the container they are then owned by root on the docker host. I can solve this but it's going to involve a bunch of fiddly docker cp, so I'm keen to have an --output-file. For this to work, it seems like I need some way to tell pip-compile that there is an existing lockfile, but it's not at --output-file.

[AndydeCleyre]

So, my question is: is there any way to create a lockfile, while respecting the contents of an existing lockfile, without overwriting that existing lockfile?

Yes, the simple solution is copy the lockfile you'd like to use as a base to the destination, before running pip-compile:

$ cp reqs.txt reqs0.txt
$ pip-compile reqs.in -o reqs0.txt

[mikepqr]

Ha! I've been staring at this problem for so long, and never really felt like I understood what --output-file did, so that never occurred to me. Perfect, thanks!

I do find --output-file's behavior (or more generally the role of the existing requirements.txt file) non-obvious, and I wonder if it's worth spelling out in the docs that it is not so much a pure output file, but rather the lockfile which will be used as input if it exists?

[atugushev]

I wonder if it's worth spelling out in the docs that it is not so much a pure output file, but rather the lockfile which will be used as input if it exists?

Feel free to open a PR 👍

[mikepqr]

Happy to! Before I get started, is there any scope for renaming the option (and changing all references in UI strings) to --lock-file, "lock file", etc. (while maintaining --output-file for backwards compatibility)?

[atugushev]

is there any scope for renaming the option (and changing all references in UI strings) to --lock-file, "lock file", etc. (while maintaining --output-file for backwards compatibility)?

Personally, I'm -1 on renaming, but feel free to open a discussion in a separate issue.

I'll close this since #1369 clears how pip-tools works with output files and it doesn't look like there's any other action required regarding pip-tools. Thanks for the issue!