Hashes are not being produced for private packages when exporting a requirements file.

[jamessmith-decisionlab]

Hi,

no hash is being created in the requirements.txt file when I run poetry export -f requirements.txt. This means pip fails when it reads the file. It's fine, pip actually gives me the hash to insert into the file, however it would be great if Poetry did it in the first place.

Also, what might be good is to have something like a --no-hashes argument for exporting, so that no hashes are added to the requirements file. My understanding is that if pip can't find any hashes at all, it won't complain about missing ones.

[petergaultney]

I am also experiencing this.

The issue is related to python-poetry/poetry#1631 - what's happening under the hood, I think, is that your poetry.lock file is using an MD5 hash rather than SHA256, and since pip would complain about an md5 anyway, the exporter is correctly not outputting it. In other words, the bug is farther upstream, at the time of lockfile generation. If poetry would capture the SHA256 in the lockfile, everything would work fine at export time.

[jamessmith-decisionlab]

In fact there are no hashes corresponding to the private package at all in the lock file. So I guess the bug is related to the lock file, but not quite how you envisaged it.

requests = ["11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"]
xxx-schema = []
rx = ["84ac6e140f2129266fd4c77a3d0903bbff0a761386079ea0cb99c8524695cac1", "ad793d79843feba2ea25c0c01be245f3163d5d469418279d4019737b93d88c78"]

It's been flagged as a bug so let's see what happens.

[petergaultney]

Ah, that's interesting. I should have said "I would guess" this is related to python-poetry/poetry#1631. :)

[adawalli]

I see this same behavior when using artifactory, in case the actual private repository matters - I would imagine it shouldn't though, since the "simple" api is being used.

[adawalli]

Also, what might be good is to have something like a --no-hashes argument for exporting, so that no hashes are added to the requirements file. My understanding is that if pip can't find any hashes at all, it won't complain about missing ones.

Perhaps I am not understanding, but this option is already supported directly in poetry (at least on the new RC's.
--without-hashes Exclude hashes from the exported file.

[jamessmith-decisionlab]

Hi,

apologies for leaving it so long before replying, firstly. One thing drives out another, I'm sure you know how it is!

The --without-hashes tip works fine, thank you very much.

I guess we should leave the issue open, however, because ideally the lock file should contain the hash(es) for the private repository?

[filipelbc]

I'm having a similar issue when using url.

pyproject.toml:

django-requestlogging = {url = "https://bitbucket.org/miohtama/django-requestlogging/get/fc515e3b3029.zip"}

poetry.lock (no hashes stored):

django-requestlogging = []

Exported requirements.txt (no hashes, wrong -e):

-e https://bitbucket.org/miohtama/django-requestlogging/get/fc515e3b3029.zip

Expected entry in requirements.txt:

https://bitbucket.org/miohtama/django-requestlogging/get/fc515e3b3029.zip \
    --hash=sha256:eef6342cc77af842be3efb6ece073310b49ce7cb65256441527f1aac7686c01b

As you can see, this is also an instance of python-poetry/poetry#897

[kroeschl]

For anyone else still hitting this issue with internal repositories that only provide MD5 hashes, the workaround I used (from this comment) was:

poetry export -f requirements.txt -o requirements.txt
pip-compile --generate-hashes -o requirements.txt requirements.txt

This generates SHA256 hashes for packages which don't already have them. Note that this is strictly worse than using just poetry because it drops any platform-specific dependencies that apply to a platform other than what you're using to run pip-compile.

[neersighted]

python-poetry/poetry#4740 should resolve this once I get it ready for review. The code should be 100% working (please test the branch and make sure it generates hashes for your MD5-only legacy repos. I'm still working on full test coverage before it's ready for merge.).

[kroeschl]

python-poetry/poetry#4740 should resolve this once I get it ready for review. The code should be 100% working (please test the branch and make sure it generates hashes for your MD5-only legacy repos. I'm still working on full test coverage before it's ready for merge.

I grabbed the branch from the linked PR and still wasn't able to poetry export and get MD5 hashes.

$ pip install git+https://github.com/python-poetry/poetry.git@a0dec5802b67035bf683003b9466ff7852314472
$ poetry lock --no-update
$ cat poetry.lock
...
depwalk = [
    {file = "depwalk-0.8.21-py3-none-any.whl", hash = "md5:7dd5aa75718f5db708a917bee146f9e9"},
]
...
$ poetry export -f requirements.txt -o ./foo/requirements.txt
$ cat ./foo/requirements.txt
...
    --hash=sha256:e3a62f0520172440ca0dcc823749319382e377f37f140a0b99ef45fecb84bfe7
depwalk==0.8.21 ; python_version >= "3.6" and python_version < "4.0"
distlib==0.3.3 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" \
...

[neersighted]

python-poetry/poetry#4740 should resolve this once I get it ready for review. The code should be 100% working (please test the branch and make sure it generates hashes for your MD5-only legacy repos. I'm still working on full test coverage before it's ready for merge.

I grabbed the branch from the linked PR and still wasn't able to poetry export and get MD5 hashes.

$ pip install git+https://github.com/python-poetry/poetry.git@a0dec5802b67035bf683003b9466ff7852314472
$ poetry lock --no-update
$ cat poetry.lock
...
depwalk = [
    {file = "depwalk-0.8.21-py3-none-any.whl", hash = "md5:7dd5aa75718f5db708a917bee146f9e9"},
]
...
$ poetry export -f requirements.txt -o ./foo/requirements.txt
$ cat ./foo/requirements.txt
...
    --hash=sha256:e3a62f0520172440ca0dcc823749319382e377f37f140a0b99ef45fecb84bfe7
depwalk==0.8.21 ; python_version >= "3.6" and python_version < "4.0"
distlib==0.3.3 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" \
...

Sorry if I was unclear -- poetry should be generating sha256 hashes for packages that don't have a sha256 hash pre-generated for them by the repo. It shouldn't ever pass through a md5 hash.

It looks like you're still not getting a hash for depwalk -- I'll have to take a closer look -- it may be that the exporter is not taking advantage of the list of hashes built by the downloader.

[dimbleby]

Not sure this wanted transferring - isn't the conclusion of the above thread that

poetry should be generating sha256 hashes for packages that don't have a sha256 hash pre-generated for them by the repo. It shouldn't ever pass through a md5 hash.

ie the export plugin will just export whatever's in the lock file, poetry proper ought to have made sure that there's a sha256 hash in that lock file

[dimbleby]

Having said which it's also true that the exporter has

poetry-plugin-export/src/poetry_plugin_export/exporter.py

Line 30 in dd3b674

ALLOWED_HASH_ALGORITHMS = ("sha256", "sha384", "sha512")

so if the intention is to put md5 hashes in the lockfile then that will need updating for the plugin to export them.

[dimbleby]

but that last bit duplicates python-poetry/poetry#20