create a fetchpypi function thats using blake2b for the crypto hash (python)

[RonnyPfannschmidt]

the modern pypi cdn hashes files using blake2b, which is also part of their url
so it would be very practical to reuse those hashes for file checking and full url building

[grahamc]

Unfortunately those hashes are checked by nix itself, and we'd need to add blake2b to nix.

[FRidh]

@RonnyPfannschmidt Do you have a link to documentation of how the hashes are computed?

[LnL7]

I have some nix expressions for creating python packages from a custom index, this uses pip with --require-hashes to handle the downloads. I could probably generalise this a bit to create a fetchpypi function that only needs a sha and pypi index.

[RonnyPfannschmidt]

im closing this one as unsustainable since sha256 are available as well as name based redirects

[FRidh]

As far as I know the name-based redirects don't work with wheels though, and I think we're going more in the direction of using wheels.

[RonnyPfannschmidt]

@FRidh wheels have urls like https://files.pythonhosted.org/packages/py2.py3/m/matlab_kernel/matlab_kernel-0.9.5-py2.py3-none-any.whl

[LnL7]

This is what I've been using to download python packages from a custom index https://gist.github.com/LnL7/e87c76d1bf4217dcdc8226bcec1e71c1.
Removing --no-binary :all: would make this use wheels by default.

[RonnyPfannschmidt]

@FRidh that being said, using wheels as source package implies killing the ability to run tests in many ways

but it could be valuable to set up a bootstrap tool-chain using downloaded wheels in the pythonpath

[FRidh]

@FRidh that being said, using wheels as source package implies killing the ability to run tests in many ways

Indeed, we would still have to fetch the tests elsewhere then. I'm not a big fan of it, but as it is likely setuptools will be used fewer in the future, developers will sometimes only upload wheels.

This is what I've been using to download python packages from a custom index https://gist.github.com/LnL7/e87c76d1bf4217dcdc8226bcec1e71c1.
Removing --no-binary :all: would make this use wheels by default.

That's a good idea, using pip as fetcher.

[LnL7]

I'll create a pr that integrates it.

[FRidh]

@LnL7 If we can just use fetchurl that would be even better #22256 (comment)

[LnL7]

That only works for pypi.python.org, I created the expression so I could use our internal pypi server at work.

[FRidh]

That I understand, having a generic function is nice to have. For use in Nixpkgs I would prefer we use a lighter-weight fetchurl though.

[LnL7]

Sure, perhaps we could swap out the implementation depending on if a custom index is used? It would be nice if I could use some of the existing packages in nixpkgs.

[LnL7]

Since fetchpypi and fetchurl are both fixed output drvs, the implementation should not matter for the output.

[RonnyPfannschmidt]

@LnL7 most pypi servers have some form of simple redirect as far as i recall, perhaps you can get by with a sinmple fetchurl wrapper as well

an pip using function would be still nice as building block for complex setups

[grahamc]

Ronny Pfannschmidt <[email protected]> writes:

im closing this one as unsustainable since sha256 are available as well as name based redirects

Also, name-based redirects I believe are intended as a stop-gap migration strategy.