Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: buildPyPIPackage and generated data #16005

Closed
wants to merge 2 commits into from
Closed

Python: buildPyPIPackage and generated data #16005

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a45685e
Python: buildPyPIPackage and buildPyPIApplication
FRidh Apr 26, 2016
6159439
pythonPackages.toolz: use buildPyPIPackage
FRidh Jun 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions pkgs/development/interpreters/python/buildpypi.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/* This function wraps buildPythonPackage and uses pre-fetched data from PyPI to build a package.
*/
{ buildPythonPackage
, fetchurl
, fetchgit
, lib
}:


{ name
, version ? ""
, ... } @ attrs:

let
pypi-src = fetchgit {
Copy link
Member

@Mic92 Mic92 Oct 27, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is fetchgit required here?
The good thing about your github structure is, that it is pretty readable.
We could generate a compressed version of your directory structure for downloading automatically via travis-ci.
Maybe putting all packages with the same prefix into the same file and compress the whole thing with xz to reduce the overall size.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.travis-ci.com/user/deployment/releases/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is fetchgit required here?

Since GitHub can produce archives for a specific revision we could use fetchurl. But, where we really want to go to is fetchTarbal` along with a sha256. That way, Hydra can evaluate it. Without fetchTarball + sha256 Hydra cannot evaluate it. That's the reason why this is on hold. See also my latest mail.

Another option to reduce the size is to work with two repositories, one that has a JSON for each package, and another one that has package names along with hashes of the JSON files. This repo we would then refer to from Nixpkgs. It would replace downloading a big file with (possibly many) tiny ones.

url = "https://github.com/FRidh/srcs-pypi.git";
rev = "12bc2a4cbcec42a867e4cc2fdef67e76b2c0b169";
sha256 = "0j3fhfnj36wsm6z3r0mnsr7c90vihi1j61nwdz31dz4y2r0w2vg4";
};

data = import (pypi-src) {inherit name;};

# Whether to actually use PyPI
pypi = !(builtins.hasAttr "src" attrs || builtins.hasAttr "srcs" attrs);

# Use Meta data from PyPI
pypimeta = if pypi then data.meta else {};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also pretty cool.


# Version. When specified, use that.
# Otherwise, if src is not given and there is a PyPI source available, then use the latest version available.
version = attrs.version or (if pypi then data.latest_version else "");
# Use src if given. Otherwise, pick the right version via PyPI.
src = attrs.src or attrs.srcs or (fetchurl data.versions.${version});

in buildPythonPackage ( attrs // {name = name + "-" + version; src=src; meta = pypimeta;} )
18 changes: 10 additions & 8 deletions pkgs/top-level/python-packages.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@ let

buildPythonApplication = args: buildPythonPackage ({namePrefix="";} // args );

buildPyPIPackage = makeOverridable (callPackage ../development/interpreters/python/buildpypi.nix {
buildPythonPackage = buildPythonPackage;
});

buildPyPIApplication = makeOverridable (callPackage ../development/interpreters/python/buildpypi.nix {
buildPythonPackage = buildPythonApplication;
});

in {

inherit python bootstrapped-pip isPy26 isPy27 isPy33 isPy34 isPy35 isPy36 isPyPy isPy3k mkPythonDerivation buildPythonPackage buildPythonApplication;
Expand Down Expand Up @@ -24745,14 +24753,8 @@ in {
};
};

toolz = buildPythonPackage rec{
name = "toolz-${version}";
version = "0.8.0";

src = pkgs.fetchurl{
url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
sha256 = "e8451af61face57b7c5d09e71c0d27b8005f001ead56e9fdf470417e5cc6d479";
};
toolz = buildPyPIPackage rec{
name = "toolz";

buildInputs = with self; [ nose ];

Expand Down