Python: buildPyPIPackage and generated data #16005

FRidh · 2016-06-05T16:03:09Z

Motivation for this change

In #15007 I presented a script to generate metadata from PyPI and a function to build Python packages using that metadata.

In this PR the metadata is no longer stored in Nixpkgs but in an external repository, https://github.com/FRidh/srcs-pypi.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- OS X
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

These two functions wrap respectively buildPythonPackage and buildPythonApplication, but use by default a repository with metadata retrieved from PyPI.

FRidh · 2016-06-05T16:04:16Z

A the time of writing the generated JSON is 80 MB.

Example usage:

toolz = buildPyPIPackage {
  name = "toolz";
  buildInputs = with self; [ nose ];
  checkPhase = ''
    nosetests toolz/tests
  '';
};

By default it picks the latest version. Optionally a version is passed in. I suppose within Nixpkgs we would always be explicit about the versions.

garbas · 2016-06-06T10:59:57Z

💯 could we get that srcs-pypi repo into nixos organization? and change the name to nixpkgs-python? that would then be a good place to rethink how we do python packaging.

FRidh · 2016-06-06T11:08:20Z

pkgs/development/python-modules/generic/buildpypi.nix

+  # Use src if given. Otherwise, pick the right version via PyPI.
+  src = attrs.src or attrs.srcs or (fetchurl data.versions.${version});
+
+in buildPythonPackage ( attrs // {name = name + "-" + version; src=src; meta = pypimeta;} )


have to fix meta data appending here

FRidh · 2016-06-21T15:16:42Z

@garbas I could rename it, but moving the repo is not up to me. By the way, I think we should keep the discussion in the main repo, that is, this one. That repo is just going to contain generated data, nothing more.

FRidh · 2016-06-22T12:03:27Z

According to #16130 (comment) Hydra cannot build packages that use this because Hydra operates in restricted mode.

FRidh · 2016-06-28T18:03:15Z

I've tested whether a Hydra would build packages that use buildPyPIPackage and it indeed doesn't. That is a pity.

FRidh · 2016-07-03T12:19:09Z

By default it picks the latest version. Optionally a version is passed in. I suppose within Nixpkgs we would always be explicit about the versions.

The reason I mentioned this is because we have stable releases to consider.
PyPI also records and exposes the upload datetime of the archives. If we store that in our JSON, and we store in Nixpkgs the datetime of the stable release, then we can use that to determine the latest version. It will still be possible to change the version (e.g. for security updates) manually.

By not specifying explicit versions the nixpkgs repo will always remain up to date.

These two functions wrap respectively buildPythonPackage and buildPythonApplication, but use by default a repository with metadata retrieved from PyPI.

Mic92 · 2016-10-15T06:46:17Z

If we do not fix the version and upgrade implicitly, would not a lot of applications break all the time?

FRidh · 2016-10-15T08:36:37Z

In my experience with the Python packages we have in Nixpkgs, there's a handful of packages that we need to set the version of explicitly. Most others seem to work fine after (minor) upgrades. And if the update breaks some packages, I would consider that fine as well considering it is Nixpkgs unstable. The amount of time that could be saved would be enormous.

Mic92 · 2016-10-27T10:12:53Z

pkgs/development/interpreters/python/buildpypi.nix

+, ... } @ attrs:
+
+let
+  pypi-src = fetchgit {


Is fetchgit required here?
The good thing about your github structure is, that it is pretty readable.
We could generate a compressed version of your directory structure for downloading automatically via travis-ci.
Maybe putting all packages with the same prefix into the same file and compress the whole thing with xz to reduce the overall size.

https://docs.travis-ci.com/user/deployment/releases/

Is fetchgit required here?

Since GitHub can produce archives for a specific revision we could use fetchurl. But, where we really want to go to is fetchTarbal` along with a sha256. That way, Hydra can evaluate it. Without fetchTarball + sha256 Hydra cannot evaluate it. That's the reason why this is on hold. See also my latest mail.

Another option to reduce the size is to work with two repositories, one that has a JSON for each package, and another one that has package names along with hashes of the JSON files. This repo we would then refer to from Nixpkgs. It would replace downloading a big file with (possibly many) tiny ones.

Mic92 · 2016-10-27T10:16:32Z

pkgs/development/interpreters/python/buildpypi.nix

+  pypi = !(builtins.hasAttr "src" attrs || builtins.hasAttr "srcs" attrs);
+
+  # Use Meta data from PyPI
+  pypimeta = if pypi then data.meta else {};


This is also pretty cool.

Mic92 · 2016-10-27T10:37:21Z

Do you know, what nixexpr actually contains? Is it just a snapshot of nixpkgs? Would every nixos installation have to download your repo eventually or would be referenced pypi information already included in nixexpr?

FRidh · 2016-11-17T13:00:21Z

@Mic92 nixexpr is indeed a snapshot. Every NixOS installation would have to download this repo in order to evaluate Python packages.

FRidh · 2017-05-03T06:18:01Z

I think the way forward is an update script that updates pname, version, and sha256 in Nix expressions, so I am closing this.

FRidh added 6.topic: python 2.status: work-in-progress This PR isn't done labels Jun 5, 2016

FRidh mentioned this pull request Jun 5, 2016

Autogenerated data in Nixpkgs #15480

Open

FRidh added 0.kind: enhancement Add something new 9.needs: reporter feedback This issue needs the person who filed it to respond 9.needs: changelog 9.needs: documentation labels Jun 6, 2016

FRidh mentioned this pull request Jun 6, 2016

Python: autogenerate PyPI source data (2) #15007

Closed

FRidh reviewed Jun 6, 2016
View reviewed changes

FRidh mentioned this pull request Jun 22, 2016

Generate Hackage/LTS nix expressions on the fly rather than have them in the repo #16130

Closed

FRidh force-pushed the srcs-pypi branch from 64fafdb to d186083 Compare June 23, 2016 11:00

FRidh force-pushed the srcs-pypi branch from d186083 to 9f0c118 Compare June 30, 2016 17:12

FRidh mentioned this pull request Jul 7, 2016

Python: use fixed point combinator #16784

Closed

9 tasks

domenkozar mentioned this pull request Aug 13, 2016

Python ecosystem improvements (placeholder issue) #1819

Open

37 tasks

FRidh added the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 19, 2016

FRidh mentioned this pull request Oct 13, 2016

homeassistant: [WIP] init at 0.30.2 #19508

Closed

7 tasks

FRidh added 2 commits October 14, 2016 09:44

Python: buildPyPIPackage and buildPyPIApplication

a45685e

These two functions wrap respectively buildPythonPackage and buildPythonApplication, but use by default a repository with metadata retrieved from PyPI.

pythonPackages.toolz: use buildPyPIPackage

6159439

FRidh force-pushed the srcs-pypi branch from 9f0c118 to 6159439 Compare October 14, 2016 07:45

FRidh changed the base branch from master to staging October 14, 2016 07:45

FRidh removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 14, 2016

Mic92 approved these changes Oct 27, 2016

View reviewed changes

FRidh mentioned this pull request Nov 16, 2016

pythonPackages.fetchpypi: fetch tarballs from pypi #20481

Closed

7 tasks

Mic92 mentioned this pull request Jan 29, 2017

[wip] Proposal/fetchpypi generalized fetching from pypi #22257

Closed

7 tasks

FRidh closed this May 3, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: buildPyPIPackage and generated data #16005

Python: buildPyPIPackage and generated data #16005

FRidh commented Jun 5, 2016

FRidh commented Jun 5, 2016 •

edited

Loading

garbas commented Jun 6, 2016

FRidh Jun 6, 2016

FRidh commented Jun 21, 2016

FRidh commented Jun 22, 2016

FRidh commented Jun 28, 2016

FRidh commented Jul 3, 2016 •

edited

Loading

Mic92 commented Oct 15, 2016 •

edited

Loading

FRidh commented Oct 15, 2016

Mic92 Oct 27, 2016 •

edited

Loading

Mic92 Oct 27, 2016

FRidh Oct 27, 2016

Mic92 Oct 27, 2016

Mic92 commented Oct 27, 2016

FRidh commented Nov 17, 2016

FRidh commented May 3, 2017

Python: buildPyPIPackage and generated data #16005

Python: buildPyPIPackage and generated data #16005

Conversation

FRidh commented Jun 5, 2016

Motivation for this change

Things done

FRidh commented Jun 5, 2016 • edited Loading

garbas commented Jun 6, 2016

FRidh Jun 6, 2016

Choose a reason for hiding this comment

FRidh commented Jun 21, 2016

FRidh commented Jun 22, 2016

FRidh commented Jun 28, 2016

FRidh commented Jul 3, 2016 • edited Loading

Mic92 commented Oct 15, 2016 • edited Loading

FRidh commented Oct 15, 2016

Mic92 Oct 27, 2016 • edited Loading

Choose a reason for hiding this comment

Mic92 Oct 27, 2016

Choose a reason for hiding this comment

FRidh Oct 27, 2016

Choose a reason for hiding this comment

Mic92 Oct 27, 2016

Choose a reason for hiding this comment

Mic92 commented Oct 27, 2016

FRidh commented Nov 17, 2016

FRidh commented May 3, 2017

FRidh commented Jun 5, 2016 •

edited

Loading

FRidh commented Jul 3, 2016 •

edited

Loading

Mic92 commented Oct 15, 2016 •

edited

Loading

Mic92 Oct 27, 2016 •

edited

Loading