Build passthru.tests automatically along with the package

[Ekleog]

I often have to manually call a build nixosTests.<foo> when checking a PR. I'm slowly adding them to passthru.tests, and since nix-community/nixpkgs-update#118 r-ryantm is taking them into account, but it'd be great if ofborg also automatically checked the derivations listed in passthru.tests

Building <package>.tests or {} in addition to <package> should (hopefully) be enough, and not be too hard to implement. This would let the maintainers responsible for determining which tests must pass to consider a change to a package valid.

What do you think about this?

[FRidh]

@grahamc we were talking earlier about these package tests. I think it would be a good idea.

I will add some tests that just do some assertions, e.g. ensuring mercurial-tortoisehg have the same version.

@Ekleog not sure though whether we should point to NixOS inside the package set...

[Ekleog]

@FRidh It's been discussed at a few places, first and foremost here and around there, for the two that stand out by memory. To sum up my current thoughts, probably missing arguments that have been raised before:

As someone who is (slowly) writing another nix-based OS (trying out some other paradigms), I also pretty much want to decouple Nixpkgs and NixOS. OTOH, we just do not have a better way to test things for the time being. passthru.tests's semantics is just “if this derivation builds then the test passed”, so it's not in itself tied to NixOS. That said, with the fate that NixOS/nixpkgs#36842 had, I don't have much hope for NixOS-less complete integration tests for most packages in the short-medium-term.

So IMO pointing to NixOS tests from passthru.tests (which is not a strict requirement for building derivations -- and BTW is just a derivation, in the end, that just by implementation detail uses NixOS as a test library) is the least bad path for the time being.

[FRidh]

Right, thanks, I recall now. Since everything goes through that one attribute it should be fine for now.

[7c6f434c]

(as someone who complains about NixOS tests being expensive to change and hard to use directly for my own environment) indeed, the NixOS tests someone already maintains should be used in any corner case they can be used to get a payoff on the effort. They do run fine on non-NixOS Nix, and Nixpkgs has a few NixOS-oriented places anyway (like unix_chkpwd or opengl-driver references), so it's not the biggest problem.

[turion]

It seems that currently these tests are not built anymore? See this PR where I have a package which fails to build, which is also a test case for another package which succeeds.

[Ekleog]

If I look at the logs, I see this:

Cannot nix-instantiate `agda.passthru.tests' because:
error: attribute 'tests' in selection path 'agda.passthru.tests' not found

warning: ignoring the user-specified setting 'restrict-eval', because it is a restricted setting and you are not a trusted user
/nix/store/3dxr0qk0qii2wz5zrwmhky2fgdrsbf9q-agdaWithPackages-2.6.1.2

So… my guess would be that for some reason passthru.tests doesn't actually get added?

[Ekleog]

Oh maybe I just understood: @ofborg build foo doesn't also build foo.passthru.tests, you (currently) have to manually specify it in this case by doing @ofborg build foo.passthru.tests — and this issue was about the default-run builds.

I don't think anyone wanted @ofborg build foo to automatically run passthru tests too yet. Maybe this is something that'd may make sense? But I don't think it's implemented yet either way :/

[turion]

@Ekleog Yes, that works! Thanks! (See NixOS/nixpkgs#98214 (comment))