Support [experimental] flakes

[philipp-baumann]

just to list; somewhen in future when we are bored and done with backbone of {rix}

[philipp-baumann]

@b-rodrigues I am currently exploring devbox get some more recent nix package versions for specific software (e.g. need to have up to date quarto for certain extensions, which really quick to get via this approach.

I really like this in combination with nixhub.io
https://www.jetpack.io/devbox/docs/quickstart/

https://www.jetpack.io/blog/using-nix-flakes-with-devbox/

[b-rodrigues]

There's also this: https://devenv.sh/

several things that we could explore!

[Kupac]

When I started my Nix journey, I decided to follow the flake path from the beginning. So naturally, when I found out about rix, I immediately started thinking how to have a flake-enabled version of it.

It could be an option with flake = FALSE by default.

I can see two ways to make this happen.

1. create a basic flake.nix, which would import default.nix. That's the most straight forward implementation. Currently, this is not possible, because fetchTarball doesn't specify a hash, and this is required by flake, to be able to create a lock file. I think it would be good practice anyway to not only pass a url, but also a hash. (example here)

2. The other option is to create a nice, editable flake.nix file, or even a multifile structure, which is more readable and editable. Of course, the functions used to build default.nix should be re-used as much as possible. This could reduce the burden of having to maintain a no-flake and a flake output of rix.

Which way would you prefer?

[b-rodrigues]

The thing I'm still wondering is what use case flakes solve/make easier than the current setup? I'm not necessarily against support flakes per se, but I'd like to know what I'm missing. I'm not necessarily convinced that flakes are a useful abstraction for development envs. What am I missing?

[Kupac]

Here's a summary in case you haven't read it: https://www.tweag.io/blog/2020-05-25-flakes/#what-problems-do-flakes-solve

For example, we could define an environment with whatever packages, and ship all the IDEs in a single flake. You could choose which one is the default. We could also ship docker container artifacts, which one could build in case you want to share your env with non nix users. There are many possibilities

[Kupac]

I started a nix project like that, but rix is a better approach for R users, who don't know nix yet.

[b-rodrigues]

What do you think would be the impact on the current codebase of supporting flakes? Seems like option 2 would be the best, and also still have minimal impact?

[Kupac]

I'm not familiar enough with the code base to be able to fully grasp the impact. But I think we can work with the current functions, and perhaps write a few more. Probably we'll need a dependency on gitr, because flakes only see files that are tracked by git. I also wonder whether it would make sense to wrap some of the functionality of nix into R functions. I'm thinking nix flake update nix flake init, nix show-config etc. This would add a system dependency on nix.

I think of creating a flake template somewhere in inst/. That template would import a few local nix files, genereated by rix, which contain the packages in the environment. Alternatively, we can generate a single flake.nix file from scratch, like we do for default.nix.

[b-rodrigues]

Composing environments does sound appealing , especially having one for packages and another for IDEs 🤔

Feel free to hack it together 😁

[philipp-baumann]

I'm not familiar enough with the code base to be able to fully grasp the impact. But I think we can work with the current functions, and perhaps write a few more. Probably we'll need a dependency on gitr, because flakes only see files that are tracked by git. I also wonder whether it would make sense to wrap some of the functionality of nix into R functions. I'm thinking nix flake update nix flake init, nix show-config etc. This would add a system dependency on nix.

I think of creating a flake template somewhere in inst/. That template would import a few local nix files, genereated by rix, which contain the packages in the environment. Alternatively, we can generate a single flake.nix file from scratch, like we do for default.nix.

I am in favor of option 2 of implementing it. We can get both, separation of concerns but also reuse the existing function stack for boilerplating. @Kupac I'm sure there is still parts in the traditional way that we can function up more efficiently for both default channels and flake ways. We can follow the approach we used for nix-build in nix_build() also for nix flake ... commands, invoke them via sys::exec_background() (async, non-blocking) and sys::exec_internal (blocking) like here: https://github.com/b-rodrigues/rix/blob/fe4d98eddd5f190d97ee9a7821d8d49750eff2a0/R/nix_build.R#L89-L93

We should definitively default for channels, but why not add flakes for those who want/need them. Regarding the need of gitr: does look quite nice, but I'd be hesistant adding another strict dependency to the list because it is a reproducibility package. We could also use requireNamespace() and check like that if it is installed. If its just git add, commit, and push we needs it's easy to just follow a system2() or sys:: wrapper approach, like gitr does, so just a few lines of code more to add to the package.

[Kupac]

FYI, I started to read the code base, and think about how to actually do this flake thing. Thanks for the pointers above, I agree 100%, and there are some good ideas in there.

[dwinkler1]

I played around with flakes a bit and created a repo that automatically generates flake templates based on the available dates and versions:

https://github.com/dwinkler1/rix-flakes

For now the flakes are all based off of my own preference shown here with just the R version changed: https://github.com/dwinkler1/rix-flakes/tree/main/templates/radian_vscode

It should be possible to use similar templates and instead of declaring the R packages in the templates like here:

https://github.com/dwinkler1/rix-flakes/blob/3ac0391184950b1567b941436541009146d2103a/templates/radian_vscode/flake.nix#L11

rix could just generate the package list and paste that into a rpackages.nix file that is imported. This what I do in my home config:

  default-r-packages-cran = import ./r-packages.nix rpkgs.rPackages;

Where r-packages.nix is:

p: with p; [
  MatchIt
  WeightIt
  Amelia
  arrow
  brms
...
]

[Kupac]

Nice work! I like the idea of simply listing packages and providing those to various envs: radian/rstudio, etc.

Before doing that, we also need to override the R package tree (rpkgs.rPackages) with:

• Packages with fixed versions
• Packages from external sources (git, local, etc)

And import the package list afterwards. I think the rix code has most of this functionality in place. You think we could easily merge your templates in there? BTW, I have a similar (but less mature) effort at: https://codeberg.org/kupac/reproducibR

We could work together to add flake support to rix.

[dwinkler1]

Sounds good!

Looking at https://docs.ropensci.org/rix/articles/d1-installing-r-packages-in-a-nix-environment.html#installing-old-packages-archived-on-cran and https://docs.ropensci.org/rix/articles/z-advanced-topic-handling-packages-with-remote-dependencies.html it seems like we can just use whatever rix is already generating. I have this pretty much (except for the specific package versions) set up manually in my home config:

  default-r-packages-cran = import ./r-packages.nix rpkgs.rPackages;
  default-r-packages-git = import ./rgit-packages.nix rpkgs;
  default-r-packages = default-r-packages-cran ++ default-r-packages-git;
  r-with-pkgs = rpkgs.rWrapper.override { packages = default-r-packages; };
  radian-with-pkgs = rpkgs.radianWrapper.override { packages = default-r-packages; };
  rstudio-with-pkgs = rpkgs.rstudioWrapper.override { packages = default-r-packages; };

We just need to figure out where to hook into rix to generate the statements and dump them into a package file. I'd say it's probably nicer to combine r-packages and rgit-packages.nix. This looks like a promising place to start: https://github.com/ropensci/rix/blob/main/R/fetchers.R

rgit-packages.nix is just

pkgs: with pkgs; [
        (rPackages.buildRPackage {
          name = "synthdid";
          src = fetchgit {
          url = "https://github.com/synth-inference/synthdid/";
          branchName = "master";
          rev = "70c1ce3eac58e28c30b67435ca377bb48baa9b8a";
          sha256 = "sha256-rxQqnpKWy4d9ZykRxfILu1lyT6Z3x++gFdC3sbm++pk=";
          };
          propagatedBuildInputs = [
                        rPackages.mvtnorm
            ];
          })
...
          ]

We can either merge the flakes here or fork a separate repo under ropensci if that is wanted. I personally like the separation of the automated repo for flakes but open to anything.

[b-rodrigues]

I haven't looked into flakes yet (I've been using Nix for two years now, shame on me lmao) but I am not against adding support for flakes in rix. But I would need commitment from the both of you to maintain this as I wouldn't be able to.

[dwinkler1]

I updated the generated flakes so that we can use the existing default.nix with 3 lines changed e.g., with sed (but this can probably be done just before writing default.nix in R:

#! /bin/sh

sed -i '1s/^/pkgs:\n/;/^\s*pkgs = import (fetchTarball/d;s/^\s*pkgs.mkShell//' default.nix

This generates a function of pkgs which returns an attrSet that is imported and used in the flakes.

See new default: https://github.com/dwinkler1/rix-flakes/blob/14c771359fe4ca697fa0c22147ae5127e6ca4afe/templates/default/flake.nix

[Kupac]

I haven't looked into flakes yet (I've been using Nix for two years now, shame on me lmao) but I am not against adding support for flakes in rix. But I would need commitment from the both of you to maintain this as I wouldn't be able to.

I believe that if we implement it correctly, and not mess it up too much, then it should be minimal maintenance. Let's create a branch or a few branches first, and we can agree on the responsibilities before merging.

Or wait! We can also just call it "experimental" ... I saw that somewhere...

[dwinkler1]

I'm not sure if this will create headaches for users but flakes also allow extra-substituters to be specified:

  nixConfig.extra-substituters = [
    "https://rstats-on-nix.cachix.org"
  ];

Works fine for me but I think we should test this on a new setup first.

[Kupac]

Just to be clear: we're aiming for R code within rix that generates a valid flake, right? Or are you thinking to call out to your rix-flake from rix to get the template?

[Kupac]

BTW, here's some changes I started to make a few months ago. Quite far from where I'd like it to be.
https://github.com/Kupac/rix/tree/add_flake_support
Also some notes while trying to wrap my head around how rix works

[dwinkler1]

Just to be clear: we're aiming for R code within rix that generates a valid flake, right? Or are you thinking to call out to your rix-flake from rix to get the template?

I think that's just an implementation detail at the end. I have a slight preference for having the basic flakes generated already because this would be useful separately for nix users. But I can also run that entirely separately from rix in the flakes repo. What I very strongly prefer is that we use as much of the existing rix code as possible by generating a file that is imported into the flake. As far as I can see that would require only two functions to change ie take a flakes flag: the one generating the header and the one generating the shell. The changes are trivial and we would get a valid nix file. We could then have the flake template as part of rix and add only the appropriate url for the repo or call nix flake init to a separate repo with the appropriate flake there. Generating the flakes is done via GitHub actions and is relatively fast (~30 sec) so I could schedule it to update twice a day.

[b-rodrigues]

What I very strongly prefer is that we use as much of the existing rix code as possible by generating a file that is imported into the flake. As far as I can see that would require only two functions to change ie take a flakes flag: the one generating the header and the one generating the shell. The changes are trivial and we would get a valid nix file.

I would be very much in favour of merging such a PR! Would there be the need to add a further function in rix to add the appropriate url or call nix flake init in a separate folder? Anything else that rix could offer to the user to make it even easier?

[dwinkler1]

Would there be the need to add a further function in rix to add the appropriate url or call nix flake init in a separate folder?

I still need to figure out the url. Flakes work a bit differently since they have their own lock file so I think the distinction of frozen-edge and bleeding-edge does not exist here. We would point to main of a repo and when the user builds the flake it creates the lock for the latest commit. I usually use more conservative setups (ie stable release) so I'll have to test that more.

Other than that a flakes deployment would generate a file similar to the current default.nix and then call nix flake init TEMPLATE where the template is the appropriate one for the provided version.

Anything else that rix could offer to the user to make it even easier?

We could build a wrapper around nix flake update to get the latest commits for bleeding-edge deployments. Probably more but I'll have to think about it.

[dwinkler1]

Just to make sure:

• latest-upstream gets the latest commit from the default branch of https://github.com/NixOS/nixpkgs -> flake: url = "github:NixOS/nixpkgs";
• frozen-edge gets the latest commit from https://github.com/rstats-on-nix/nixpkgs/tree/r-daily -> flake: url = "github:rstats-on-nix/nixpkgs/r-daily";
• bleeding-edge points directly to https://github.com/rstats-on-nix/nixpkgs/tree/r-daily -> flake: url = "github:rstats-on-nix/nixpkgs/r-daily"; + run nix flake update every time
• r-devel points directly to https://github.com/rstats-on-nix/nixpkgs/tree/r-devel -> flake: url = "github:rstats-on-nix/nixpkgs/r-devel"; + run nix flake update every time
• Equivalent to previous for r-devel-bioc-devel and r-bioc-devel

Just FYI this is what the flake.lock entry looks like for r-daily:

    "nixpkgs": {
      "locked": {
        "lastModified": 1739406499,
        "narHash": "sha256-l/oc6ZOOiiI10aB66yoFS0BOeo/74hHa555+JxcdLm8=",
        "owner": "rstats-on-nix",
        "repo": "nixpkgs",
        "rev": "c45c7f25a8c97e04735c465ec10e268798a98d58",
        "type": "github"
      },
      "original": {
        "owner": "rstats-on-nix",
        "ref": "r-daily",
        "repo": "nixpkgs",
        "type": "github"
      }
    }

[b-rodrigues]

yep, that’s right

there’s also dated branches, as you know

[dwinkler1]

Awesome. We can already handle the date branches and r versions automatically. I was worried about generating the *-edge versions, but we can have templates pointing to those branches. This reinforces my preference for having the flakes generated separately. My understanding is that currently rix does not depend on nix and I would like to keep it that way. We could check if nix is installed, and if so run nix flake init ..., otherwise print the appropriate command to run on any machine. I would probably add a init_flake function to rix that can be called to get the flake but warns the user if there is no settings file. What do you think?

FYI the flakes handle missing files gracefully but are just useless :)

[b-rodrigues]

I would probably add a init_flake function to rix that can be called to get the flake but warns the user if there is no settings file. What do you think?

that sounds like a reasonable default. Also, could I be greedy and ask if you open a PR, that you also add some pointers for a vignette, something like "Why use flakes" something to that effect? To make them easier to introduce to beginners (let me be clear, that’s not like it would be a blocker for merging, but it would help me draft something).

[dwinkler1]

Sure, give me a few weeks; I need to finish my thesis first. :) I expected this addition to be much more complicated and take more discussion. If anyone wants to test in the meantime I have the templates at:

https://github.com/dwinkler1/rix-flakes/blob/main/templates.nix

The script to patch the current rix default.nix to something consumable by a flake comes with the r-default template.

In addition to latest-upstream I also added unstable-upstream which points to the nixpkgs-unstable branch:

For standalone Nix users, nixpkgs-unstable channel branch is the rolling release, where packages pass only basic build tests and are upgraded continuously.
https://wiki.nixos.org/wiki/Nixpkgs/en

Running from main directly is just not for the faint of heart (i.e., me).

[b-rodrigues]

Sure, give me a few weeks; I need to finish my thesis first.

sure, good luck 😄