[RFC] Proposal: Engineering a Common "Plugin" Infrastructure

[deliciouslytyped]

Edit: It may be easier to start by looking at https://discourse.nixos.org/t/wip-rfc-a-new-nixpkgs-frontend-for-language-infrastructures/3447
Edit: I made a WIP PoC library https://github.com/deliciouslytyped/nix-rootedoverlay/ , it still needs a lot of work but seems to be functional.
Edit: Help wanted: someone to hand-hold me through the fixpoint stuff needed for overrides in this infrastructure I think I figured it out.

There are various separate, probably incompatible implementations of plugin infrastructure across nixpkgs, examples being at least:

(NOT REVIEWED indicates the author is tentatively aware of a plugin system but has not inspected the implementation)

• TODO, list here
• a not-yet-upstreamed implementation for Ghidra cannibalized from Idris
• NOT REVIEWED: Idris (see above)
• NOT REVIEWED: firefox / other mozilla software
• NOT REVIEWED: Eclipse
• package managers of interpreted languages (TODO: any 'sub'-package manager?)
  • NOT REVIEWED: python packages
  • NOT REVIEWED: ruby
• NOT REVIEWED: Jekyll
• NOT REVIEWED: redmine
• NOT REVIEWED: perl perl: add .withPackages #59215
• HYPOTHETICAL wine package management
• NOT REVIEWED retroarch
• NOT REVIEWED gnome-shell
• NOT REVIEWED terraform (was suggested for it's simplicity)
• NOT REVIEWED r/rstudio/rwrapper/rpackages
• NOT REVIEWED llvmPackages
  No implementation:
• jenkins (?)

CONSIDER: asking maintainers of the various systems for input

The more advanced implementations usually (TODO: substantiate this) use some kind of fix-point mechanism which can be quite hard to understand.

Would it be possible and reasonable to factor out a plugin handling API that can be specialized to various desired implementations, such that the internal mechanisms do not need to be understood by the user? This could also result in the overall simplification of these components of nixpkgs.

A generic plugin system should:

• provide one or more (possibly extensible) methods of applying plugins to a package:
  • feeding plugins back into the original derivation, e.g.
    {..., plugins ? ... }: mkDerivation ...
  • merging base and plugin derivations via something like symlinkJoin.
    This has the advantage of being faster and less wasteful than rebuilding the base package, but may fail with improper symlink handling? This also may have other failure modes such as runner makeWrapper scripts referring to the original derivation, resulting in the plugins being ignored because they are in a new derivation. (TODO: elaborate on this, this is an issue I ran into.)
  • ...other...?
• provide the ability to have in-tree (nixpkgs) plugins
• provide the ability to have out-of-tree plugins via overrides, overlays, or some similar mechanism
• ...other...?

Pending questions:

• What makes a plugin system a plugin system? (I.e. what is the scope of this?)
• how does Add pkgs.overrideWithScope #44196 relate?

[deliciouslytyped]

If a desirable solution already exists, then other plugin systems listed in the (TODO) aggregation should be migrated to it.

[deliciouslytyped]

This post: syntax/semantics/ergonomics

I quite like the rootPackage.withPackages (p: with p; [ a b c ]) pattern.

Under what circumstances should it be considered meaningful to include non-child packages of the rootPackage in the list? Are there conditions where it's useful to pass in external packages? Say, something like python37.withPackages (p: [ pkgs.hello ]), or should the intended result of such a thing be achieved some other way?
Edit: a use case may be hybrid environments like interfacing between Python and R?

This should not be the same as accepting out-of-tree plugins; I think it would be reasonable to accept something like functions in the plugin list, where during evaluation the root package is applied to the plugins if necessary, and passed on to another builder (or some similar approach).
This is done in my current implementation of ghidra plugins because they need access to a build script located in the root package. (TODO substantiating example)

Alternatively, external overrides should be applied strictly through some pattern like the following:
(rootPackage.overridePackages { d = parent: ...; }).withPackages (p: with p; [ a b c d ])
The parent: parameter may lead to infinite recursion with improper use? (This is the kind of thing that might be useful to mention in API docs, if it's not possible to reasonably prevent altogether)
Though I find the parenthesization a bit awkward.

[deliciouslytyped]

Per suggestions I have attempted to explore makeScope/newScope/... , but I haven't been able to understand them yet enough to get them to work right. My current understanding is that would at least cover my minimum desired functionality for providing overrides.

[joepie91]

Some quick additional notes before I forget, though I don't have much time to elaborate right now:

• Mods for games are also plugins, not sure whether those require special handling.
• There's currently a disconnect in cases where plugins can be managed from within the application as well as from Nix configuration; there should probably be some sort of way to reconcile that, though it may involve extra tooling.
• PlatformIO orchestrates a bunch of different tools for interacting with different embedded devices and architectures; it currently downloads those on runtime. Not sure if that would fall under the conceptual meaning of 'plugins' here.

[ghost]

Packages managed by the package managers of (at least interpreted) languages, like Ruby gems, are also plugins, right?

[joepie91]

More: Jekyll has plugins, if I recall correctly.

[deliciouslytyped]

@joepie91
To address your points in some order:

2 )
The application managed plugins problem is something that has bothered me for a while and I'm not sure what to do about it.

My knee-jerk reaction is that applications need to support application-external plugin/setting management if we don't want strange things to happen? The reason I mention settings management, is because that's usually how plugins get added to the "current set of running plugins", right? My mental model is currently based off of firefox - for which I'm not sure of the details (but there's this mess of notes nix-community/home-manager#606 ) - which is to say, I think (?) I'm agreeing with you.

1 )
I don't see yet why game mods are special? I'm imagining that specific installation functionality would be handled by domain specific handlers anyway. Application specific hooks.

3 )
I will talk to you about this on IRC and update this post.

@eighal yeah I'm inclined to think so. My current heuristic for "plugins" is whether there is some central thing that needs to have plugins fed back into it, that being the interpreter/'s package environment.
But maybe I'm going about this from the wrong direction. Instead of thinking about plugins, maybe we should be thinking about "things that have a domain specific set of 'sub'packages, that must be fed-back the packages"? I wouldn't be surprised if this basically reduces to the thing going on with the fixpoint stuff.

Misc:
added ruby and jekyll to tentative list.

Edit:
Just a subpackageset is easy to make, it gets hairy with this fixpoint stuff which is needed for overrides.

[deliciouslytyped]

TL;DR: if it has something that looks like a .withPackages (p: ...), or looks like it would want a .withPackages (p: ...) then at least that is relevant.

[aanderse]

The redmine service has plugins and themes. This is currently not implemented well.

[coreyoconnor]

Projects with internal package managers for handling plugins I've worked with: jenkins, retroarch, gnome-shell. These all can download their own plugins/extensions which, sometimes, works. Other times there are requirements to the .so/.jar/.js that are not part of the provided environment. Hence: those need to be built via nix and, preferably, block the installation of built artifacts dynamically.

As @deliciouslytyped mentions: applications themselves need some consideration to prevent bad things happening.

EG: in retroarch an update for bundled core can be requested but will fail with an odd error as that .so is RO.

[coreyoconnor]

I noticed the question mark next to jenkins and decided to dig in a bit mroe.

• The relevant code is at nixos/modules/services/continuous-integration/jenkins/default.nix
• the tool referenced jenkinsPlugins2nix does not (afaict) build the plugin. Only packages a pre-built plugin.
• only runtime dependencies would impact actually using the plugin
• jenkins itself is not built by nix but only packaged

I agree this puts jenkins in a different group than the above.

[joepie91]

Addition: KDE has a plugin manager as well, for "plasma widgets" and such, I believe.

Edit: and Thunar has file manager plugins too, though no interactive manager I think - the Thunar package in nixpkgs also already has a plugins option.

[deliciouslytyped]

@coreyoconnor The reason I "separated" it was because on a cursory look I didn't see an implementation of handling plugins - hence: "No implementation". If it had an implementation I would have listed it under the rest. (This is just a comment on that and not really addressing anything else.)

Edit: which is to say, it has plugins, there is just no implementation of handling them in nixpkgs. Though now that I think about it, the list might get really long if we start considering everything everywhere that takes plugins. :P - which doesn't mean that I think we shouldn't consider everything. However, I guess after the first 20 or 30 examples it will be time to start trying to figure out archetypes.

[deliciouslytyped]

I'm currently poking around overlays and...that stuff. Still trying to figure out how it works.
I spent the day prototyping stuff with a type system but I'm not quite at the goal of the previously mentioned syntax I want yet...

https://www.youtube.com/watch?v=W85mF1zWA2o (@nbp NixCon talk on overlays) is helpful.

[deliciouslytyped]

I have a preliminary system for Ghidra I will try upstreaming soon, otherwise I'm still exploring the problem space.

[deliciouslytyped]

#45269

[deliciouslytyped]

Possibly related NixOS/rfcs#3

[aanderse]

Just noticed another item which might be of interest. squirrel-sql has database drivers that can be considered plugins. See nixpkgs/pkgs/development/tools/database/squirrel-sql/default.nix for details.

[deliciouslytyped]

Misc wishlist: It would be nice if enableDebugging worked properly across package sets (where appropriate).

Edit:
Possible mechanism: enableDebugging should check if the target package has an enableDebugging attribute, if yes it should call it - this should be nailed down as an api convention.

Currently what I'm doing for plugin sets is merging a .pkgs attribute into to the package sets base overlay and merging it into .callPackage's scope. So maybe functions that rely on overriding something should also override it in a pkgs passed to these, but that sounds kind of messy.

[deliciouslytyped]

Forgot to link my WIP POC: #60664

[deliciouslytyped]

I turned it into a WIP library https://github.com/deliciouslytyped/nix-rootedoverlay/ , it's pretty broken right now. Help wanted.

Edit: I think I fixed stuff but I'm still trying to write tests.

[aanderse]

As a new data point I found that moodle plugins are scattered all throughout the directory structure of the package. Typically plugins for an application should go under 1 or maybe 2 different directories (say plugins/ and/or themes/, for example), but with moodle they go all over the place, depending on what type of plugin you're looking at.

For example plugins classified as Enrollment plugins go under enrol/, while those classified as Blocks go under blocks/, etc... At this time it appears moodle supports upwards to 50 different types of plugins, each with a different folder to be installed under.

[deliciouslytyped]

Where can I find the moodle stuff?

From the perspective of my WIP rootedoverlay stuff:
Not sure which of the two you're trying to point out here? - I'm assuming the latter:

• From a packaging standpoint personally I'd just shove everything in a top level plugins directory, and the top level plugin list can handle exposing things to the user
• From the standpoint of "applying" plugins, I think this would still fit into a supplying a custom Root -> Plugin -> Root function, such that this function can handle a plugin that specifies it's installation details.

Does that sound reasonable?
Edit: rewording

[aanderse]

@deliciouslytyped I was mostly just providing another data point... but yes, the second point sounds good 😄

[deliciouslytyped]

All good, thanks :D

[deliciouslytyped]

I'm currently working on a POC for the R package set.

[deliciouslytyped]

There's a somewhat broken implementation at https://github.com/deliciouslytyped/nix-rer/ currently.
The major bugs off the top of my head are

• propagated inputs aren't handled correctly (I'm not sure how the original implementation dealt with this) so surrogate.cran.curl doesn't build if it's the dependency of something, but it builds by itself.
• some packages are not found both because I don't have a way to search the global namespace yet, and I haven't added the bioc package sets.
  Also the codebase is still quite a mess.

but it seems to basically work.

Edit:
Oneliner example:

nix-shell -E "(import ./. {}).surrogate.withPackages (p: [ p.cran.purrr ])" -v --run "R --quiet -e 'library(purrr); map(list(1,2,3), ~(..1+1))'"

[deliciouslytyped]

https://discourse.nixos.org/t/wip-rfc-a-new-nixpkgs-frontend-for-language-infrastructures/3447

[deliciouslytyped]

@barakber I'm thinking of writing a library set for Prolog. I'm not yet familiar with how prolog libraries work, can you maybe explain a bit how

nixpkgs/pkgs/development/compilers/swi-prolog/default.nix

Line 11 in 9239460

packInstall = swiplPath: pack:

is intended to work? Thanks!

[deliciouslytyped]

I'm currently working on porting the Idris infrastructure.

[barakber]

Sure. packInstall is a Nix function that is using https://www.swi-prolog.org/pldoc/man?section=prologpack from the shell, to install a list of "packs" (from https://www.swi-prolog.org/pack/list) during the installation phase of the SWI-prolog Nix derivation.
The goal is really to have a "swipl.withPackages" that does all pack_install()s for you.

[deliciouslytyped]

#46842

[deliciouslytyped]

It seems like a good idea to read through and steal some ideas from https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md , there appear to have been developments since I last read it. (edit: it's more likely that I just didn't notice a bunch of things...)

[deliciouslytyped]

TODO: read through https://discourse.nixos.org/t/scaling-the-python-package-set-in-nixpkgs/3749/ , evaluate relevance in both directions, steal ideas

[deliciouslytyped]

TODO: look at whats going on in #53834 (comment)

[stale]

I marked this as stale due to inactivity. → More info

[toraritte]

This discussion seems very relevant: Status of lang2nix approaches

[deliciouslytyped]

If that's relevant, the dream2nix project may be as well.