From 02519f097000855ceba36dd8bb789db97ffe7660 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 6 Sep 2022 13:37:44 -0400 Subject: [PATCH 01/19] nix-store-layer: Copy template --- rfcs/0000-nix-store-layer.md | 58 ++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 rfcs/0000-nix-store-layer.md diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md new file mode 100644 index 000000000..43569cf57 --- /dev/null +++ b/rfcs/0000-nix-store-layer.md @@ -0,0 +1,58 @@ +--- +feature: (fill me in with a unique ident, my_awesome_feature) +start-date: (fill me in with today's date, YYYY-MM-DD) +author: (name of the main author) +co-authors: (find a buddy later to help out with the RFC) +shepherd-team: (names, to be nominated and accepted by RFC steering committee) +shepherd-leader: (name to be appointed by RFC steering committee) +related-issues: (will contain links to implementation PRs) +--- + +# Summary +[summary]: #summary + +One paragraph explanation of the feature. + +# Motivation +[motivation]: #motivation + +Why are we doing this? What use cases does it support? What is the expected +outcome? + +# Detailed design +[design]: #detailed-design + +This is the core, normative part of the RFC. Explain the design in enough +detail for somebody familiar with the ecosystem to understand, and implement. +This should get into specifics and corner-cases. Yet, this section should also +be terse, avoiding redundancy even at the cost of clarity. + +# Examples and Interactions +[examples-and-interactions]: #examples-and-interactions + +This section illustrates the detailed design. This section should clarify all +confusion the reader has from the previous sections. It is especially important +to counterbalance the desired terseness of the detailed design; if you feel +your detailed design is rudely short, consider making this section longer +instead. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Alternatives +[alternatives]: #alternatives + +What other designs have been considered? What is the impact of not doing this? + +# Unresolved questions +[unresolved]: #unresolved-questions + +What parts of the design are still TBD or unknowns? + +# Future work +[future]: #future-work + +What future work, if any, would be implied or impacted by this feature +without being directly part of the work? From 9d8a26c95953efa67a5de42d3d68d911cf723b00 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 6 Sep 2022 13:37:49 -0400 Subject: [PATCH 02/19] nix-store-layer: First draft --- rfcs/0000-nix-store-layer.md | 286 ++++++++++++++++++++++++++++++++--- 1 file changed, 266 insertions(+), 20 deletions(-) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md index 43569cf57..a977445f6 100644 --- a/rfcs/0000-nix-store-layer.md +++ b/rfcs/0000-nix-store-layer.md @@ -1,7 +1,7 @@ --- -feature: (fill me in with a unique ident, my_awesome_feature) -start-date: (fill me in with today's date, YYYY-MM-DD) -author: (name of the main author) +feature: nix-store-layer +start-date: 2022-09-06 +author: John Ericson co-authors: (find a buddy later to help out with the RFC) shepherd-team: (names, to be nominated and accepted by RFC steering committee) shepherd-leader: (name to be appointed by RFC steering committee) @@ -11,48 +11,294 @@ related-issues: (will contain links to implementation PRs) # Summary [summary]: #summary -One paragraph explanation of the feature. +Allow building a Nix executable with just the store layer. +Do this to manage complexity, and promote a "Greater Nix" community. # Motivation [motivation]: #motivation -Why are we doing this? What use cases does it support? What is the expected -outcome? +## Managing complexity + +We have recently added a ton of functionality to Nix. +Most of this is in Flakes, which just has enormous surface area, but things like floating content-addressed derivations and other RFCs add complexity to the core store layer too. + +If we view Nix as one monolithic whole, it will grow too complex and unwieldy, and we will be unable to manage it as we the complexity bogs us down. +However, if we embrace layering we can "divide and conquer" the project, and manage that complexity. +This will ensure the continued sustainability of Nix. + +We currently embrace layering somewhat as an implementation detail, but only as an implementation detail. +The division between `libnixstore`, `libfetchers`, `libexpr`, etc. is not yet exposed to users, emphasized in documentation (though this is changing thanks to @Fricklerhandwerk's efforts with new [architecture documentation](https://github.com/NixOS/nix/pull/6888)!). + +We should instead fully embrace it: + +- Docs + + - More advanced documentation can explain layering for those that want a deeper understanding of Nix. + + - Even more basic documentation can still benefit from separate terminology before the layering is fully explained. + See https://www.haskellforall.com/2022/08/stop-calling-everything-nix.html for a phenomenal take-down of how calling everything "Nix" today confuses users and leaves them unable to articulate what parts of the ecosystem are frustrating then. + +- Separate executable ensure lower layers build in isolation, new integration tests those executable without aid of high-layer info. + +- New NixOS-Foundation-authorized teams foster and advocate for layers in isolation + +### Starting with the store layer + +Ultimately, I would like to take this approach to all the layers. +But I want to constrain the scope of this RFC to keep it tight an actionable. + +Layering between e.g. Flakes and the Nix Language doesn't yet exist in the implementation in the form of a library separation. +I don't want to be "blocked" on major new development work, and anything involving Flakes is also far more controversial, and best avoided at first. + +Conversely, the store layer is already quite well separated. +The orthogonality between it and other layers "proven" in the wild by projects like Guix (more on that latter part in the next section). +The gap is also widest in terms of the layer of abstraction between, on one hand, nascent "infrastructure projects" like + +- CA derivations +- Secrets in the store +- Trustless remote building +- Daemon and Hydra protocol rationalization +- Windows Support +- Computed derivations (which are built by the other derivations) +- IPFS store + +etc. and "UX projects" like + +- Flakes +- TOML Flakes +- Hard-wired module system + +So focusing on the lowest layer first, we get the most "bang for buck" in terms of managing extremely different sorts of work separately. + +### A disclaimer + +To be clear, none of this is to say we should abandon the idea of Nix as a whole. +There can still be governance of Nix as a whole, that teams and people focused on "infra" or "flakes" would ultimately need to report to. +The goal is not to overreact, but strike a balance between: + +1. Making sure Nix as a whole continues to make sense +2. Make sure layers make sense in isolation not just in the context of the way they are currently used. + +## Pluralism + +As Nix grows more popularity, it will be inevitable that different groups want to explore in different directions. +This is the *pluralism* of a larger community, and we should embrace it as a strength. + +There are many possible ways in which to write down packages, The Nix language and Nixpkgs idioms, and Guix, for example, are just two points in a much larger space. +There are also many possible ways set up build farms. +Our current central dispatcher, many remote-builder agents model, point-to-point protocol model is also just point in a much larger design space. + +The "derivation language" and store *interface* however, seems to me at least to be a very natural design. +There are a few tweaks and generalizations we can make, but I struggle to envision what wildly different alternatives one might want instead. + +An stable, small interface that fosters lots of design exploration above and below is known from networking as a *narrow waist*. +The oil shell blog as a [great post](https://www.oilshell.org/blog/2022/02/diagrams.html) with more details on the concept. +It's not every day that a project happens upon a great narrow waist design, but I believe we've discovered a very good one with Nix, and that should be seen as a *key asset*, even if it is not how we recruit "regular users". + +By making a store-only Nix, we put more emphasis on this key interface. +All functionality the store-only Nix offers factors through this interface. +The upper half of Nix likewise uses the lower half through this interface. +The daemon protocol represents this interface for IPC, and allows either half to be swapped for a different implementation. + +To help explain the community-building benefits, it might help to go over some specific examples. + +### Tvix and go-nix + +In https://tvl.fyi/blog/rewriting-nix, TVL announced that, frustrated in trying to refactor Nix into something more modular and flexible, they were aiming to make a new implementation from scratch. + +Since then, what has emerged is that [*Tvix*](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator, +and [*go-nix*](https://github.com/nix-community/go-nix) is a new implementation of the store layer. + +First of all, the fact that they are planning on two completely separate implementation oriented around this same "narrow waist" is testament to the appeal of the design. + +Second of all, note that per their blog post, they have separate, orthogonal experiments they wish to run on both sides of the store interface divide. +Above, they want to experiment with radically different evaluation strategies, especially to speed up Nixpkgs evaluation. +Below, they want to experiment with the standardized containerization technologies that exist for new ways of sandboxing and distributing builds with less bespoke Nix-specific code. + +I think these are both great goals, and for the sake of the ecosystem as a whole, it should be as easy as possible to run such new experiments. +In particular, a novel evaluator should be usable with the standard C++ Nix store layer, and a novel store layer should be reusable with the standard C++ Nix evaluator. + +Yes, strictly speaking, we only need a stable daemon protocol to accommodate that goal, which we have. +But ensuring the C++ Nix components can be built separately for use in isolation further send the message that such experimentation is *good* and *welcomed*. +Messaging matters, and making our layered architecture "official" as this RFC proposes I think sends a better message. + +### Guix + +Guix is more diverged from Nix than Tvix + go-nix, and thus hints more at the end breadth of the design space yet to be explored. + +The store layer is the same, but the layers above, instead of being a implementing of the Nix language, is a completely different design with Guile Scheme. +The choice of language is just the tip of the iceberg here. +More profoundly, they also have a more "library" than "interpreter" model where packages depend on Guix as a library, which talks to a small rump daemon. +Guile sits far lower in their stack than the Nix language interpreter does; it is as if we rewrote some of our C++ into nix language code, and nix language code could do enough side effects to make that possible. + +The point of this discursion is to show that not only are radically different implementation of the same spec possible on either side of the store interface (what Tvix + go-nix aim for), but radically different designs not going for comparability also. + +Guix currently uses a stripped-down fork of C++ for its core daemon. +Clearly, it would be nicer than that if, as this RFC proposes, we supporting building just such a stripped-down daemon with*out* any forking needed. +Then we could all collaborate on one bit C++ that didn't drag in features Guix didn't want, no forking needed. + +Still, the long term goal of Guix is to rewrite that remaining C++ into Guile Scheme too? +At that point, does that mean the benefits of this RFC for Guix are gone too? + +I don't think so. +It makes total sense that Guix wants an implementation they fully control, in the language they prefer, and I have no interest in dissuading them from that goal! +But it would be still nice to have full interop so Nix can work with the Guix daemon, and Guix with the Nix daemon. + +This is especially important in "institutional settings", such as high performance computing (HPC) build farms for science, software development shops, and everything in between. +HPC and academic use in particular is something Nix and Guix are both interested in. + +Firstly, this RFC has benefits for bureaucratic expediency. +There is usually a lot of red tape needed to get a new technology deployed thought a build farm. +If Guix and Nix users have to separately ask for their build farm store layer backend to be rolled out, that is twice the headache for IT, with half the stakeholders asking for each deploymenet. +If, on the other hand, Guix and Nix users separately agree on one store layer backend to be rolled out they both can use, that is twice has many people asking for a single deployment --- a much stronger ask on IT. + +Secondly, and perhaps more abstractly, I think this project allows both projects to better utilize their own design and resources. + +Nix and Guix have completely independent visions above the store layer --- it is for here that Guix was created. +This is where both projects are choosing to innovate post "fork" (Guix from day 1 with Guile, us more recently with Flakes). +This is where the projects compete at the level of *ideas*. +Below the store interface, conversely, I think everyone wants the same things. +I have yet to hear of any store-layer idea that is "Guix-y but not Nix-y", or "Nix-y but not Guix-y". +Here the projects are not competing on *what* is being implemented, but *how well* it is being implemented. + +In such a situation, interoperability is a free win. +Since there is no underlying philosophical difference at this layer, there is straightjacket imposed from trying to be interoperable. +And while it's good and fine to compete on implementation, including rewriting the renaming C++, it's nice to be able abandon that competition at any more moment and join forces on a shared implementation, freeing up resources for other things. +To be very clear, this doesn't mean I am advocating that Guix "give up" on its independence from Nix --- maybe go-nix will end up being the dominant implementation and we all just use that! +I do not what the future holds, but I want to make sure we keep our options open, and each project is allowed to boost the other as much as possible without sacrificing design flexibility. + +Ultimately, while the single-machine single-user Nix experience is quite good, the shared build-farm multi-user experience with Nix and Guix is quite a bit *worse* than it could be. +I want to see major improvements in that area, and I want to see all projects benefits, and I really don't care whole ends up delivering those features first so long as the rest of us can benefit. + +I do not expect Guix to be immediately sold on this plan, but as that larger project, I think it behooves us to take the first steps to build trust and coordination. # Detailed design [design]: #detailed-design -This is the core, normative part of the RFC. Explain the design in enough -detail for somebody familiar with the ecosystem to understand, and implement. -This should get into specifics and corner-cases. Yet, this section should also -be terse, avoiding redundancy even at the cost of clarity. +## The split itself + +Allow building a store-only version of Nix. +This is a Nix executable that links `libstore` but not `libfetchers`, or `libexpr`. +Plenty of commands like `nix daemon`, `nix log`, and the `nix store` sub-commands don't care about evaluation, fetching, or flakes at all. +https://github.com/NixOS/nix/issues/6182 is a draft PR implementing this, splitting `libcmd` into two parts so the CLI code reused. +We will finish it off with as many commands as are reasonable to include, and merge it. + +## Additional in-code obligations + +### Tests + +The current test suite uses Nix language for most tests of store-layer functionality. +But it also allows using a separate daemon with most tests. + +1. To start, we should test the full Nix against the minimal Nix's daemon, in addition to our regular tests. + +2. Longer term, we should write new tests that don't use the Nix language. + E.g. we might create a `read-derivation` complement of `show-derivation` that accepts a nicer JSON representation of a derivation as input. + This will allow the store-only Nix to be tested in isolation. + +### CI + +The store-only Nix and its tests should be built as part of CI, as "first class" as our existing CI jobs. +That means both in the channel-blocking Hydra eval, and per PR. +If we hit the limits of Github Actions in per-PR CI, we should consider using Hydra instead / in addition as has already been discussed. + +### Manual + +It should be possible to build a store-only manual without information on the other layers too. +This would be the manual that is distributed with the store-only Nix. +Of course, store-only and full can share sections, so we aren't duplicating work. + +## Out-of-tree obligations + +### A store team + +An official NixOS-foundation-authorized teams should be set up to manage store layer design decisions (below the threshold of needing an RFC) that don't effect that the rest of Nix. +\[Some sort of decision that affect all layers is out of scope, must be deliberated with stakeholders from other layers too, probably should be RFC due to such large scope.\] + +This team should establish communication with counter parties in Guix leadership. + +### Website + +The fully and store-only version of Nix should both be presented for download on the website. +This should be just like how Plasma, Gnome, and headless installer images for NixOS are all offered. # Examples and Interactions [examples-and-interactions]: #examples-and-interactions -This section illustrates the detailed design. This section should clarify all -confusion the reader has from the previous sections. It is especially important -to counterbalance the desired terseness of the detailed design; if you feel -your detailed design is rudely short, consider making this section longer -instead. +## Maximal Nix unlike today + +I cannot emphasize this enough, but the interface of maximal store+exprs+flakes Nix remains *exactly* like today. +In particular, lower level commands can be used with higher level "installables" (arguments), so e.g. +``` +nix show-derivation flake#bar +``` +will still work. + +## Good tasks anyways! + +Lots of the plan above I think is good work we should be doing anyways, regardless of whether we expose a store-only Nix. +If you believe this, then the "cost" of this RFC is a lot less. + +### Tests + +Splitting the test suite per natural layer of the implementation is good work because it combines the specificity of unit tests with the real-world-ness of integration tests. +"entire kitchen sink" tests make it harder to narrow down root causes of failures. + +### Manual + +The documentation team is already working to clean up the manual, and this effort already involves emphasizing layering. +So whether we formally make a store-only Nix or not, I suspect the overhauled manual will natural have easy boundaries from which to "extract" the store-only manual. + +## Security + +The daemon is a privileged process. +Even if with upcoming changes it shouldn't need root, it does tasks like administrating OS sandboxes correctly which still are security-critical. +Having less code in the story-only Nix daemon, even if we think the removed code was "dead anyways" is always good. # Drawbacks [drawbacks]: #drawbacks -Why should we *not* do this? +Creating new teams, trying to build ties with other communities sounds scary. # Alternatives [alternatives]: #alternatives -What other designs have been considered? What is the impact of not doing this? +Do technical parts without governance or documentation parts. +But that feels to me like turning an ongoing shift in focus to a one-off change that is likely to bit-rot. # Unresolved questions [unresolved]: #unresolved-questions -What parts of the design are still TBD or unknowns? +What should the store-only Nix be called? # Future work [future]: #future-work -What future work, if any, would be implied or impacted by this feature -without being directly part of the work? +## Standardization across projects + +If we establish informal interop across store-layer implementations with Guix, a next step would be establish some sort of living standard that both communities have equal say in. +(Of course, implementations are free to implement features in excess of what the standard requires!) +The new store team can lead the process from our end. + +## Stabilization + +There is a looming question on how to stabilize Nix's big backlog of unstable features (New CLI, Flakes). +There is a lot of bad-blood over Flakes, both the feature itself and the way it has been rolled out. +I think the stabilization process can be an opportunity to heal old wounds. + +At a minimum, this can involve stabilizing the new CLI before Flakes. +But even that that is a lot of new feature surface area to review. +I think even better is stabilizing just the store-only new CLI first. + +This is easily the least controversial part of our unstable feature backlog, and yet there is still plenty to discuss. +Questions like + +- logging +- store paths on `stdout` at end of build? +- Should commands like `show-derivation` should use `--json` by default +- Flat vs hierarchical commands +- is `--derivation` a good flag? (I think not!) + +are all in-scope. + +Having a conversation just on this narrow first batch of stabilization both builds trust, and ensures these still-important issues they aren't lost in flame wars over more divisive topics. From fb641e83fae4f9c567e79955f7ccd61bcbcfe37b Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 06:52:24 -0400 Subject: [PATCH 03/19] nix-store-layer: pluralism -> marketplace of ideas Thanks @fricklerhandwerk for the suggestion. --- rfcs/0000-nix-store-layer.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md index a977445f6..58433bd8f 100644 --- a/rfcs/0000-nix-store-layer.md +++ b/rfcs/0000-nix-store-layer.md @@ -79,10 +79,11 @@ The goal is not to overreact, but strike a balance between: 1. Making sure Nix as a whole continues to make sense 2. Make sure layers make sense in isolation not just in the context of the way they are currently used. -## Pluralism +## Marketplace of Ideas As Nix grows more popularity, it will be inevitable that different groups want to explore in different directions. This is the *pluralism* of a larger community, and we should embrace it as a strength. +We do that be fostering a *marketplace of ideas*. There are many possible ways in which to write down packages, The Nix language and Nixpkgs idioms, and Guix, for example, are just two points in a much larger space. There are also many possible ways set up build farms. From 3410991d99be8f3990bd499ad81e20ca8e596e3e Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 06:55:35 -0400 Subject: [PATCH 04/19] nix-store-layer: Tweak Tvix and go-nix section --- rfcs/0000-nix-store-layer.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md index 58433bd8f..5049fa211 100644 --- a/rfcs/0000-nix-store-layer.md +++ b/rfcs/0000-nix-store-layer.md @@ -106,17 +106,20 @@ To help explain the community-building benefits, it might help to go over some s ### Tvix and go-nix In https://tvl.fyi/blog/rewriting-nix, TVL announced that, frustrated in trying to refactor Nix into something more modular and flexible, they were aiming to make a new implementation from scratch. +More recently, in https://tvl.fyi/blog/tvix-status-september-22 they lay out a basic approach of two projects: -Since then, what has emerged is that [*Tvix*](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator, -and [*go-nix*](https://github.com/nix-community/go-nix) is a new implementation of the store layer. +- [*Tvix*](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator, + +- [*go-nix*](https://github.com/nix-community/go-nix) is a new implementation of the store layer. First of all, the fact that they are planning on two completely separate implementation oriented around this same "narrow waist" is testament to the appeal of the design. -Second of all, note that per their blog post, they have separate, orthogonal experiments they wish to run on both sides of the store interface divide. +Second of all, note that per they have separate, orthogonal experiments they wish to run on both sides of the store interface divide. Above, they want to experiment with radically different evaluation strategies, especially to speed up Nixpkgs evaluation. -Below, they want to experiment with the standardized containerization technologies that exist for new ways of sandboxing and distributing builds with less bespoke Nix-specific code. +Below, they want to experiment with the standardized containerization technologies that already exist for new ways of sandboxing and distributing builds with less bespoke Nix-specific code. +They also want to apply the layering paradigm *within* go-nix, fostering even more modularity. -I think these are both great goals, and for the sake of the ecosystem as a whole, it should be as easy as possible to run such new experiments. +I think these are great goals, and for the sake of the ecosystem as a whole, it should be as easy as possible to run such new experiments. In particular, a novel evaluator should be usable with the standard C++ Nix store layer, and a novel store layer should be reusable with the standard C++ Nix evaluator. Yes, strictly speaking, we only need a stable daemon protocol to accommodate that goal, which we have. From 9bce871c7b37d6853c771575e36fe70dac6868c5 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 06:58:13 -0400 Subject: [PATCH 05/19] nix-store-layer: Make Store team future work --- rfcs/0000-nix-store-layer.md | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md index 5049fa211..6f430de23 100644 --- a/rfcs/0000-nix-store-layer.md +++ b/rfcs/0000-nix-store-layer.md @@ -40,8 +40,6 @@ We should instead fully embrace it: - Separate executable ensure lower layers build in isolation, new integration tests those executable without aid of high-layer info. -- New NixOS-Foundation-authorized teams foster and advocate for layers in isolation - ### Starting with the store layer Ultimately, I would like to take this approach to all the layers. @@ -70,15 +68,6 @@ etc. and "UX projects" like So focusing on the lowest layer first, we get the most "bang for buck" in terms of managing extremely different sorts of work separately. -### A disclaimer - -To be clear, none of this is to say we should abandon the idea of Nix as a whole. -There can still be governance of Nix as a whole, that teams and people focused on "infra" or "flakes" would ultimately need to report to. -The goal is not to overreact, but strike a balance between: - -1. Making sure Nix as a whole continues to make sense -2. Make sure layers make sense in isolation not just in the context of the way they are currently used. - ## Marketplace of Ideas As Nix grows more popularity, it will be inevitable that different groups want to explore in different directions. @@ -214,13 +203,6 @@ Of course, store-only and full can share sections, so we aren't duplicating wor ## Out-of-tree obligations -### A store team - -An official NixOS-foundation-authorized teams should be set up to manage store layer design decisions (below the threshold of needing an RFC) that don't effect that the rest of Nix. -\[Some sort of decision that affect all layers is out of scope, must be deliberated with stakeholders from other layers too, probably should be RFC due to such large scope.\] - -This team should establish communication with counter parties in Guix leadership. - ### Website The fully and store-only version of Nix should both be presented for download on the website. @@ -267,7 +249,7 @@ Creating new teams, trying to build ties with other communities sounds scary. # Alternatives [alternatives]: #alternatives -Do technical parts without governance or documentation parts. +Do programming parts without the documentation and website parts. But that feels to me like turning an ongoing shift in focus to a one-off change that is likely to bit-rot. # Unresolved questions @@ -278,11 +260,25 @@ What should the store-only Nix be called? # Future work [future]: #future-work +## Nix Store Team + +Now that we have this division in the implementation, we also have the opportunity to leverage it for governance purposes. +An official, NixOS-foundation-authorized team could be set up to manage store layer design decisions (below the threshold of needing an RFC) that don't effect that the rest of Nix. +\[Some sort of decision that affect all layers is out of scope, must be deliberated with stakeholders from other layers too, probably should be RFC due to such large scope.\] + +To be clear, this is to say we should abandon the idea of Nix as a whole. +There can still be governance of Nix as a whole; this team, and similar hypothetical, say, Flakes, Nix language, or User Experience teams would ultimately need to report to. +The goal is not to overreact, but strike a balance between: + +1. Making sure Nix as a whole continues to make sense +2. Make sure layers make sense in isolation not just in the context of the way they are currently used. + ## Standardization across projects If we establish informal interop across store-layer implementations with Guix, a next step would be establish some sort of living standard that both communities have equal say in. (Of course, implementations are free to implement features in excess of what the standard requires!) -The new store team can lead the process from our end. + +A new store team, per the above, could lead the process from our end, since the other parts of Nix are not shared with Guix and thus out of scope for this sort of cross-project standardization. ## Stabilization From fbb0720600b706a58623ad1bb996bb6601996bf0 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 07:18:44 -0400 Subject: [PATCH 06/19] nix-store-layer: Start tidying Guix section --- rfcs/0000-nix-store-layer.md | 49 +++++++++++------------------------- 1 file changed, 15 insertions(+), 34 deletions(-) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md index 6f430de23..5f333339f 100644 --- a/rfcs/0000-nix-store-layer.md +++ b/rfcs/0000-nix-store-layer.md @@ -124,46 +124,27 @@ The choice of language is just the tip of the iceberg here. More profoundly, they also have a more "library" than "interpreter" model where packages depend on Guix as a library, which talks to a small rump daemon. Guile sits far lower in their stack than the Nix language interpreter does; it is as if we rewrote some of our C++ into nix language code, and nix language code could do enough side effects to make that possible. -The point of this discursion is to show that not only are radically different implementation of the same spec possible on either side of the store interface (what Tvix + go-nix aim for), but radically different designs not going for comparability also. +The point of this discursion is to show that not only are radically different implementation of the same spec possible on either side of the store interface (what Tvix + go-nix aim for), but radically different designs not at all trying to be comparable also. -Guix currently uses a stripped-down fork of C++ for its core daemon. -Clearly, it would be nicer than that if, as this RFC proposes, we supporting building just such a stripped-down daemon with*out* any forking needed. -Then we could all collaborate on one bit C++ that didn't drag in features Guix didn't want, no forking needed. - -Still, the long term goal of Guix is to rewrite that remaining C++ into Guile Scheme too? -At that point, does that mean the benefits of this RFC for Guix are gone too? - -I don't think so. -It makes total sense that Guix wants an implementation they fully control, in the language they prefer, and I have no interest in dissuading them from that goal! -But it would be still nice to have full interop so Nix can work with the Guix daemon, and Guix with the Nix daemon. - -This is especially important in "institutional settings", such as high performance computing (HPC) build farms for science, software development shops, and everything in between. -HPC and academic use in particular is something Nix and Guix are both interested in. +All that said, below the store layer there is no difference in vision. +Because our communities are so separate, it would be easy to come up with diverging versions of how derivations, store objects, etc. should work. +That we have not done so I think is testament to the broad applicability of the Nix store design to many diverse groups of people with diverse goals. -Firstly, this RFC has benefits for bureaucratic expediency. -There is usually a lot of red tape needed to get a new technology deployed thought a build farm. -If Guix and Nix users have to separately ask for their build farm store layer backend to be rolled out, that is twice the headache for IT, with half the stakeholders asking for each deploymenet. -If, on the other hand, Guix and Nix users separately agree on one store layer backend to be rolled out they both can use, that is twice has many people asking for a single deployment --- a much stronger ask on IT. +What hope to do with Guix, then, is convene both projects to standardize this store layer in a way that supports both projects' goals. +Complementing the idea of a "marketplace of ideas" is when there is a certain design (like the Nix store layer), that is so broadly popular as to be a sort of "natural monopoly", that we should foster the most expansive and general idea of it as an exercise in coalition building and outreach. -Secondly, and perhaps more abstractly, I think this project allows both projects to better utilize their own design and resources. - -Nix and Guix have completely independent visions above the store layer --- it is for here that Guix was created. -This is where both projects are choosing to innovate post "fork" (Guix from day 1 with Guile, us more recently with Flakes). -This is where the projects compete at the level of *ideas*. -Below the store interface, conversely, I think everyone wants the same things. -I have yet to hear of any store-layer idea that is "Guix-y but not Nix-y", or "Nix-y but not Guix-y". -Here the projects are not competing on *what* is being implemented, but *how well* it is being implemented. - -In such a situation, interoperability is a free win. -Since there is no underlying philosophical difference at this layer, there is straightjacket imposed from trying to be interoperable. -And while it's good and fine to compete on implementation, including rewriting the renaming C++, it's nice to be able abandon that competition at any more moment and join forces on a shared implementation, freeing up resources for other things. -To be very clear, this doesn't mean I am advocating that Guix "give up" on its independence from Nix --- maybe go-nix will end up being the dominant implementation and we all just use that! -I do not what the future holds, but I want to make sure we keep our options open, and each project is allowed to boost the other as much as possible without sacrificing design flexibility. +Guix currently uses a stripped-down fork of C++ for its core daemon. +Clearly, it would be nicer than that if, as this RFC proposes, we supporting building just such a stripped-down daemon with*out* any forking needed. +Then we could all collaborate on one portion of C++ that didn't drag in features Guix didn't want, no forking needed. -Ultimately, while the single-machine single-user Nix experience is quite good, the shared build-farm multi-user experience with Nix and Guix is quite a bit *worse* than it could be. -I want to see major improvements in that area, and I want to see all projects benefits, and I really don't care whole ends up delivering those features first so long as the rest of us can benefit. +That said, Guix has a long-term goal of rewriting that remaining C++ into Guile Scheme too. +So to be clear, I *don't* think there is interest in sharing a store layer implementation as both project's reference copy. +But that's fine. +Having separate Nix and Guix reference implementations means both parties have recourse to implement ideas as they see fit prior to standardization. +Insofar as Guix is the smaller community, we can compare them to Mozilla in web standardization committees having a proper amount of say because they can do what they like with Firefox. I do not expect Guix to be immediately sold on this plan, but as that larger project, I think it behooves us to take the first steps to build trust and coordination. +Making a stand-alone Nix store executable demonstrates we are serious about layering and serious about standardizing that layer, and not just trying to get Guix users to use Nix instead. # Detailed design [design]: #detailed-design From 6f11ff92a0b84ba5763d2fddcda0c903b84115d4 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 07:19:00 -0400 Subject: [PATCH 07/19] nix-store-layer: Boild down Guix section further Implementation discussion is unneeded for now. --- rfcs/0000-nix-store-layer.md | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md index 5f333339f..f905cbccd 100644 --- a/rfcs/0000-nix-store-layer.md +++ b/rfcs/0000-nix-store-layer.md @@ -130,19 +130,9 @@ All that said, below the store layer there is no difference in vision. Because our communities are so separate, it would be easy to come up with diverging versions of how derivations, store objects, etc. should work. That we have not done so I think is testament to the broad applicability of the Nix store design to many diverse groups of people with diverse goals. -What hope to do with Guix, then, is convene both projects to standardize this store layer in a way that supports both projects' goals. +What hope to do with Guix, then, is convene both projects to make their store store layers interoperate. Complementing the idea of a "marketplace of ideas" is when there is a certain design (like the Nix store layer), that is so broadly popular as to be a sort of "natural monopoly", that we should foster the most expansive and general idea of it as an exercise in coalition building and outreach. -Guix currently uses a stripped-down fork of C++ for its core daemon. -Clearly, it would be nicer than that if, as this RFC proposes, we supporting building just such a stripped-down daemon with*out* any forking needed. -Then we could all collaborate on one portion of C++ that didn't drag in features Guix didn't want, no forking needed. - -That said, Guix has a long-term goal of rewriting that remaining C++ into Guile Scheme too. -So to be clear, I *don't* think there is interest in sharing a store layer implementation as both project's reference copy. -But that's fine. -Having separate Nix and Guix reference implementations means both parties have recourse to implement ideas as they see fit prior to standardization. -Insofar as Guix is the smaller community, we can compare them to Mozilla in web standardization committees having a proper amount of say because they can do what they like with Firefox. - I do not expect Guix to be immediately sold on this plan, but as that larger project, I think it behooves us to take the first steps to build trust and coordination. Making a stand-alone Nix store executable demonstrates we are serious about layering and serious about standardizing that layer, and not just trying to get Guix users to use Nix instead. From c17e63b3d9bc91f77333de05c41df30a9af703d3 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 07:27:39 -0400 Subject: [PATCH 08/19] nix-store-layer: Make incrementality of design clear Do this by putting the steps in order with numbers, and showing how the hardest part can come last. Thanks @fricklerhandwerk for the suggestion. --- rfcs/0000-nix-store-layer.md | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0000-nix-store-layer.md index f905cbccd..1474fe070 100644 --- a/rfcs/0000-nix-store-layer.md +++ b/rfcs/0000-nix-store-layer.md @@ -139,7 +139,10 @@ Making a stand-alone Nix store executable demonstrates we are serious about laye # Detailed design [design]: #detailed-design -## The split itself +The goals motivated above a broken down into small steps that we can execute in isolation. +This keeps the cost of this work initially lower, and generally reduces risk. + +## 1. The split itself Allow building a store-only version of Nix. This is a Nix executable that links `libstore` but not `libfetchers`, or `libexpr`. @@ -147,38 +150,31 @@ Plenty of commands like `nix daemon`, `nix log`, and the `nix store` sub-command https://github.com/NixOS/nix/issues/6182 is a draft PR implementing this, splitting `libcmd` into two parts so the CLI code reused. We will finish it off with as many commands as are reasonable to include, and merge it. -## Additional in-code obligations - -### Tests - -The current test suite uses Nix language for most tests of store-layer functionality. -But it also allows using a separate daemon with most tests. - -1. To start, we should test the full Nix against the minimal Nix's daemon, in addition to our regular tests. - -2. Longer term, we should write new tests that don't use the Nix language. - E.g. we might create a `read-derivation` complement of `show-derivation` that accepts a nicer JSON representation of a derivation as input. - This will allow the store-only Nix to be tested in isolation. - -### CI +Initially, we can test this store-only version of Nix with no changes to the test suite, by running full Nix with the store-only Nix's daemon. +Support for testing Nix against a separately-built daemon already exists and is in use today. The store-only Nix and its tests should be built as part of CI, as "first class" as our existing CI jobs. That means both in the channel-blocking Hydra eval, and per PR. -If we hit the limits of Github Actions in per-PR CI, we should consider using Hydra instead / in addition as has already been discussed. +If we hit the limits of Github Actions in per-PR CI, we should consider using Hydra instead / in addition, something that has already been discussed. -### Manual +## 2. Manual It should be possible to build a store-only manual without information on the other layers too. This would be the manual that is distributed with the store-only Nix. Of course, store-only and full can share sections, so we aren't duplicating work. -## Out-of-tree obligations - -### Website +## 3. Website The fully and store-only version of Nix should both be presented for download on the website. This should be just like how Plasma, Gnome, and headless installer images for NixOS are all offered. +## 4. Store-specific Tests + +The current test suite uses Nix language for most tests of store-layer functionality. +We should write new tests that don't use the Nix language. +E.g. we might create a `read-derivation` complement of `show-derivation` that accepts a nicer JSON representation of a derivation as input. +This will allow the store-only Nix to be tested in isolation, but these tests can also be used with full Nix. + # Examples and Interactions [examples-and-interactions]: #examples-and-interactions From 8fd8c47ced31208aff41c7aae6f858e0b3bff462 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 07:49:14 -0400 Subject: [PATCH 09/19] nix-store-layer: Give RFC number --- rfcs/{0000-nix-store-layer.md => 0134-nix-store-layer.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename rfcs/{0000-nix-store-layer.md => 0134-nix-store-layer.md} (100%) diff --git a/rfcs/0000-nix-store-layer.md b/rfcs/0134-nix-store-layer.md similarity index 100% rename from rfcs/0000-nix-store-layer.md rename to rfcs/0134-nix-store-layer.md From 008243f858a174f8ff6420280cb6ffa5f85df6ce Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 Sep 2022 10:33:11 -0400 Subject: [PATCH 10/19] nix-store-layer: Fix typo Meaning was reversed! --- rfcs/0134-nix-store-layer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 1474fe070..8039eb455 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -233,7 +233,7 @@ Now that we have this division in the implementation, we also have the opportuni An official, NixOS-foundation-authorized team could be set up to manage store layer design decisions (below the threshold of needing an RFC) that don't effect that the rest of Nix. \[Some sort of decision that affect all layers is out of scope, must be deliberated with stakeholders from other layers too, probably should be RFC due to such large scope.\] -To be clear, this is to say we should abandon the idea of Nix as a whole. +To be clear, this is *not* to say we should abandon the idea of Nix as a whole. There can still be governance of Nix as a whole; this team, and similar hypothetical, say, Flakes, Nix language, or User Experience teams would ultimately need to report to. The goal is not to overreact, but strike a balance between: From 5aa53f1d49848f1c1729951d8a97d116eed6bd6b Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 16 Sep 2022 16:30:34 -0500 Subject: [PATCH 11/19] Update rfcs/0134-nix-store-layer.md Thanks! Co-authored-by: Tor Bjornrud --- rfcs/0134-nix-store-layer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 8039eb455..644f69c64 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -36,7 +36,7 @@ We should instead fully embrace it: - More advanced documentation can explain layering for those that want a deeper understanding of Nix. - Even more basic documentation can still benefit from separate terminology before the layering is fully explained. - See https://www.haskellforall.com/2022/08/stop-calling-everything-nix.html for a phenomenal take-down of how calling everything "Nix" today confuses users and leaves them unable to articulate what parts of the ecosystem are frustrating then. + See https://www.haskellforall.com/2022/08/stop-calling-everything-nix.html for a phenomenal take-down of how calling everything "Nix" today confuses users and leaves them unable to articulate what parts of the ecosystem are frustrating. - Separate executable ensure lower layers build in isolation, new integration tests those executable without aid of high-layer info. From b8df7cc556697a8131815566409f9299e01c19cc Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 16 Sep 2022 16:35:50 -0500 Subject: [PATCH 12/19] nix-store-layer: Fix typo Thanks! Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com> --- rfcs/0134-nix-store-layer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 644f69c64..2cadcbb9e 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -230,7 +230,7 @@ What should the store-only Nix be called? ## Nix Store Team Now that we have this division in the implementation, we also have the opportunity to leverage it for governance purposes. -An official, NixOS-foundation-authorized team could be set up to manage store layer design decisions (below the threshold of needing an RFC) that don't effect that the rest of Nix. +An official, NixOS-foundation-authorized team could be set up to manage store layer design decisions (below the threshold of needing an RFC) that don't affect that the rest of Nix. \[Some sort of decision that affect all layers is out of scope, must be deliberated with stakeholders from other layers too, probably should be RFC due to such large scope.\] To be clear, this is *not* to say we should abandon the idea of Nix as a whole. From d8738d30b973c964df9b4a6e3eda0d2468bdf191 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 20 Sep 2022 19:12:50 -0500 Subject: [PATCH 13/19] Apply suggestions from code review Thanks so much!!! Co-authored-by: Valentin Gagarin --- rfcs/0134-nix-store-layer.md | 66 +++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 32 deletions(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 2cadcbb9e..012b15d6d 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -22,28 +22,30 @@ Do this to manage complexity, and promote a "Greater Nix" community. We have recently added a ton of functionality to Nix. Most of this is in Flakes, which just has enormous surface area, but things like floating content-addressed derivations and other RFCs add complexity to the core store layer too. -If we view Nix as one monolithic whole, it will grow too complex and unwieldy, and we will be unable to manage it as we the complexity bogs us down. +If we view Nix as one monolithic whole, it will grow too complex and unwieldy, and we will be unable to manage it as the complexity bogs us down. However, if we embrace layering we can "divide and conquer" the project, and manage that complexity. This will ensure the continued sustainability of Nix. We currently embrace layering somewhat as an implementation detail, but only as an implementation detail. -The division between `libnixstore`, `libfetchers`, `libexpr`, etc. is not yet exposed to users, emphasized in documentation (though this is changing thanks to @Fricklerhandwerk's efforts with new [architecture documentation](https://github.com/NixOS/nix/pull/6888)!). +The division between `libnixstore`, `libfetchers`, `libexpr`, etc. is not yet exposed to users, or emphasized in documentation (though this is changing thanks to @fricklerhandwerk's efforts with new [architecture documentation](https://github.com/NixOS/nix/pull/7066)!). We should instead fully embrace it: -- Docs +- Expand documentation - More advanced documentation can explain layering for those that want a deeper understanding of Nix. - Even more basic documentation can still benefit from separate terminology before the layering is fully explained. See https://www.haskellforall.com/2022/08/stop-calling-everything-nix.html for a phenomenal take-down of how calling everything "Nix" today confuses users and leaves them unable to articulate what parts of the ecosystem are frustrating. -- Separate executable ensure lower layers build in isolation, new integration tests those executable without aid of high-layer info. +- Introduce separate executables to ensure lower layers can be build in isolation, i.e., without requiring higher layers. + + - Add integration tests for those executables that don't require the higher layers, either. This is to ensure the lower-layer executables work correctly in isolation. ### Starting with the store layer -Ultimately, I would like to take this approach to all the layers. -But I want to constrain the scope of this RFC to keep it tight an actionable. +Ultimately, we should take this approach to all the layers. +The constrained scope here is to keep this RFC actionable. Layering between e.g. Flakes and the Nix Language doesn't yet exist in the implementation in the form of a library separation. I don't want to be "blocked" on major new development work, and anything involving Flakes is also far more controversial, and best avoided at first. @@ -52,7 +54,7 @@ Conversely, the store layer is already quite well separated. The orthogonality between it and other layers "proven" in the wild by projects like Guix (more on that latter part in the next section). The gap is also widest in terms of the layer of abstraction between, on one hand, nascent "infrastructure projects" like -- CA derivations +- content-addressed derivations - Secrets in the store - Trustless remote building - Daemon and Hydra protocol rationalization @@ -60,30 +62,30 @@ The gap is also widest in terms of the layer of abstraction between, on one hand - Computed derivations (which are built by the other derivations) - IPFS store -etc. and "UX projects" like +and projects aiming at improving user experience, such as - Flakes - TOML Flakes - Hard-wired module system -So focusing on the lowest layer first, we get the most "bang for buck" in terms of managing extremely different sorts of work separately. +Focusing on the lowest layer first, we get the most "bang for buck" in terms of managing extremely different sorts of work separately. ## Marketplace of Ideas As Nix grows more popularity, it will be inevitable that different groups want to explore in different directions. This is the *pluralism* of a larger community, and we should embrace it as a strength. -We do that be fostering a *marketplace of ideas*. +We do that by fostering a *marketplace of ideas*. -There are many possible ways in which to write down packages, The Nix language and Nixpkgs idioms, and Guix, for example, are just two points in a much larger space. +There are many possible ways in which to declare packages: the Nix language and Nixpkgs idioms, and Guix, for example, are just two examples of what is possible in principle. There are also many possible ways set up build farms. -Our current central dispatcher, many remote-builder agents model, point-to-point protocol model is also just point in a much larger design space. +Our current model of a central dispatcher, many remote-builder agents, and a client–server where only one side initiates, is also just one point in a much larger design space of possible solutions. The "derivation language" and store *interface* however, seems to me at least to be a very natural design. There are a few tweaks and generalizations we can make, but I struggle to envision what wildly different alternatives one might want instead. -An stable, small interface that fosters lots of design exploration above and below is known from networking as a *narrow waist*. -The oil shell blog as a [great post](https://www.oilshell.org/blog/2022/02/diagrams.html) with more details on the concept. -It's not every day that a project happens upon a great narrow waist design, but I believe we've discovered a very good one with Nix, and that should be seen as a *key asset*, even if it is not how we recruit "regular users". +A small, stable interface that allows for design exploration above and below it is known as a [narrow waist](https://www.oilshell.org/cross-ref.html?tag=narrow-waist#narrow-waist). +It's not every day that a project happens upon a great narrow waist design. +I believe we've discovered a very good one with Nix, and that should be seen as a *key asset*, even if it is not how we recruit "regular users". By making a store-only Nix, we put more emphasis on this key interface. All functionality the store-only Nix offers factors through this interface. @@ -97,13 +99,13 @@ To help explain the community-building benefits, it might help to go over some s In https://tvl.fyi/blog/rewriting-nix, TVL announced that, frustrated in trying to refactor Nix into something more modular and flexible, they were aiming to make a new implementation from scratch. More recently, in https://tvl.fyi/blog/tvix-status-september-22 they lay out a basic approach of two projects: -- [*Tvix*](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator, +- [Tvix](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator, -- [*go-nix*](https://github.com/nix-community/go-nix) is a new implementation of the store layer. +- [go-nix](https://github.com/nix-community/go-nix) is a new implementation of the store layer. First of all, the fact that they are planning on two completely separate implementation oriented around this same "narrow waist" is testament to the appeal of the design. -Second of all, note that per they have separate, orthogonal experiments they wish to run on both sides of the store interface divide. +Second of all, note that they have separate, orthogonal experiments they wish to run on both sides of the store interface divide. Above, they want to experiment with radically different evaluation strategies, especially to speed up Nixpkgs evaluation. Below, they want to experiment with the standardized containerization technologies that already exist for new ways of sandboxing and distributing builds with less bespoke Nix-specific code. They also want to apply the layering paradigm *within* go-nix, fostering even more modularity. @@ -117,20 +119,20 @@ Messaging matters, and making our layered architecture "official" as this RFC pr ### Guix -Guix is more diverged from Nix than Tvix + go-nix, and thus hints more at the end breadth of the design space yet to be explored. +Guix is further removed from Nix than Tvix + go-nix, and thus hints more at the end breadth of the design space yet to be explored. -The store layer is the same, but the layers above, instead of being a implementing of the Nix language, is a completely different design with Guile Scheme. +The store layer is the same, but the layers above, instead of being an alternative implementation of the Nix language, have a completely different design using Guile Scheme. The choice of language is just the tip of the iceberg here. More profoundly, they also have a more "library" than "interpreter" model where packages depend on Guix as a library, which talks to a small rump daemon. -Guile sits far lower in their stack than the Nix language interpreter does; it is as if we rewrote some of our C++ into nix language code, and nix language code could do enough side effects to make that possible. +Guile sits far lower in their stack than the Nix language interpreter does; it is as if we rewrote some of our C++ into Nix language code, and Nix language code could do enough side effects to make that possible. -The point of this discursion is to show that not only are radically different implementation of the same spec possible on either side of the store interface (what Tvix + go-nix aim for), but radically different designs not at all trying to be comparable also. +The point of this discursion is to show that not only are radically different implementation of the same specification possible on either side of the store interface (what Tvix + go-nix aim for), but also radically different designs not at all trying to be compatible. All that said, below the store layer there is no difference in vision. Because our communities are so separate, it would be easy to come up with diverging versions of how derivations, store objects, etc. should work. That we have not done so I think is testament to the broad applicability of the Nix store design to many diverse groups of people with diverse goals. -What hope to do with Guix, then, is convene both projects to make their store store layers interoperate. +What I hope to do with Guix, then, is convene both projects to make their store store layers interoperate. Complementing the idea of a "marketplace of ideas" is when there is a certain design (like the Nix store layer), that is so broadly popular as to be a sort of "natural monopoly", that we should foster the most expansive and general idea of it as an exercise in coalition building and outreach. I do not expect Guix to be immediately sold on this plan, but as that larger project, I think it behooves us to take the first steps to build trust and coordination. @@ -146,26 +148,26 @@ This keeps the cost of this work initially lower, and generally reduces risk. Allow building a store-only version of Nix. This is a Nix executable that links `libstore` but not `libfetchers`, or `libexpr`. -Plenty of commands like `nix daemon`, `nix log`, and the `nix store` sub-commands don't care about evaluation, fetching, or flakes at all. -https://github.com/NixOS/nix/issues/6182 is a draft PR implementing this, splitting `libcmd` into two parts so the CLI code reused. +Plenty of commands like `nix daemon`, `nix log`, and the `nix store` sub-commands don't care about evaluation, fetching, or Flakes at all. +[NixOS/nix#6182](https://github.com/NixOS/nix/issues/6182) implements this, splitting `libcmd` into two parts so the CLI code is reused. We will finish it off with as many commands as are reasonable to include, and merge it. Initially, we can test this store-only version of Nix with no changes to the test suite, by running full Nix with the store-only Nix's daemon. Support for testing Nix against a separately-built daemon already exists and is in use today. The store-only Nix and its tests should be built as part of CI, as "first class" as our existing CI jobs. -That means both in the channel-blocking Hydra eval, and per PR. +That means both in the channel-blocking Hydra evaluation, and for each pull request. If we hit the limits of Github Actions in per-PR CI, we should consider using Hydra instead / in addition, something that has already been discussed. ## 2. Manual -It should be possible to build a store-only manual without information on the other layers too. +It should be possible to build a store-only manual without information on the other layers, too. This would be the manual that is distributed with the store-only Nix. -Of course, store-only and full can share sections, so we aren't duplicating work. +Of course, store-only and full Nix can share sections, so we aren't duplicating work. ## 3. Website -The fully and store-only version of Nix should both be presented for download on the website. +The full and store-only version of Nix should both be presented for download on the website. This should be just like how Plasma, Gnome, and headless installer images for NixOS are all offered. ## 4. Store-specific Tests @@ -195,7 +197,7 @@ If you believe this, then the "cost" of this RFC is a lot less. ### Tests Splitting the test suite per natural layer of the implementation is good work because it combines the specificity of unit tests with the real-world-ness of integration tests. -"entire kitchen sink" tests make it harder to narrow down root causes of failures. +"Entire kitchen sink" tests make it harder to narrow down root causes of failures. ### Manual @@ -238,11 +240,11 @@ There can still be governance of Nix as a whole; this team, and similar hypothet The goal is not to overreact, but strike a balance between: 1. Making sure Nix as a whole continues to make sense -2. Make sure layers make sense in isolation not just in the context of the way they are currently used. +2. Make sure layers make sense in isolation, and not just in the context of the way they are currently used. ## Standardization across projects -If we establish informal interop across store-layer implementations with Guix, a next step would be establish some sort of living standard that both communities have equal say in. +If we establish informal interoperability across store-layer implementations with Guix, a next step would be establish some sort of living standard that both communities have equal say in. (Of course, implementations are free to implement features in excess of what the standard requires!) A new store team, per the above, could lead the process from our end, since the other parts of Nix are not shared with Guix and thus out of scope for this sort of cross-project standardization. From ec44f73762a7cbd996f342566e00b11c7b3149be Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 21 Sep 2022 15:50:46 -0400 Subject: [PATCH 14/19] Apply suggestions from code review Thanks @fricklerhandwerk, just tweaked a few things from your suggestions to make these. --- rfcs/0134-nix-store-layer.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 012b15d6d..981d08780 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -48,10 +48,11 @@ Ultimately, we should take this approach to all the layers. The constrained scope here is to keep this RFC actionable. Layering between e.g. Flakes and the Nix Language doesn't yet exist in the implementation in the form of a library separation. -I don't want to be "blocked" on major new development work, and anything involving Flakes is also far more controversial, and best avoided at first. +Separating Flakes into its own library would require major new development work, and I don't want to "block" exposing the layering for the first time to users on such work when there is an easier way to try this idea how. +Flakes also has more development going on due being unstable, so it is nice to not get in its way. Conversely, the store layer is already quite well separated. -The orthogonality between it and other layers "proven" in the wild by projects like Guix (more on that latter part in the next section). +Its conceptual independence from Nix's other layers is "proven" in the wild by projects like Guix. The gap is also widest in terms of the layer of abstraction between, on one hand, nascent "infrastructure projects" like - content-addressed derivations @@ -96,8 +97,8 @@ To help explain the community-building benefits, it might help to go over some s ### Tvix and go-nix -In https://tvl.fyi/blog/rewriting-nix, TVL announced that, frustrated in trying to refactor Nix into something more modular and flexible, they were aiming to make a new implementation from scratch. -More recently, in https://tvl.fyi/blog/tvix-status-september-22 they lay out a basic approach of two projects: +[TVL announced](https://tvl.fyi/blog/rewriting-nix) that, frustrated in trying to refactor Nix into something more modular and flexible, they were aiming to make a new implementation from scratch. +More recently, [they lay out a basic approach](https://tvl.fyi/blog/tvix-status-september-22) of two projects: - [Tvix](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator, @@ -157,7 +158,7 @@ Support for testing Nix against a separately-built daemon already exists and is The store-only Nix and its tests should be built as part of CI, as "first class" as our existing CI jobs. That means both in the channel-blocking Hydra evaluation, and for each pull request. -If we hit the limits of Github Actions in per-PR CI, we should consider using Hydra instead / in addition, something that has already been discussed. +If we hit the limits of GitHub Actions in per-pull-request CI, we should consider using Hydra either instead or in addition to GitHub Actions, something that has already been discussed. ## 2. Manual From 063f872075e80874a802eee506d858173c1052cd Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 3 Oct 2022 12:45:36 -0400 Subject: [PATCH 15/19] nix-store-layer: Cut stabilization in future work down Can just refer to https://github.com/NixOS/rfcs/pull/136 now. --- rfcs/0134-nix-store-layer.md | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 981d08780..e062dd9fd 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -253,22 +253,5 @@ A new store team, per the above, could lead the process from our end, since the ## Stabilization There is a looming question on how to stabilize Nix's big backlog of unstable features (New CLI, Flakes). -There is a lot of bad-blood over Flakes, both the feature itself and the way it has been rolled out. -I think the stabilization process can be an opportunity to heal old wounds. - -At a minimum, this can involve stabilizing the new CLI before Flakes. -But even that that is a lot of new feature surface area to review. -I think even better is stabilizing just the store-only new CLI first. - -This is easily the least controversial part of our unstable feature backlog, and yet there is still plenty to discuss. -Questions like - -- logging -- store paths on `stdout` at end of build? -- Should commands like `show-derivation` should use `--json` by default -- Flat vs hierarchical commands -- is `--derivation` a good flag? (I think not!) - -are all in-scope. - -Having a conversation just on this narrow first batch of stabilization both builds trust, and ensures these still-important issues they aren't lost in flame wars over more divisive topics. +[RFC 136](https://github.com/NixOS/rfcs/pull/136) is a plan on how to stabilize these features that greatly leverages this. +See it for further details. From 7b6cdbc5e1e72ada2d25f56cd53ce723081ea16d Mon Sep 17 00:00:00 2001 From: John Ericson Date: Fri, 18 Nov 2022 08:24:29 -0500 Subject: [PATCH 16/19] Apply suggestions from code review Co-authored-by: Linus Heckemann Co-authored-by: Valentin Gagarin --- rfcs/0134-nix-store-layer.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index e062dd9fd..0c44e02e1 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -133,7 +133,7 @@ All that said, below the store layer there is no difference in vision. Because our communities are so separate, it would be easy to come up with diverging versions of how derivations, store objects, etc. should work. That we have not done so I think is testament to the broad applicability of the Nix store design to many diverse groups of people with diverse goals. -What I hope to do with Guix, then, is convene both projects to make their store store layers interoperate. +What I hope to do with Guix, then, is convene both projects to make their store layers interoperate. Complementing the idea of a "marketplace of ideas" is when there is a certain design (like the Nix store layer), that is so broadly popular as to be a sort of "natural monopoly", that we should foster the most expansive and general idea of it as an exercise in coalition building and outreach. I do not expect Guix to be immediately sold on this plan, but as that larger project, I think it behooves us to take the first steps to build trust and coordination. @@ -209,7 +209,7 @@ So whether we formally make a store-only Nix or not, I suspect the overhauled ma The daemon is a privileged process. Even if with upcoming changes it shouldn't need root, it does tasks like administrating OS sandboxes correctly which still are security-critical. -Having less code in the story-only Nix daemon, even if we think the removed code was "dead anyways" is always good. +Having less code in the store-only Nix daemon, even if we think the removed code was "dead anyways" is always good. # Drawbacks [drawbacks]: #drawbacks @@ -234,7 +234,7 @@ What should the store-only Nix be called? Now that we have this division in the implementation, we also have the opportunity to leverage it for governance purposes. An official, NixOS-foundation-authorized team could be set up to manage store layer design decisions (below the threshold of needing an RFC) that don't affect that the rest of Nix. -\[Some sort of decision that affect all layers is out of scope, must be deliberated with stakeholders from other layers too, probably should be RFC due to such large scope.\] +Decisions that would affect all layers is out of scope for that team, must be deliberated with stakeholders from other layers too, and probably should be RFCs. To be clear, this is *not* to say we should abandon the idea of Nix as a whole. There can still be governance of Nix as a whole; this team, and similar hypothetical, say, Flakes, Nix language, or User Experience teams would ultimately need to report to. From 5bc6ea6b113cb45ad1a095478e04b6c0973183f8 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 11 Jan 2023 11:00:11 -0500 Subject: [PATCH 17/19] 134: We have a shepherd team! Co-authored-by: Linus Heckemann --- rfcs/0134-nix-store-layer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 0c44e02e1..bb508ff75 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -3,8 +3,8 @@ feature: nix-store-layer start-date: 2022-09-06 author: John Ericson co-authors: (find a buddy later to help out with the RFC) -shepherd-team: (names, to be nominated and accepted by RFC steering committee) -shepherd-leader: (name to be appointed by RFC steering committee) +shepherd-team: edolstra, infinisil, tomberek, L-as, andir +shepherd-leader: edolstra related-issues: (will contain links to implementation PRs) --- From cae00b16e041f04b1882cc4a5917b1ab2872ac20 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 23 Jan 2023 20:49:07 -0500 Subject: [PATCH 18/19] 134: Move manual and store-specific tests to future work As discussed in today's meeting. --- rfcs/0134-nix-store-layer.md | 35 ++++++++++++++++------------------- 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index bb508ff75..438506588 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -142,11 +142,6 @@ Making a stand-alone Nix store executable demonstrates we are serious about laye # Detailed design [design]: #detailed-design -The goals motivated above a broken down into small steps that we can execute in isolation. -This keeps the cost of this work initially lower, and generally reduces risk. - -## 1. The split itself - Allow building a store-only version of Nix. This is a Nix executable that links `libstore` but not `libfetchers`, or `libexpr`. Plenty of commands like `nix daemon`, `nix log`, and the `nix store` sub-commands don't care about evaluation, fetching, or Flakes at all. @@ -160,24 +155,13 @@ The store-only Nix and its tests should be built as part of CI, as "first class" That means both in the channel-blocking Hydra evaluation, and for each pull request. If we hit the limits of GitHub Actions in per-pull-request CI, we should consider using Hydra either instead or in addition to GitHub Actions, something that has already been discussed. -## 2. Manual - -It should be possible to build a store-only manual without information on the other layers, too. -This would be the manual that is distributed with the store-only Nix. -Of course, store-only and full Nix can share sections, so we aren't duplicating work. - -## 3. Website +For initial testing, we can run the regular nix as a client against the daemon from the store-only Nix. +This will ensure the code is somewhat covered without requiring any new tests. +(The infrastructure for testing with an arbitrary daemon already exists.) The full and store-only version of Nix should both be presented for download on the website. This should be just like how Plasma, Gnome, and headless installer images for NixOS are all offered. -## 4. Store-specific Tests - -The current test suite uses Nix language for most tests of store-layer functionality. -We should write new tests that don't use the Nix language. -E.g. we might create a `read-derivation` complement of `show-derivation` that accepts a nicer JSON representation of a derivation as input. -This will allow the store-only Nix to be tested in isolation, but these tests can also be used with full Nix. - # Examples and Interactions [examples-and-interactions]: #examples-and-interactions @@ -230,6 +214,19 @@ What should the store-only Nix be called? # Future work [future]: #future-work +## Store-specific Tests + +The current test suite uses Nix language for most tests of store-layer functionality. +We should write new tests that don't use the Nix language. +E.g. we might create a `read-derivation` complement of `show-derivation` that accepts a nicer JSON representation of a derivation as input. +This will allow the store-only Nix to be tested in isolation, but these tests can also be used with full Nix. + +## Manual + +It should be possible to build a store-only manual without information on the other layers, too. +This would be the manual that is distributed with the store-only Nix. +Of course, store-only and full Nix can share sections, so we aren't duplicating work. + ## Nix Store Team Now that we have this division in the implementation, we also have the opportunity to leverage it for governance purposes. From 3194ad43a5c302b5b261eafb25c0f1426203dbee Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 1 Feb 2023 17:17:37 -0500 Subject: [PATCH 19/19] 134: Alt name moved to future work --- rfcs/0134-nix-store-layer.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/rfcs/0134-nix-store-layer.md b/rfcs/0134-nix-store-layer.md index 438506588..752611c9d 100644 --- a/rfcs/0134-nix-store-layer.md +++ b/rfcs/0134-nix-store-layer.md @@ -209,7 +209,7 @@ But that feels to me like turning an ongoing shift in focus to a one-off change # Unresolved questions [unresolved]: #unresolved-questions -What should the store-only Nix be called? +None at this time # Future work [future]: #future-work @@ -247,6 +247,12 @@ If we establish informal interoperability across store-layer implementations wit A new store team, per the above, could lead the process from our end, since the other parts of Nix are not shared with Guix and thus out of scope for this sort of cross-project standardization. +### Naming + +A standard shared between Nix, Guix, and other such projects presumably shouldn't be called "Nix" or even "Nix Store". +Otherwise those other projects' communities will feel like second-class stakeholders. +We can find a "neutral" name for the store layer that doesn't bias one project over another as part of establishing the standard. + ## Stabilization There is a looming question on how to stabilize Nix's big backlog of unstable features (New CLI, Flakes).