Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0134] Carve out a store-only Nix #134

Merged
merged 19 commits into from
Feb 20, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply suggestions from code review
Thanks so much!!!

Co-authored-by: Valentin Gagarin <[email protected]>
  • Loading branch information
Ericson2314 and fricklerhandwerk authored Sep 21, 2022
commit d8738d30b973c964df9b4a6e3eda0d2468bdf191
66 changes: 34 additions & 32 deletions rfcs/0134-nix-store-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,28 +22,30 @@ Do this to manage complexity, and promote a "Greater Nix" community.
We have recently added a ton of functionality to Nix.
Most of this is in Flakes, which just has enormous surface area, but things like floating content-addressed derivations and other RFCs add complexity to the core store layer too.

If we view Nix as one monolithic whole, it will grow too complex and unwieldy, and we will be unable to manage it as we the complexity bogs us down.
If we view Nix as one monolithic whole, it will grow too complex and unwieldy, and we will be unable to manage it as the complexity bogs us down.
However, if we embrace layering we can "divide and conquer" the project, and manage that complexity.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"layering" does not really capture what this is about.

Suggested change
However, if we embrace layering we can "divide and conquer" the project, and manage that complexity.
However, if we define Nix's internal architecture more clearly, we can "divide and conquer" the project, and manage that complexity.

This will ensure the continued sustainability of Nix.

We currently embrace layering somewhat as an implementation detail, but only as an implementation detail.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The layered architecture of Nix is not very prominent.

Copy link
Member Author

@Ericson2314 Ericson2314 Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
We currently embrace layering somewhat as an implementation detail, but only as an implementation detail.
We currently care about layering enough to partition the implementation into separate `libnix*` libraries, but this is only an implementation detail.

how is this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. (Although I would prefer to phrase it as separation of concerns instead of layering, but that may be me having Heard- and read "layering" way too often.)

The division between `libnixstore`, `libfetchers`, `libexpr`, etc. is not yet exposed to users, emphasized in documentation (though this is changing thanks to @Fricklerhandwerk's efforts with new [architecture documentation](https://github.com/NixOS/nix/pull/6888)!).
The division between `libnixstore`, `libfetchers`, `libexpr`, etc. is not yet exposed to users, or emphasized in documentation (though this is changing thanks to @fricklerhandwerk's efforts with new [architecture documentation](https://github.com/NixOS/nix/pull/7066)!).

We should instead fully embrace it:

- Docs
- Expand documentation

- More advanced documentation can explain layering for those that want a deeper understanding of Nix.

- Even more basic documentation can still benefit from separate terminology before the layering is fully explained.
See https://www.haskellforall.com/2022/08/stop-calling-everything-nix.html for a phenomenal take-down of how calling everything "Nix" today confuses users and leaves them unable to articulate what parts of the ecosystem are frustrating.

- Separate executable ensure lower layers build in isolation, new integration tests those executable without aid of high-layer info.
- Introduce separate executables to ensure lower layers can be build in isolation, i.e., without requiring higher layers.

- Add integration tests for those executables that don't require the higher layers, either. This is to ensure the lower-layer executables work correctly in isolation.

### Starting with the store layer

Ultimately, I would like to take this approach to all the layers.
But I want to constrain the scope of this RFC to keep it tight an actionable.
Ultimately, we should take this approach to all the layers.
The constrained scope here is to keep this RFC actionable.

Layering between e.g. Flakes and the Nix Language doesn't yet exist in the implementation in the form of a library separation.
I don't want to be "blocked" on major new development work, and anything involving Flakes is also far more controversial, and best avoided at first.
Expand All @@ -52,38 +54,38 @@ Conversely, the store layer is already quite well separated.
The orthogonality between it and other layers "proven" in the wild by projects like Guix (more on that latter part in the next section).
The gap is also widest in terms of the layer of abstraction between, on one hand, nascent "infrastructure projects" like

- CA derivations
- content-addressed derivations
- Secrets in the store
- Trustless remote building
- Daemon and Hydra protocol rationalization
- Windows Support
- Computed derivations (which are built by the other derivations)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that a term? I think in the Bazel/Buck world they are called dynamic dependencies.

- IPFS store

etc. and "UX projects" like
and projects aiming at improving user experience, such as

- Flakes
- TOML Flakes
- Hard-wired module system

So focusing on the lowest layer first, we get the most "bang for buck" in terms of managing extremely different sorts of work separately.
Focusing on the lowest layer first, we get the most "bang for buck" in terms of managing extremely different sorts of work separately.

## Marketplace of Ideas

As Nix grows more popularity, it will be inevitable that different groups want to explore in different directions.
This is the *pluralism* of a larger community, and we should embrace it as a strength.
We do that be fostering a *marketplace of ideas*.
We do that by fostering a *marketplace of ideas*.

There are many possible ways in which to write down packages, The Nix language and Nixpkgs idioms, and Guix, for example, are just two points in a much larger space.
There are many possible ways in which to declare packages: the Nix language and Nixpkgs idioms, and Guix, for example, are just two examples of what is possible in principle.
There are also many possible ways set up build farms.
Our current central dispatcher, many remote-builder agents model, point-to-point protocol model is also just point in a much larger design space.
Our current model of a central dispatcher, many remote-builder agents, and a client–server where only one side initiates, is also just one point in a much larger design space of possible solutions.

The "derivation language" and store *interface* however, seems to me at least to be a very natural design.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By derivation language, you mean the ATerm-based language that drv files are written in?

Copy link
Member Author

@Ericson2314 Ericson2314 Sep 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The abstract syntax of that. Basically "a derivation is an execve system call arguments (prog, args, env vars), input black box data, and input other drvs' outputs". I think that right there is a very natural design.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That terms needs a more formal introduction at some point, I think. I wrote about it in this blog post, if you care about a more detailed explanation.

There are a few tweaks and generalizations we can make, but I struggle to envision what wildly different alternatives one might want instead.

An stable, small interface that fosters lots of design exploration above and below is known from networking as a *narrow waist*.
The oil shell blog as a [great post](https://www.oilshell.org/blog/2022/02/diagrams.html) with more details on the concept.
It's not every day that a project happens upon a great narrow waist design, but I believe we've discovered a very good one with Nix, and that should be seen as a *key asset*, even if it is not how we recruit "regular users".
A small, stable interface that allows for design exploration above and below it is known as a [narrow waist](https://www.oilshell.org/cross-ref.html?tag=narrow-waist#narrow-waist).
It's not every day that a project happens upon a great narrow waist design.
I believe we've discovered a very good one with Nix, and that should be seen as a *key asset*, even if it is not how we recruit "regular users".

By making a store-only Nix, we put more emphasis on this key interface.
All functionality the store-only Nix offers factors through this interface.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what to make of this sentence. Is this what you mean? https://en.wiktionary.org/wiki/factor_through

Maybe we can just remove it?

Suggested change
All functionality the store-only Nix offers factors through this interface.

Expand All @@ -97,13 +99,13 @@ To help explain the community-building benefits, it might help to go over some s
In https://tvl.fyi/blog/rewriting-nix, TVL announced that, frustrated in trying to refactor Nix into something more modular and flexible, they were aiming to make a new implementation from scratch.
More recently, in https://tvl.fyi/blog/tvix-status-september-22 they lay out a basic approach of two projects:

- [*Tvix*](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator,
- [Tvix](https://cs.tvl.fyi/depot/-/tree/tvix) is a new implementation of the Nix language evaluator,

- [*go-nix*](https://github.com/nix-community/go-nix) is a new implementation of the store layer.
- [go-nix](https://github.com/nix-community/go-nix) is a new implementation of the store layer.

First of all, the fact that they are planning on two completely separate implementation oriented around this same "narrow waist" is testament to the appeal of the design.

Second of all, note that per they have separate, orthogonal experiments they wish to run on both sides of the store interface divide.
Second of all, note that they have separate, orthogonal experiments they wish to run on both sides of the store interface divide.
Above, they want to experiment with radically different evaluation strategies, especially to speed up Nixpkgs evaluation.
Below, they want to experiment with the standardized containerization technologies that already exist for new ways of sandboxing and distributing builds with less bespoke Nix-specific code.
They also want to apply the layering paradigm *within* go-nix, fostering even more modularity.
Expand All @@ -117,20 +119,20 @@ Messaging matters, and making our layered architecture "official" as this RFC pr

### Guix

Guix is more diverged from Nix than Tvix + go-nix, and thus hints more at the end breadth of the design space yet to be explored.
Guix is further removed from Nix than Tvix + go-nix, and thus hints more at the end breadth of the design space yet to be explored.

The store layer is the same, but the layers above, instead of being a implementing of the Nix language, is a completely different design with Guile Scheme.
The store layer is the same, but the layers above, instead of being an alternative implementation of the Nix language, have a completely different design using Guile Scheme.
The choice of language is just the tip of the iceberg here.
More profoundly, they also have a more "library" than "interpreter" model where packages depend on Guix as a library, which talks to a small rump daemon.
Guile sits far lower in their stack than the Nix language interpreter does; it is as if we rewrote some of our C++ into nix language code, and nix language code could do enough side effects to make that possible.
Guile sits far lower in their stack than the Nix language interpreter does; it is as if we rewrote some of our C++ into Nix language code, and Nix language code could do enough side effects to make that possible.

The point of this discursion is to show that not only are radically different implementation of the same spec possible on either side of the store interface (what Tvix + go-nix aim for), but radically different designs not at all trying to be comparable also.
The point of this discursion is to show that not only are radically different implementation of the same specification possible on either side of the store interface (what Tvix + go-nix aim for), but also radically different designs not at all trying to be compatible.

All that said, below the store layer there is no difference in vision.
Because our communities are so separate, it would be easy to come up with diverging versions of how derivations, store objects, etc. should work.
That we have not done so I think is testament to the broad applicability of the Nix store design to many diverse groups of people with diverse goals.

What hope to do with Guix, then, is convene both projects to make their store store layers interoperate.
What I hope to do with Guix, then, is convene both projects to make their store store layers interoperate.
Complementing the idea of a "marketplace of ideas" is when there is a certain design (like the Nix store layer), that is so broadly popular as to be a sort of "natural monopoly", that we should foster the most expansive and general idea of it as an exercise in coalition building and outreach.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this sentence hard to understand, I think it tries to make too many points at once.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How's this?

Suggested change
Complementing the idea of a "marketplace of ideas" is when there is a certain design (like the Nix store layer), that is so broadly popular as to be a sort of "natural monopoly", that we should foster the most expansive and general idea of it as an exercise in coalition building and outreach.
Complementing the idea of a "marketplace of ideas" is when there is a certain design (like the Nix store layer), that is so broadly popular as to be a sort of "natural monopoly".
Rather than keep that designed "bundled" with more opinionated ideas (like the rest of Nix), we should foster the most expansive and general idea of the broadly popular idea as an exercise in outreach to other communities and coalition building with them.


I do not expect Guix to be immediately sold on this plan, but as that larger project, I think it behooves us to take the first steps to build trust and coordination.
Expand All @@ -146,26 +148,26 @@ This keeps the cost of this work initially lower, and generally reduces risk.

Allow building a store-only version of Nix.
This is a Nix executable that links `libstore` but not `libfetchers`, or `libexpr`.
Plenty of commands like `nix daemon`, `nix log`, and the `nix store` sub-commands don't care about evaluation, fetching, or flakes at all.
https://github.com/NixOS/nix/issues/6182 is a draft PR implementing this, splitting `libcmd` into two parts so the CLI code reused.
Plenty of commands like `nix daemon`, `nix log`, and the `nix store` sub-commands don't care about evaluation, fetching, or Flakes at all.
[NixOS/nix#6182](https://github.com/NixOS/nix/issues/6182) implements this, splitting `libcmd` into two parts so the CLI code is reused.
We will finish it off with as many commands as are reasonable to include, and merge it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move the store-only parts of Nix into a separate drv output where non-dependency on the "full" output is enforced?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now it would just be a separate derivation, but it would be nice to try to build both where possible.

Separate outputs might play funny with NixOS but I think we can. We can also just build libraries and exes in separate derivations.


Initially, we can test this store-only version of Nix with no changes to the test suite, by running full Nix with the store-only Nix's daemon.
Support for testing Nix against a separately-built daemon already exists and is in use today.

The store-only Nix and its tests should be built as part of CI, as "first class" as our existing CI jobs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no reason to emphasize this.

Suggested change
The store-only Nix and its tests should be built as part of CI, as "first class" as our existing CI jobs.
The store-only Nix and its tests should be built as part of CI.

That means both in the channel-blocking Hydra eval, and per PR.
That means both in the channel-blocking Hydra evaluation, and for each pull request.
Copy link
Member

@lheckemann lheckemann Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the store-only Nix as the for NixOS's Nix daemon, and run Nix's daemon tests against a store-only-built daemon? Once we have the layering, I see no reason not to use only the necessary parts wherever possible.

EDIT: the later part about tests addresses this I guess.

If we hit the limits of Github Actions in per-PR CI, we should consider using Hydra instead / in addition, something that has already been discussed.

## 2. Manual

It should be possible to build a store-only manual without information on the other layers too.
It should be possible to build a store-only manual without information on the other layers, too.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
It should be possible to build a store-only manual without information on the other layers, too.
It should be possible to build a store-only manual without information on the other layers, too.

It should be possible, but not necessary for us to implement, build, host, etc.

This would be the manual that is distributed with the store-only Nix.
Of course, store-only and full can share sections, so we aren't duplicating work.
Of course, store-only and full Nix can share sections, so we aren't duplicating work.

## 3. Website

The fully and store-only version of Nix should both be presented for download on the website.
The full and store-only version of Nix should both be presented for download on the website.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What exactly do you mean by presented? Provided as binaries or discussed/explained/promoted?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, provided downloads and discuss/explain what they are, but not necessary promote.

I am not going as far as saying they should be given equal billing. It's OK if users just download the entire Nix and aren't forced to learn about the layering. We can revisit that later if this is a success :).

This should be just like how Plasma, Gnome, and headless installer images for NixOS are all offered.

## 4. Store-specific Tests
Expand Down Expand Up @@ -195,7 +197,7 @@ If you believe this, then the "cost" of this RFC is a lot less.
### Tests

Splitting the test suite per natural layer of the implementation is good work because it combines the specificity of unit tests with the real-world-ness of integration tests.
"entire kitchen sink" tests make it harder to narrow down root causes of failures.
"Entire kitchen sink" tests make it harder to narrow down root causes of failures.

### Manual

Expand Down Expand Up @@ -238,11 +240,11 @@ There can still be governance of Nix as a whole; this team, and similar hypothet
The goal is not to overreact, but strike a balance between:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The goal is not to overreact, but strike a balance between:
The goal is to strike a balance between:


1. Making sure Nix as a whole continues to make sense
2. Make sure layers make sense in isolation not just in the context of the way they are currently used.
2. Make sure layers make sense in isolation, and not just in the context of the way they are currently used.

## Standardization across projects
Copy link

@arcuru arcuru Sep 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you reached out to Guix yet? When do you think it would be valuable to get their input?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have! But I think for this RFC the ball is purely in our court. This RFC is not about making technical decisions that would impact Guix, but about ratifying the idea of a store layer that is usable in arbitrary ways including the Nix language and Scheme via Guix.

I'll continue talking to them, and I encourage anyone else that wants to reach and say hi too — let's build more bridges between our communities — but if we don't do something like this I don't see why they should take us very seriously about store layer interop.


If we establish informal interop across store-layer implementations with Guix, a next step would be establish some sort of living standard that both communities have equal say in.
If we establish informal interoperability across store-layer implementations with Guix, a next step would be establish some sort of living standard that both communities have equal say in.
(Of course, implementations are free to implement features in excess of what the standard requires!)

A new store team, per the above, could lead the process from our end, since the other parts of Nix are not shared with Guix and thus out of scope for this sort of cross-project standardization.
Expand Down