Support arbitrary stores in Perl bindings #9863

Ericson2314 · 2024-01-26T21:47:06Z

Motivation

Fix #9859

Context

It's a breaking change but that's fine; we can just update Hydra to use the new bindings.

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

CC @Ma27

perl/lib/Nix/Store.xs

Ericson2314 · 2024-01-29T15:46:01Z

I'll keep this a draft until we have a draft of the Hydra changes ready to go, CC @Ma27.

Ma27 · 2024-02-04T10:23:54Z

perl/t/init.t

+
+use Nix::Store;
+
+my $s = new Nix::Store("dummy://");


fwiw perlcritic complains about that and suggests to use Nix::Store->new() instead (noticed that while modifying Hydra for that).

Fine with me! I don't know Perl enough to disagree :)

Implements support for Nix's new Perl bindings[1]. The current state basically does `openStore()`, but always uses `auto` and doesn't support stores at other URIs. Even though the stores are cached inside the Perl implementation, I decided to instantiate those once in the Nix helper module. That way store openings aren't cluttered across the entire codebase. Also, there are two stores used later on - MACHINE_LOCAL_STORE for `auto`, BINARY_CACHE_STORE for the one from `store_uri` in `hydra.conf` - and using consistent names should make the intent clearer then. This doesn't contain any behavioral changes, i.e. the build product availability issue from NixOS#1352 isn't fixed. This patch only contains the migration to the new API. [1] NixOS/nix#9863

Ericson2314 · 2024-02-07T14:29:51Z

NixOS/hydra#1359 is here, so undrafting!

tomberek

does newRV_noinc need to be available, got compiled away?

I'm getting:

$ perl -I ./result/lib/perl5/site_perl/5.36.0 -e 'use Nix::Store;'
Can't load './result/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi/auto/Nix/Store/Store.so' for module Nix::Store: ./result/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi/auto/Nix/Store/Store.so: undefined symbol: Perl_newRV_noinc at /nix/store/sjm6zhcjq1l6ghcal55vmpy48zd0rhm7-perl-5.38.2/lib/perl5/5.38.2/x86_64-linux-thread-multi/DynaLoader.pm line 206.
 at -e line 1.
Compilation failed in require at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

(or i'm using perl wrong, very likely)

Fix NixOS#9859 It's a breaking change but that's fine; we can just update Hydra to use the new bindings.

Ericson2314 · 2024-02-08T03:32:33Z

Not really sure @tomberek, I don't know it well either. I updated the pm file maybe that will make a difference.

edolstra · 2024-02-08T09:16:43Z

perl/lib/Nix/Store.xs

+            if (!libStoreInitialized) {
+                initLibStore();


initLibStore() internally already has a initLibStoreDone flag, so libStoreInitialized seems superfluous.

That seems to be for a slightly different purpose? It doesn't make it run one, it just makes sure you ran it at least once.

(It could work the way this does, but I rather do that in a follow-up PR.)

edolstra · 2024-02-08T09:18:05Z

perl/lib/Nix/Store.xs

+                libStoreInitialized = true;
+            }
+            if (items == 1) {
+                _store = openStore();


It's nicer to use the static initializer trick (because it's atomic), i.e.

static auto _store = {( openStore(); )}; RETVAL = new StoreWrapper { .store = ref<Store>{_store} };

Wouldn't that mean the default store is always opened?

Ericson2314 · 2024-02-08T14:30:15Z

Because my two questions, I think it would be better to come back to those points in a follow-up PR.

Implements support for Nix's new Perl bindings[1]. The current state basically does `openStore()`, but always uses `auto` and doesn't support stores at other URIs. Even though the stores are cached inside the Perl implementation, I decided to instantiate those once in the Nix helper module. That way store openings aren't cluttered across the entire codebase. Also, there are two stores used later on - MACHINE_LOCAL_STORE for `auto`, BINARY_CACHE_STORE for the one from `store_uri` in `hydra.conf` - and using consistent names should make the intent clearer then. This doesn't contain any behavioral changes, i.e. the build product availability issue from NixOS#1352 isn't fixed. This patch only contains the migration to the new API. [1] NixOS/nix#9863

Implements support for Nix's new Perl bindings[1]. The current state basically does `openStore()`, but always uses `auto` and doesn't support stores at other URIs. Even though the stores are cached inside the Perl implementation, I decided to instantiate those once in the Nix helper module. That way store openings aren't cluttered across the entire codebase. Also, there are two stores used later on - MACHINE_LOCAL_STORE for `auto`, BINARY_CACHE_STORE for the one from `store_uri` in `hydra.conf` - and using consistent names should make the intent clearer then. This doesn't contain any behavioral changes, i.e. the build product availability issue from #1352 isn't fixed. This patch only contains the migration to the new API. [1] NixOS/nix#9863

nixos-discourse · 2024-09-18T03:00:05Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-09-16-nix-team-meeting-minutes-178-177/52270/4

- NixOS/nix#9863 - NixOS/nix@bc08502 However, the library before and after this change exposes the same library version since it was [introduced] in Nix 1.0 which dificults writing compat code directly on nix-serve. [introduced]: NixOS/nix@73fe687

- NixOS/nix#9863 - NixOS/nix@bc08502 However, the library before and after this change exposes the same library version since it was [introduced] in Nix 1.0 which difficults writing compat code directly on nix-serve. [introduced]: NixOS/nix@73fe687

When NixOS#9863 converted these free functions into member functions, the implicit `this` argument was not accounted for when iterating over the variable number of arguments.

When NixOS#9863 converted the `Nix::Store` free functions into member functions, the implicit `this` argument was not accounted for when iterating over the variable number of arguments.

When NixOS#9863 converted the `Nix::Store` free functions into member functions, the implicit `this` argument was not accounted for when iterating over the variable number of arguments in some functions.

When #9863 converted the `Nix::Store` free functions into member functions, the implicit `this` argument was not accounted for when iterating over the variable number of arguments in some functions. (cherry picked from commit 5cf9e18)

@roberth

* Add FIXME * Update tests/functional/flakes/symlink-paths.sh * Add `inputs.self.submodules` flake attribute This allows a flake to specify that it needs Git submodules to be enabled (or disabled, if we ever change the default) on the top-level flake. This requires the input to be refetched, but since the first fetch is lazy, this shouldn't be expensive. Currently the only attribute allowed by `inputs.self` is `submodules`, but more can be added in the future (e.g. a `lazy` attribute to opt in to lazy tree behaviour). Fixes NixOS#5312, NixOS#9842. * Add a test for a flake referencing a flake that has inputs.self * nix-profile.fish: set --local NIX_LINK Using `set --local` is better than using `set`/`set --erase`. `--local` will preserve any existing `NIX_LINK` value. And the local variable is automatically removed for any execution path. * nix-profile.fish: Look for ca-bundle.crt in $NIX_PROFILES There seems to be no good reason for `nix-profile.fish` and `nix-profile-daemon.fish` to differ in how they look for the location of the `ca-bundle.crt` that might be installed by one of the packages. As `$NIX_PROFILES` points to user local paths, not checking there is strictly less useful, it seems? * Revert "Revert "Adapt scheduler to work with dynamic derivations"" This fixes dynamic derivations, reverting NixOS#9081. I believe that this time around, NixOS#9052 is fixed. When I first rebased this, tests were failing (which wasn't the case before). The cause of those test failures were due to the crude job in which the outer goal tried to exit with the inner goal's status. Now, that error handling has been reworked to be more faithful. The exit exit status and exception of the inner goal is returned by the outer goal. The exception was what was causing the test failures, but I believe it was not having the right error code (there is more than one for failure) that caused NixOS#9081. The only cost of doing things the "right way" was that I had to introduce a hacky `preserveException` boolean. I don't like this, but, then again, none of us like anything about how the scheduler works. Issue NixOS#11927 is still there to clean everything up, subsuming the need for any `preserveException` because I doubt we will be fishing information out of state machines like this at all. This reverts commit 8440afb. Co-Authored-By: Eelco Dolstra <[email protected]> * Add BLAKE3 hashing algorithm This uses the single-threaded C-based routines from libblake3. This is not optimal performance-wise but should be a good starting point for nix compatibility with BLAKE3 hashing until a more performant implementation based on the multi-threaded BLAKE3 routines (written in Rust) can be developed. * Add BLAKE3 to documentation * Parser: Respect the accessor of the source file for relative paths * Parser: Respect the accessor of the source file for relative paths Previously we only returned paths in rootFS, which is wrong and only worked because currently all our source trees are in rootFS. * pre-commit/check-merge-conflicts-2: fix use outside dev shell Note that this is just a script that is meant to run outside a derivation (but also can be called by a derivation builder). `touch $out` does not belong in it. `touch $out` worked accidentally in the derivation-based check, and also in the dev shell, but if pre-commit is invoked without the dev shell it would fail. * test: Fix shellcheck by giving git-hashing scripts shebangs This seems to be the way to do it now, even though I can't run them without setting at least one env var. I'll only fix shellcheck for now. Don't shoot the messenger. It isn't quite clear to me why the previous commit masked this problem, but I'm glad shellcheck has an effect or more effect now. * test: Use skipTest instead of exit 0 This way shellcheck is ok with it, and it conveys that a significant chunk of the test is skipped. * Set FD_CLOEXEC on sockets created by curl Curl creates sockets without setting FD_CLOEXEC/SOCK_CLOEXEC, this can cause connections to remain open forever when using commands like `nix shell` This change sets the FD_CLOEXEC flag using a CURLOPT_SOCKOPTFUNCTION callback. * Document Store Derivations and Deriving Paths (NixOS#12290) This is a big step documenting the store layer on its own, separately from the evaluator (and `builtins.derivation`). Co-authored-by: Robert Hensing <[email protected]> * Rename to "content-address*ing* derivation" "content-address*ed*" derivation is misleading because all derivations are *themselves* content-addressed. What may or may not be content-addressed is not derivation itself, but the *output* of the derivation. The outputs are not *part* of the derivation (for then the derivation wouldn't be complete before we built it) but rather separate entities produced by the derivation. "content-adddress*ed*" is not correctly because it can only describe what the derivation *is*, and that is not what we are trying to do. "content-address*ing*" is correct because it describes what the derivation *does* --- it produces content-addressed data. * Fix possible deref of null git_config * libfetchers: Drop no longer needed curl dependency * Misc code cleanups * Move git-lfs-fetch into its own source file * getLfsEndpointUrl(): Use our RAII helpers * Add release note * smudgeLfs: Use default value Eventually this should probably become a struct of options. * Add TODO * Restore NAR hash assertion * Remove stray line * Add release note * Update src/libstore/unix/build/local-derivation-goal.cc Co-authored-by: Robert Hensing <[email protected]> * nix flake prefetch: Add --out-link option This makes `nix flake prefetch` more useful for scripting and prevents the result from being GC'ed prematurely. * Only try to chmod /nix/var/nix/profiles/per-user when necessary Co-authored-by: Eelco Dolstra <[email protected]> * copyPathToStore(): Preserve symlinks E.g. in a derivation attribute `foo = ./bar`, if ./bar is a symlink, we should copy the symlink to the store, not its target. This restores the behaviour of Nix <= 2.19. * Don't import a symlink This is a workaround to avoid differing evaluation results between Nix 2.19 and >= 2.20 (NixOS#12449). * Get rid of `impureOutputHash` I do not believe there is any problem with computing `hashDerivationModulo` the normal way with impure derivations. Conversely, the way this used to work is very suspicious because two almost-equal derivations that only differ in depending on different impure derivations could have the same drv hash modulo. That is very suspicious because there is no reason to think those two different impure derivations will end up producing the same content-addressed data! Co-authored-by: Alain Zscheile <[email protected]> * lockFlake(): When refetching a locked flake, use the locked ref Otherwise we may accidentally update a lock when we shouldn't. Fixes NixOS#12445. * Formatting * Add test * feat: access tokens per repo * feat: test and document access-token prefix support * Add `inputs.self.lfs` * Add release note * fix: ensure access-token matches are complete * fix: add comment about longest-possible match * fix: linting * doc: recursive-nix: advertise requiredSystemFeatures It's best we teach users that the "foo" derivation is less than pure in the sense that it cannot be built just on any system, in particular that builders cannot be selected arbitrarily but based on their system-features. The `"recursive-nix"` system-feature is automatically defined by `--extra-experimental-features recursive-nix` * Merge release notes * Expose a bunch of things in the Legacy SSH Store for Hydra * More interesting dynamic derivations example Co-authored-by: Samuel Ainsworth <[email protected]> * Introduce `DerivationOptions` This is a first step towards PR NixOS#10760, and the issues it addresses. See the Doxygen for details. Thanks to these changes, we are able to drastically restrict how the rest of the code-base uses `ParseDerivation`. Co-Authored-By: HaeNoe <[email protected]> * Add `SSHMaster::Connection::trySetBufferSize` It is unused in Nix currently, but will be used in Hydra. This reflects what Hydra does in NixOS/hydra#1387. We may probably to use it more widely for better SSH store performance, but this needs to be subject to more testing before we do that. * Allow setting `ssh://` pipe size Exposed for Hydra. We could make it fancier but with (a) new store settings (b) switch to `ssh-ng://` both in the works, it doesn't seem worth it. * getDefaultNixPath(): Don't add symlinks if the target doesn't exist * resolveLookupPathPath(): Fix caching of negative lookups This avoids spamming in case the missing search path entry does not exist (NixOS#12480). * dep-built-drv-2.sh: Don't fail with "cannot create symlink" * JSONLogger: Log to a file descriptor instead of another Logger Logging to another Logger was kind of nonsensical - it was really just an easy way to get it to write its output to stderr, but that only works if the underlying logger writes to stderr. This change is needed to make it easy to log JSON output somewhere else (like a file or socket). * Support libgit2 1.9.0 For when the overlay is used with nixos-unstable. 1.9.0 has our patches. * Use getStandardError() * Fix a few warnings * Move code related to NIX_MAN_DIR from libstore to nix-cli This is a prerequisite to properly fixing man-pages once and for all [1]. Note that this patch leaves manpages for legacy commands in a borked state, pending the movement of manpages from nix-manual to nix-cli [2]. [1]: https://github.com/NixOS/nix/issues/12382 [2]: https://github.com/NixOS/nix/issues/12382#issuecomment-2663782043 * windows: fix compilation after recent changes Specifically last few week's merges involving legacy SSH options and dynamic derivations. * Restore detailed Nix CLI version ... as intended. Requirements: - don't build fresh libraries for each git commit - have git commit in the CLI Bug: - echo ${version} went into the wrong file => use the fact that it's a symlink, not just for reading but also for writing. * Write just ./.version on all components This way it's easier to get right. See previous commit. * startDaemon(): Detect if the daemon crashes before creating the socket This avoids timeouts like those seen in https://github.com/NixOS/nix/actions/runs/13376958708/job/37358120348?pr=6962. * Make 'logger' a std::unique_ptr This prevents it from being leaked (see bb411e4 for an example of this). * Fix crash on macOS * stopProgressBar() -> logger->stop() * Remove startProgressBar() * Remove createDefaultLogger() * tests: Fix installTests * Don't override default man search paths By appending a colon to MANPATH NIX_MAN_DIR gets prepended to the final MANPATH before default search paths. This makes man still consider default search paths, but prefers NIX_MAN_DIR (if it exists). It still makes sense to point NIX_MAN_DIR to a correct location by moving man pages build from nix-manual.man to nix-cli.man, but this should fix most common use-cases where nix is installed globally. * repl: suppress progress bar in printValue() * packaging: Move layers from dependencies to components This makes it easier to implement batch overriding for the components. * packaging: Add overrideAllMesonComponents * packaging: Add source overriding "methods" * test: Ignore packaging-overriding check on darwin for now * packaging: Add patch count to version * packaging: Make patch count lazier This makes `nix.version` quicker to evaluate, which should speed up package listing operations. If you want an accurate count, use `lib.optionals` in your override instead of `null` values. * libstore: fix expected bytes in progress bar * doc: Fix ccacheStdenvPackages typo * packaging: Remove dead code ... from nixpkgs, my bad. * packaging: Restore libgit2 USE_SSH=exec ... when nixpkgs is nixos-unstable or the overlay is used. * Remove nixfmt override Closes NixOS#12418 IFD in nixfmt repo * Move MountedSourceAccessor to libutil * Remove mounted-source-accessor.hh * Remove redundant call to canonPath() The CanonPath constructor already does that. * Add a storeFS accessor for paths resulting from IFD Hopefully fixes NixOS#11503. * Add test * MountedSourceAccessor: Remove redundant pathExists() method * Add a UnionSourceAccessor * Use UnionSourceAccessor to mount the chroot store on top of the real store directory * UnionSourceAccessor: Don't filter out underlying files of the wrong type NixOS#12512 (comment) * Remove sourcePathToStorePath() It's no longer needed now that all store paths inside the evaluator are logical rather than real paths. * In pure eval mode, restrict rootFS to just the Nix store Note that in pure mode, we don't need to use the union FS even when using a chroot store, since the user shouldn't have access to the physical /nix/store. * Remove unused variable * Introduce `EvalStore::storePath` This abstracts over a common case. Good for brevity, and enabling further experiments. * Fix dev shell There was one `inputs.nixFmt` left after 573ffac. * libfetchers-tests: Add back git-utils.cc Seems like this got dropped at some point during meson migration, so put it back in the build system. Drop all tests for `parseGitUrl`, since that function doesn't exist and migrating doesn't look sensible because git-lfs stuff seems to use `ParsedURL`. * doc: ssl-cert-file leaks into OSX builds * Fix perl store bindings When NixOS#9863 converted the `Nix::Store` free functions into member functions, the implicit `this` argument was not accounted for when iterating over the variable number of arguments in some functions. * libexpr: Fix use-after-free of StaticEnv::up It's not very clear what the ownership model is here, but one thing is certain: `.up` can't be destroyed before the StaticEnv that refers to it is. Changing a non-owning pointer to taking shared ownership of the parent `StaticEnv` prevents the `.up` from being freed. I'm not a huge fan of the inverted ownership, where child `StaticEnv` takes a refcount of the parent, but this seems like the least intrusive way to fix the use-after-free. This shouldn't cause any shared_ptr cycles to appear (hopefully). * tests/functional: Add flake-based regression for debugger use-after-free This is the simplest reproducer I have. It would be great to find a repro without flakes, but I guess this should be ok for now. * Include the Nix version in the title of the manual This makes it easy to see at a glance what the version of the manual is, e.g. "Nix 2.27.0 Reference Manual". * nix flake archive: Skip relative path inputs Fixes NixOS#12438. * packaging: Use correct stdenv for x86_64-darwin * nix flake archive: Recurse into relative path inputs We can't ignore them entirely, since we do want to archive their transitive inputs. Fixes NixOS#12438. * refact: Rename url -> hostAndPath https://github.com/NixOS/nix/pull/12465/files#r1955286197 > Perhaps that is a misnomer. * libstore: fix curl callback function signature * Fix mingw build https://hydra.nixos.org/build/291153007 * fix: update work meeting calendar link * release notes: 2.27.0 * Edit release notes * Add contributors * Fix date * Add more release notes * Expand manual on derivation outputs Note, this includes some text adapted from from Eelco's dissertation * Add host attribute of github/gitlab flakerefs to URL serialization `GitArchiveInputScheme::toUrl` currently drops the `host` attribute, creating invalid urls when locking `github:` or `gitlab:` urls pointing to alterative instances and serializing the input back to a url. ``` ❯ cat flake.nix { inputs.gnome-2048 = { url = "gitlab:GNOME/gnome-2048?host=gitlab.gnome.org"; flake = false; }; outputs = inputs: {}; } f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35 ❯ nix flake metadata warning: creating lock file '/private/var/folders/fb/f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35/flake.lock': • Added input 'gnome-2048': 'gitlab:GNOME/gnome-2048/70e0e430ca4bf590990433a3abdce6b631d50e6e?narHash=sha256-bya45ug2mDSU4SMn0fSBlZCuPl9y15B12ubKeb2A58s%3D' (2025-02-21) Resolved URL: path:/private/var/folders/fb/f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35 Locked URL: path:/private/var/folders/fb/f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35?lastModified=1740744684&narHash=sha256-nxUL/JiTYbZX2c1XiN/TC6aA1hf%2B1YXsUvhL7ASY2uE%3D Path: /nix/store/f4xczpwhdxs8gal1rika1c5bvhyd472l-source Last modified: 2025-02-28 13:11:24 Inputs: └───gnome-2048: gitlab:GNOME/gnome-2048/70e0e430ca4bf590990433a3abdce6b631d50e6e?narHash=sha256-bya45ug2mDSU4SMn0fSBlZCuPl9y15B12ubKeb2A58s%3D (2025-02-21 23:18:46) ``` Note the gnome-2048 input url missing the original host query. The Url after this commit: ``` [...] Inputs: └───gnome-2048: gitlab:GNOME/gnome-2048/70e0e430ca4bf590990433a3abdce6b631d50e6e?host=gitlab.gnome.org&narHash=sha256-bya45ug2mDSU4SMn0fSBlZCuPl9y15B12ubKeb2A58s%3D (2025-02-21 23:18:46) ``` * packaging/everything.nix: Use a multi-output derivation This should fix a few packaging regressions. `dev` also includes a merged `includes/`, which may be helpful until inter-component includes are fixed properly. * .mergify.yml: Add backport 2.27-maintenance entry * remove fricklerhandwerk from CODEOWNERS stepping aside as a Nix maintainer: https://discourse.nixos.org/t/time-to-step-aside/61050 * Revert "Revert "Revert "Adapt scheduler to work with dynamic derivations""" The bug reappeared after all, and the fix introduced a different bug. I just reverted on 2.27 first, in NixOS#12576, but upon further introspection and discussion with @roberth, with preparing for and travelling to Planet Nix I will not be able to fix it on `master` soon enough for a revert to not be warranted here in the meantime also. This reverts commit c985252. * `SingleDerivedPath` should be const in recursive data structures * Rework derivation input resolution I refactored the way that input resolution works in `DerivationGoal`. To be honest, it is probably unclear to the reader whether this new way is better or worse. I suppose *intrinsic* motivation, I can say that - the more structured use of `inputGoal` (a local variable) is better than the shotgrun approach with `inputDrvOutputs` - A virtual `waiteeDone` was a hack, and now it's gone. However, the *real* motivation of this is not the above things, but that it is needed for my mammoth refactor fixing NixOS#11897 and NixOS#11928. It is nice that this step could come first, rather than making that refactor even bigger. * manual: Edit * doc: note that function bindings are accessible in default values Co-authored-by: Robert Hensing <[email protected]> * packaging: Typo in setVersionLayer / preConfigure Apparently dead code in our use case, but good to keep nonetheless. Credit: ztzg in NixOS#12498 (review) * libstore: curl retry: reset content-encoding and don't use string after move * progress-bar: Make pause/resume nestable * repl: Fix value printing corruption The resume call would get some non-flushed(?) data. Extending the pause to include the newline makes the complete flush part of the pause. * refactor: RAII logger suspension * repl: Fix :print corruption See preceding commits. * libfetchers/git: fix double quoting in error message * libflake: fix double quoting when updating flakes * libstore/remote-store: avoid old-style casting for maxConnections Type-checking works better this way as (type) style casting is too permissive. * Fix minor documentation typos Was reading the store chapter and came across a few small typos and edits. * rapidcheck: change to working arbitrary instances Here we're switching to combinators instead of dereference operator. It turns out the dereference operator was being executed upon test setup, meaning that we were only using a only single value for each of the executions of the property tests! Really not good. And on Windows, we instead get: operator* is not allowed in this context https://github.com/emil-e/rapidcheck/blob/ff6af6fc683159deb51c543b065eba14dfcf329b/src/gen/detail/GenerationHandler.cpp#L16C31-L16C71 Now a few of the property tests fail, because we're generating cases which haven't been exercised before. * coerceToSingleDerivedPathUnchecked: pass through experimental features This fixes a few of the property tests, now that the property tests are actually generating arbitrary data - some of that data now requiring experimental features to function properly. * DerivedPathTest: disable prop_legacy_round_rip until fixed * c-api: fix a few memory leaks * flake: Enable UBSAN for checks Doing this makes catching non-obvious bugs easier. GHA CI workload is already a concern and there isn't much benefit in running the tests with and without sanitizers at the same time, so UBSAN is enabled for default checks. This change doesn't affect production builds in any way, but is rather a step in the direction of improving automated testing during development. Relates to NixOS#10969. * build(deps): bump cachix/install-nix-action from 30 to 31 Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 30 to 31. - [Release notes](https://github.com/cachix/install-nix-action/releases) - [Commits](cachix/install-nix-action@v30...v31) --- updated-dependencies: - dependency-name: cachix/install-nix-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * libutil/windows: Finally use the correct constructor for std::wstring C++ is very intuitive /s [1]. Fixes NixOS#12631. [1]: https://godbolt.org/z/jMa9GP5sq * port crash-handler from lix to nix It was first introduced in https://git.lix.systems/lix-project/lix/commit/19e0ce2c03d8e0baa16998b086665664c420c1df In Nix we only register the crash handler in main instead of initNix, because library user may want to use their own crash handler. Sample output: Mar 12 08:38:06 eve nix[2303762]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included: Mar 12 08:38:06 eve nix[2303762]: Exception: nix::SysError: error: writing to file: Resource temporarily unavailable Mar 12 08:38:06 eve nix[2303762]: Stack trace: Mar 12 08:38:06 eve nix[2303762]: 0# 0x000000000076876A in nix 1# 0x00007FDA40E9F20A in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6 2# std::unexpected() in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6 3# 0x00007FDA40E9F487 in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6 4# nix::writeFull(int, std::basic_string_view<char, std::char_traits<char> >, bool) in /home/joerg/git/nix/inst/lib/libnixutil.so 5# nix::writeLine(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) in /home/joerg/git/nix/inst/lib/libnixutil.so 6# nix::JSONLogger::write(nlohmann::json_abi_v3_11_3::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> const&) in /home/joerg/git/nix/inst/lib/libnixutil.so 7# nix::JSONLogger::logEI(nix::ErrorInfo const&) in /home/joerg/git/nix/inst/lib/libnixutil.so 8# nix::Logger::logEI(nix::Verbosity, nix::ErrorInfo) in nix 9# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/joerg/git/nix/inst/lib/libnixmain.so 10# 0x000000000087A563 in nix 11# 0x00007FDA40BD41FE in /nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6 12# __libc_start_main in /nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6 13# 0x00000000006F4DF5 in nix Co-authored-by: eldritch horrors <[email protected]> * Remove use of $NIX_HELD_LOCKS This variable was once used to communicate already acquired store path locks between Nix and the build hook, but this hasn't been the case since 9bcb4d2. So let's get rid of it. * libfetchers/git: fix caching head when using shallow clones the old code was using the wrong cache directory, which lead to a warning on every nix flake update Update src/libfetchers/git.cc * Advanced attributes organize This is supposed to firstly improve the docs as they are, and secondly hint at how the core conceptual information ought to be moved to the store derivation section of the manual. Co-authored-by: Jörg Thalheim <[email protected]> * Factor out "last 10 log lines" error message code This will help avoid duplication later. In particular, the next commit will not need to duplicate as much. * Inline `buildDone` from `DerivationGoal` into use sites The basic idea is that while we have duplicated this function, we now have one call-site in the local build case, and one call site in the build hook case. This unlocks big opportunities to specialize each copy, since they really shouldn't be doing the same things. By the time we are are done, there should not be much duplication left. See NixOS#12628 for further info. * Start simplifying `{Local,}DerivationGoal` cleanup code Thanks to the previous commit, we can inline all these small callbacks. In the build-hook case, they were empty, and now they disappear entirely. While `LocalDerivationGoal` can be used in the hook case (we use it based on whether we have a local store, not based on whether we are using the build hook, a decision which comes later), the previous commit's inline moved the code into a spot where we know we are cleaning up after local building, *not* after running the build hook. This allows for much more simplification. * Remove dead hook code in `LocalDerivationGoal::tryLocalBuild` The `assert` above proves that `hook` is not set. * Avoid pointless mutation The code that was in between is now gone. We can just set `st` correctly the first time. * Remove `privateNetwork` variable from local drv goal Can just inline its definition, it was immutable. * Simplify local drv goal a bit more - `chrootParentDir` can be a local variable instead of a class variable. - `getChildStatus` can be inlined. Again, we have the `assert(!hook);` in the local building case, which makes for a simpler thing inlined. * Remove `registerOutputs` from drv goal Easy to inline in one spot, and assert in the other. * Remove `signRealisation` from drv goal We can move this method from `LocalStore` to `Store` --- even if we only want the actual builder to sign things in many cases, there is no reason to try to enforce this policy by spurious moving the method to a subclass. Now, we might technically sign class, but CA derivations is experimental, and @Ericson2314 is going to revisit all this stuff with issue NixOS#11896 anyways. * Do no store timestamps in the build result in the build hook case The variables are only set by CGroup mechanisms in `killSandbox` in the local build. In the build hook case, these variables will not be set, so there is nothing to do. * Inline the try-catch `BuildError` in the hook case In the local building case, there is many things which can through `BuildError`, but in the hook case there is just this one. We can therefore simplify the code by "cinching" down the logic just to the spot the error is thrown. There is other code outside `libstore/build` which also uses `BuildError`, but I believe those cases are mistakes. The point of `BuildError` is the narrow technical use-cases of "errors which should not be fatal with `--keep-going`". Using it outside the building/scheduling code doesn't really make sense in that regard. It seems likely that those usages were instead merely because "oh, this error has something to do with building, so I guess `BuildError` is better than `Error`". It is quite likely that I myself used `BuildError` incorrectly as described above :). * Simplify hook error status logic The simplification here is due to a long-standing bug, but it is not worth fixing the bug at this time. Instead we've finally written up an issue for the bug, and referenced the issue number in the code. * Make debug message more precise * Remove unused parameter to the goal constructor It has been unused since 37fca66. * JSONLogger: Acquire a lock to prevent log messages from clobbering each other --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Eelco Dolstra <[email protected]> Co-authored-by: John Ericson <[email protected]> Co-authored-by: Illia Bobyr <[email protected]> Co-authored-by: John Ericson <[email protected]> Co-authored-by: silvanshade <[email protected]> Co-authored-by: Robert Hensing <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: MaxHearnden <[email protected]> Co-authored-by: Robert Hensing <[email protected]> Co-authored-by: Aiden Fox Ivey <[email protected]> Co-authored-by: Sandro Jäckel <[email protected]> Co-authored-by: Jörg Thalheim <[email protected]> Co-authored-by: Alain Zscheile <[email protected]> Co-authored-by: Thomas Bereknyei <[email protected]> Co-authored-by: Leandro Reina <[email protected]> Co-authored-by: Someone <[email protected]> Co-authored-by: Samuel Ainsworth <[email protected]> Co-authored-by: HaeNoe <[email protected]> Co-authored-by: Sergei Zimmerman <[email protected]> Co-authored-by: Brian McKenna <[email protected]> Co-authored-by: Philipp Otterbein <[email protected]> Co-authored-by: Ivan Trubach <[email protected]> Co-authored-by: Silvan Mosberger <[email protected]> Co-authored-by: Fabian Möller <[email protected]> Co-authored-by: Yannik Sander <[email protected]> Co-authored-by: Valentin Gagarin <[email protected]> Co-authored-by: Jörg Thalheim <[email protected]> Co-authored-by: Farid Zakaria <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jade Lovelace <[email protected]> Co-authored-by: eldritch horrors <[email protected]> Co-authored-by: Las <[email protected]>

Ericson2314 requested a review from edolstra as a code owner January 26, 2024 21:47

Ericson2314 force-pushed the perl-open-other-store branch 2 times, most recently from cfd638f to e3ccb7e Compare January 26, 2024 22:06

Ericson2314 mentioned this pull request Jan 26, 2024

Build: use nix path-info to test for build output availability NixOS/hydra#1352

Open

edolstra reviewed Jan 29, 2024

View reviewed changes

perl/lib/Nix/Store.xs Outdated Show resolved Hide resolved

Ericson2314 marked this pull request as draft January 29, 2024 15:44

Ma27 reviewed Feb 4, 2024

View reviewed changes

Ma27 mentioned this pull request Feb 7, 2024

Update perl bindings for nix#9863 NixOS/hydra#1359

Merged

Ericson2314 marked this pull request as ready for review February 7, 2024 14:30

Ericson2314 force-pushed the perl-open-other-store branch from e3ccb7e to 23b651e Compare February 7, 2024 17:59

tomberek reviewed Feb 8, 2024

View reviewed changes

Support arbitrary stores in Perl bindings

bc08502

Fix NixOS#9859 It's a breaking change but that's fine; we can just update Hydra to use the new bindings.

Ericson2314 force-pushed the perl-open-other-store branch from 23b651e to bc08502 Compare February 8, 2024 03:32

edolstra reviewed Feb 8, 2024

View reviewed changes

edolstra approved these changes Feb 8, 2024

View reviewed changes

Ericson2314 merged commit f2f54cf into NixOS:master Feb 8, 2024

Ericson2314 deleted the perl-open-other-store branch February 8, 2024 14:30

Ericson2314 mentioned this pull request Feb 22, 2024

(Towards) stable C bindings for libutil, libexpr #8699

Merged

12 tasks

fabianhjr mentioned this pull request Sep 18, 2024

Support Nix::Store from 2.21 onwards edolstra/nix-serve#60

Merged

B4dM4n added a commit to B4dM4n/nix that referenced this pull request Feb 21, 2025

Fix perl store bindings

60d3495

When NixOS#9863 converted these free functions into member functions, the implicit `this` argument was not accounted for when iterating over the variable number of arguments.

B4dM4n mentioned this pull request Feb 21, 2025

Fix perl store bindings #12543

Merged

mergify bot mentioned this pull request Feb 21, 2025

Fix perl store bindings (backport #12543) #12545

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support arbitrary stores in Perl bindings #9863

Support arbitrary stores in Perl bindings #9863

Ericson2314 commented Jan 26, 2024

Ericson2314 commented Jan 29, 2024

Ma27 Feb 4, 2024

Ericson2314 Feb 5, 2024

Ericson2314 commented Feb 7, 2024

tomberek left a comment

Ericson2314 commented Feb 8, 2024

edolstra Feb 8, 2024

Ericson2314 Feb 8, 2024

edolstra Feb 8, 2024 •

edited

Loading

Ericson2314 Feb 8, 2024

Ericson2314 commented Feb 8, 2024 •

edited

Loading

nixos-discourse commented Sep 18, 2024

Support arbitrary stores in Perl bindings #9863

Support arbitrary stores in Perl bindings #9863

Conversation

Ericson2314 commented Jan 26, 2024

Motivation

Context

Priorities and Process

Ericson2314 commented Jan 29, 2024

Ma27 Feb 4, 2024

Choose a reason for hiding this comment

Ericson2314 Feb 5, 2024

Choose a reason for hiding this comment

Ericson2314 commented Feb 7, 2024

tomberek left a comment

Choose a reason for hiding this comment

Ericson2314 commented Feb 8, 2024

edolstra Feb 8, 2024

Choose a reason for hiding this comment

Ericson2314 Feb 8, 2024

Choose a reason for hiding this comment

edolstra Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

Ericson2314 Feb 8, 2024

Choose a reason for hiding this comment

Ericson2314 commented Feb 8, 2024 • edited Loading

nixos-discourse commented Sep 18, 2024

edolstra Feb 8, 2024 •

edited

Loading

Ericson2314 commented Feb 8, 2024 •

edited

Loading