Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix sandbox builds on ventura #149

Merged
merged 1 commit into from
Jan 19, 2024
Merged

Conversation

j-baker
Copy link
Contributor

@j-baker j-baker commented Dec 29, 2023

The Cargo bundled in this project links against the operating system's libcurl. This can be demonstrated with

$ otool -L $(which cargo)
/nix/store/s8rb4j0rh3wm66r4hmgj4axcic321bak-rust-default-1.73.0/bin/cargo:
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.101.2)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

The system's libcurl dynamically depends on the system's libcrypto. On MacOS 14, this (at some point) opens /private/etc/ssl/openssl.cnf. With the Nix sandbox on, this fails the build with

       > 8082083840:error:02FFF001:system library:func(4095):Operation not permitted:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/bio/bss_file.c:122:fopen('/private/etc/ssl/openssl.cnf', 'rb')
       > 8082083840:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/bio/bss_file.c:127:
       > 8082083840:error:0EFFF002:configuration file routines:CRYPTO_internal:system lib:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/conf/conf_def.c:202:
       For full logs, run 'nix log /nix/store/h05s2pzw6qb5m50njk0vr7j6cimhcxmj-foo.drv'.

as reported in #148.

This could be 'fixed' in three places.

  1. On the Apple side. This probably won't happen.
  2. Inside Nix, by changing the sandbox definition. I'd argue that since this bug isn't really reachable with 'normal' Nix binaries (where you would link against the Nix openssl) it's not the ideal place to make the change. Additionally, it adds a Nix version dependency on usage of this tool. This is discussed here: aarch64-darwin: sandbox issue with /private/etc/ssl/openssl.cnf NixOS/nix#9625. For users where I work, this is painful as we deploy Nix using Nix darwin, and by default deploy a module using the Rust overlay, so we get a chicken and egg problem as each dev updates to Ventura.
  3. Here. We can change the link to point to a nixpkgs version of libcurl rather than the OS provided version, and this avoids the MacOS libressl implementation. This feels right because it seems analogous to 'I tried to run a random binary on NixOS and it couldn't find /lib64/ld-linux-x86-64.so.2', traditionally a derivation author's responsibility.

I chose 3 as being likely the most pragmatic solution, so here we are!

After this MR, we have:

$ otool -L $(which cargo)
/nix/store/km3i2b7dqb1h2ayy2qg239266ixkzxgl-rust-default-1.75.0/bin/cargo:
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.101.2)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/nix/store/c39qm57grkavw8a4hkramahpnspm1inq-curl-8.4.0/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

In theory it might be more consistent/better to additionally replace the various other links with more nixy links, but I figured that the surgical approach was less intrusive at this time.

The Cargo bundled in this project links against the operating system's
libcurl. This can be demonstrated with

```
$ otool -L $(which cargo)
/nix/store/s8rb4j0rh3wm66r4hmgj4axcic321bak-rust-default-1.73.0/bin/cargo:
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.101.2)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
```

The system's libcurl dynamically depends on the system's libcrypto. On
MacOS 14, this (at some point) opens `/private/etc/ssl/openssl.cnf`.
With the Nix sandbox on, this fails the build with

```
       > 8082083840:error:02FFF001:system library:func(4095):Operation not permitted:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/bio/bss_file.c:122:fopen('/private/etc/ssl/openssl.cnf', 'rb')
       > 8082083840:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/bio/bss_file.c:127:
       > 8082083840:error:0EFFF002:configuration file routines:CRYPTO_internal:system lib:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/conf/conf_def.c:202:
       For full logs, run 'nix log /nix/store/h05s2pzw6qb5m50njk0vr7j6cimhcxmj-foo.drv'.
```

This could be 'fixed' in three places.

1. On the Apple side. This probably won't happen.
2. Inside Nix, by changing the sandbox definition. I'd argue that since
   this bug isn't really reachable with 'normal' Nix binaries (where you
   would link against the Nix openssl) it's not the ideal place to make
   the change. Additionally, it adds a Nix version dependency on usage
   of this tool.
3. Here. We can change the link to point to a nixpkgs version of
   libcurl rather than the OS provided version.

I chose 3 as being likely the most pragmatic solution, so here we are!
@n8henrie
Copy link

Thank you @j-baker!

I can confirm the patch fixes the issue for me, using my test flake from NixOS/nix#9625:

$ nix build
error: builder for '/nix/store/d8nfiihbhdg75wwzifscghnjc6344s5r-foo.drv' failed with exit code 1;
       last 10 log lines:
       > Finished cargoSetupPostPatchHook
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       > building
       > Executing cargoBuildHook
       > ++ env CC_AARCH64_APPLE_DARWIN=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/cc CXX_AARCH64_APPLE_DARWIN=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/c++ CARGO_TARGET_AARCH64_APPLE_DARWIN_LINKER=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/cc CC_AARCH64_APPLE_DARWIN=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/cc CXX_AARCH64_APPLE_DARWIN=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/c++ CARGO_TARGET_AARCH64_APPLE_DARWIN_LINKER=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/cc CARGO_BUILD_TARGET=aarch64-apple-darwin HOST_CC=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/cc HOST_CXX=/nix/store/zh206nri10wwp68g7qf9ccya1irynx5g-clang-wrapper-16.0.6/bin/c++ cargo build -j 8 --target aarch64-apple-darwin --frozen --profile release
       > Auto configuration failed
       > 8019775488:error:02FFF001:system library:func(4095):Operation not permitted:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/bio/bss_file.c:122:fopen('/private/etc/ssl/openssl.cnf', 'rb')
       > 8019775488:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/bio/bss_file.c:127:
       > 8019775488:error:0EFFF002:configuration file routines:CRYPTO_internal:system lib:/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/conf/conf_def.c:202:
       For full logs, run 'nix log /nix/store/d8nfiihbhdg75wwzifscghnjc6344s5r-foo.drv'.
$
$ nix build \
    --override-input rust-overlay \
        github:j-baker/rust-overlay/647bff9f5e10d7f1756d86eee09831e6b1b06430
warning: not writing modified lock file of flake 'git+file:///private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.TT1IrttYh8':
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/2b24e1f369f00f5ae9876e15e12f77e12c9c2374' (2023-12-29)
  → 'github:j-baker/rust-overlay/647bff9f5e10d7f1756d86eee09831e6b1b06430' (2023-12-29)
$ echo $?
0

@j-baker
Copy link
Contributor Author

j-baker commented Jan 18, 2024

hi @oxalica , just wanted to see if this could get a review?

@tolbrino
Copy link

fwiw this patch works nicely for me as well

@n8henrie
Copy link

I'm also curious about thoughts on:

In theory it might be more consistent/better to additionally replace the various other links with more nixy links

and happy to help contribute / test here if thought to be a good idea.

@oxalica oxalica merged commit d20edfd into oxalica:master Jan 19, 2024
29 checks passed
al3xtjames added a commit to al3xtjames/nixpkgs that referenced this pull request Mar 30, 2024
Modern versions of macOS link the system-provided curl library against
the system-provided libressl library. On recent versions of macOS, the
system libressl library reads from /private/etc/ssl/openssl.cnf. As this
path is not included in the default Nix sandbox profile, applications
that use the system curl library will report a permission error [1].

This issue affects the bootstrap version of cargo and can be seen while
building rustc for darwin with the sandbox enabled [2]. This change
works around the sandbox failure by using install_name_tool to patch the
cargo binary to use curl provided by Nix, which was the approach used in
oxalica/rust-overlay [3].

[1]: NixOS/nix#9625
[2]: https://gist.github.com/al3xtjames/06bf71ceffd745eef20be8ce03b982c5
[3]: oxalica/rust-overlay#149
al3xtjames added a commit to al3xtjames/nixpkgs that referenced this pull request Dec 6, 2024
Modern versions of macOS link the system-provided curl library against
the system-provided libressl library. On recent versions of macOS, the
system libressl library reads from /private/etc/ssl/openssl.cnf. As this
path is not included in the default Nix sandbox profile, applications
that use the system curl library will report a permission error [1].

This issue affects the bootstrap version of cargo and can be seen while
building rustc for darwin with the sandbox enabled [2]. This change
works around the sandbox failure by using install_name_tool to patch the
cargo binary to use curl provided by Nix, which was the approach used in
oxalica/rust-overlay [3].

[1]: NixOS/nix#9625
[2]: https://gist.github.com/al3xtjames/06bf71ceffd745eef20be8ce03b982c5
[3]: oxalica/rust-overlay#149
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants