darwin.stdenv: Update to LLVM 15 and clang to 15.0.7 #229786

reckenrode · 2023-05-04T01:55:01Z

Description of changes

Bumps LLVM and clang to 15.0.7 on Darwin. This was prompted by #229210 because the 13.3 SDK does not build with LLVM 11. This PR reworks the Darwin stdenv to decouple it from the bootstrap tools is written not to need updates to bootstrap tools and includes changes to allow LLVM to be bumped without requiring a bump to the bootstrap tools.

See the comments in 309dc9d8f1e explaining the rework of the Darwin stden bootstrap process. My hope is this will make future bumps easier and more approachable for non-Darwin developers. It should also make experimentation a little safer. I’d like to see if any space could be saved by enabling LTO during the bootstrap, but that’s out of scope for this PR.

Issues to be resolved

These have all been resolved. The changes needed for the bootstrap are included in this PR. The others are being handled in separate PRs, which will be linked below.

Disable LLVM tests when building the stdenv. This causes an infinite recursion currently. I have a workaround, but ideally it should be disabled in the bootstrap.
pkg-config requires disabling a warning, or the build fails. It’s vendoring a very old version of GLib. Unfortunately, nixpkgs does not provide a way to use the nixpkgs glib package. I also tried, and it caused an infinite recursion.
codesign in bootstrap tools is too old for LLVM, but I’ve included a workaround. A bootstrap tools update would obviate the change.
Updating platform definition now that LLVM understands apple-m1. This should likely go in another PR, but it’s included here for testing. See apple-m1 compiler optimisations for aarch64-darwin? #166401.

Testing Done

Running nixpkgs-review on a stdenv change is obviously going to rebuild an infeasible amount of packages. I have attempted to build and run a number of packages to catch common failures. I’ve tried to change things needed to build the stdenv from those that fix common problems. See below for PRs related to those (and sandbox issues I encountered along the way).

Related PRs

These fail to build due to the changes.

These are Darwin sandbox-related fixes. They’re not strictly necessary, but they allow Darwin users to build more things with the sandbox enabled.

This fix is related to running the store on case-sensitive APFS. It’s the first thing that has broken for me. Most case sensitivity issues during builds are due to /tmp being insensitive.

meson: fix build with case-sensitive store on Darwin #234105
(more fixes coming tomorrow)

Things done

RaitoBezarius · 2023-05-04T03:15:12Z

Hello, thank you for your contribution, please read the CONTRIBUTING guide and target staging adequately (without mass pinging everyone by reading the guide).

pkgs/stdenv/darwin/unpack-bootstrap-tools-aarch64.sh

eliasnaur · 2023-05-10T16:28:24Z

lib/systems/platforms.nix

-      arch = "armv8.3-a+crypto+sha2+aes+crc+fp16+lse+simd+ras+rdm+rcpc";
-      cpu = "apple-a13";
+      arch = "armv8.5a+crc+crypto+dotprod+fp16fml+ras+lse+rdm+rcpc+sm4+sha3+sha2+aes";
+      cpu = "apple-m1";


Just curious: does this change by any chance affect the hack I recently had to add:

nixpkgs/pkgs/development/compilers/gcc/common/platform-flags.nix

Lines 11 to 12 in 9846599

# --with-cpu on aarch64-darwin fails with "Unknown cpu used in --with-cpu=apple-a13".

(lib.optional (!isAarch64Darwin && p ? cpu) "--with-cpu=${p.cpu}")

Probably not, but one can hope.

Not as far as I can tell. There’s an open issue at iains/gcc-darwin-arm64#86 for support.

eliasnaur · 2023-05-10T17:17:27Z

pkgs/stdenv/darwin/default.nix

+        # Having this set when trying to cross-compile to Linux causes problems, so only set it
+        # when building for Darwin.


Nit: consider dropping this comment. It seems clear that darwin specific stuff should be guarded by platform checks, by principle.

reckenrode · 2023-05-20T21:08:04Z

I pushed the current branch. I expect I’ll be opening separate PRs to fix things that don’t build with the new clang that aren’t really needed for the bootstrap (or fail immediately). I still need to do some cleanup on commit messages before marking this ready and review any other issues that could be resolved/affected. I dropped the platforms.nix changes for now.

xnu does not require a Python that supports configd, so use a minimal one without support. Doing this prevents an infinite recuresion when building configd with xpc, which is needed to build with a newer LLVM.

Clang 15 does not like the fake xpc headers. Use the real ones instead. Doing this no longer causes an infinite recursion because xnu now depends on python3Minimal, which does not include configd support.

cctools-llvm is a replacement for cctools that replaces as much of cctools with equivalents from LLVM that it can reasonably do. This was motivated by wanting to reduce dependencies on cctools, which are updated infrequently by upstream. To provide a motivating example, the version of `strip` included in cctools cannot properly strip the archives in compiler-rt in LLVM 15. Paths are left to bootstrap tools, resulting in failed requisites checks in the final stdenv build. Since `strip` needs replaced, other tools were replaced as well. Note that this has to be done in cctools (or some equivalent) because some derivations (noteably LLVM) use the bintools of the stdenv directly instead of going through the wrapper. The following tools from LLVM are not used in this derivation: * LLD - not fully compatible with ld64 yet and potentially too big of a change; * libtool - not a drop-in replacement yet because it does not support linker passthrough, which is needed by xcbuild; and * install_name_tool - fails when trying to build swift-corefoundation. If other incompatibilities are found, other tools can be reverted. However, testing for the stdenv bump has shown the others seem to work pretty well as replacements. One final caveat: some tools are not duplicated or linked from cctools. As a rule of thumb, the actual tools on a Darwin system were used to determine which ones to link, so there are some differences (e.g., in names or availability).

@eliasnaur

Closes NixOS#230870. Thanks to @eliasnaur for the test case and for rasining awareness and to @veprbl for the work done on NixOS#111385. This takes a slightly different approach from those two PRs. The hook is set unconditionally. The stdenv bootstrap doesn’t really need CF at all, so setting the hook is harmless. This simplifies things.

… SDK

reckenrode · 2023-05-26T04:51:55Z

The last push should reflect the final PR. I am keeping this draft until I can get the PRs submitted for the failures due to the stdenv change, which should be tomorrow.

reckenrode · 2023-05-26T12:35:21Z

The ofborg eval is failing on llvmPackages.clang-polly-unwrapped, but that attribute is not part of llvmPackages_15. It appears it’s not included been part of any llvmPackages since 11.

In preparation for bumping the LLVM used by Darwin, this change refactors and reworks the stdenv build process. When it made sense, existing behaviors were kept to avoid causing any unwanted breakage. However, there are some differences. The reasoning and differences are discussed below. - Improved cycle times - Working on the Darwin stdenv was a tedious process because `allowedRequisites` determined what was allowed between stages. If you made a mistake, you might have to wait a considerable amount of time for the build to fail. Using assertions makes many errors fail at evaluation time and makes moving things around safer and easier to do. - Decoupling from bootstrap tools - The stdenv build process builds as much as it can in the early stages to remove the requirement that the bootstrap tools need bumped in order to bump the stdenv itself. This should lower the barrier to updates and make it easier to bump in the future. It also allows changes to be made without requiring additional tools be added to the bootstrap tools. - Patterned after the Linux stdenv - I tried to follow the patterns established in the Linux stdenv with adaptations made to Darwin’s needs. My hope is this makes the Darwin stdenv more approable for non-Darwin developers who made need to interact with it. It also allowed some of the hacks to be removed. - Documentation - Comments were added explaining what was happening and why things were being done. This is particular important for some stages that might not be obvious (such as the sysctl stage). - Cleanup - Converting the intermediate `allowedRequisites` to assertions revealed that many packages were being referenced that no longer exist or have been renamed. Removing them reduces clutter and should help make the stdenv bootstrap process be more understandable.

cole-h · 2023-05-26T21:33:44Z

Running the outpaths.nix check locally, this is what I see right before the eval fails:

[...snip...]
instantiated 'llvm-manpages-15.0.7' -> '/nix/store/2msilg6yh3p7493r1xv7mdb0j432s4mz-llvm-manpages-15.0.7.drv'
evaluating attribute 'recurseForDerivations'
evaluating attribute 'llvmPackages'
evaluating attribute '__unfix__'
evaluating attribute 'recurseForDerivations'
evaluating attribute 'bintools'
evaluating file '/home/vin/workspace/vcs/nixpkgs/master/pkgs/development/compilers/llvm/11/bintools/default.nix'
evaluating attribute 'recurseForDerivations'
evaluating attribute 'bintools-unwrapped'
evaluating attribute 'recurseForDerivations'
evaluating attribute 'bintoolsNoLibc'
evaluating attribute 'recurseForDerivations'
evaluating attribute 'clang'
evaluating attribute 'aarch64-darwin'
evaluating attribute 'recurseForDerivations'
evaluating attribute 'clang-manpages'
evaluating attribute 'aarch64-darwin'
evaluating attribute 'recurseForDerivations'
evaluating attribute 'clang-polly-unwrapped'
evaluating attribute 'aarch64-darwin'
error:
[...snip...]

Note that this is specifically a problem on the aarch64-darwin eval -- all other platforms are fine. You can reproduce this by looking at https://github.com/NixOS/ofborg#running-meta-checks-locally and commenting out all the architectures except for aarch64-darwin. (I ran with -vvv to get the above output.)

Dunno what's going on there, but maybe this will help you diagnose.

reckenrode · 2023-05-29T01:29:47Z

Closing this in favor of tracking issue #234710, so I can submit changes incrementally while working through the ofborg issue and the gnugrep issue on x86_64-darwin.

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label May 4, 2023

github-actions bot added the 6.topic: stdenv Standard environment label May 4, 2023

ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild 10.rebuild-linux-stdenv This PR causes stdenv to rebuild labels May 4, 2023

ofborg bot requested review from rrbutani, RaitoBezarius, primeos, dtzWill, sternenseemann, Ericson2314, alyssais and lovek323 May 4, 2023 03:11

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels May 4, 2023

reckenrode force-pushed the stdenv-clang15 branch from 2f64aad to 0c20741 Compare May 4, 2023 03:15

reckenrode changed the base branch from master to staging May 4, 2023 04:13

reckenrode changed the title ~~darwin.stdenv: Update to LLVM 15 and clang 15.0.7~~ darwin.stdenv: Update to LLVM 15 and clang to 15.0.7 May 4, 2023

rrbutani reviewed May 4, 2023

View reviewed changes

pkgs/stdenv/darwin/unpack-bootstrap-tools-aarch64.sh Outdated Show resolved Hide resolved

ofborg bot requested a review from rrbutani May 4, 2023 04:32

ConnorBaker mentioned this pull request May 6, 2023

apple_sdks: init with 11.1, 11.3, 12.1, 12.3, 13.1, and 13.3 #229210

Closed

12 tasks

reckenrode force-pushed the stdenv-clang15 branch from 0c20741 to 117997b Compare May 10, 2023 07:22

reckenrode mentioned this pull request May 10, 2023

stdenv/darwin: set NIX_COREFOUNDATION_RPATH via hook #230870

Closed

12 tasks

reckenrode force-pushed the stdenv-clang15 branch 3 times, most recently from d0d9939 to dba9d01 Compare May 10, 2023 16:13

eliasnaur reviewed May 10, 2023

View reviewed changes

reckenrode force-pushed the stdenv-clang15 branch from dba9d01 to 5ca231a Compare May 20, 2023 21:06

reckenrode force-pushed the stdenv-clang15 branch from 5ca231a to 1c2dcf0 Compare May 26, 2023 04:38

reckenrode added 10 commits May 26, 2023 00:47

indent: fix build with clang 13 or newer

4515948

pkg-config: fix build with newer clang

8d8d673

rewrite-tbd: avoid infinite recursion when cmake is not cmakeMinimal

102351c

xnu: use python3Minimal instead of python3

7a96bc9

xnu does not require a Python that supports configd, so use a minimal one without support. Doing this prevents an infinite recuresion when building configd with xpc, which is needed to build with a newer LLVM.

xnu: fix build with newer clang

80ced31

configd: fix build with newer LLVM and bootstrap

02b1996

Clang 15 does not like the fake xpc headers. Use the real ones instead. Doing this no longer causes an infinite recursion because xnu now depends on python3Minimal, which does not include configd support.

cctools: make cctools-llvm the default on Darwin

00d7447

apple_sdk_11_0.stdenv: replace extraBuildInputs with CF from the 11.0…

3f442fe

… SDK

reckenrode force-pushed the stdenv-clang15 branch from 1c2dcf0 to 4683307 Compare May 26, 2023 04:50

reckenrode force-pushed the stdenv-clang15 branch from 4683307 to 455c7ad Compare May 26, 2023 19:48

darwin.stdenv: bump LLVM version to 15

7a4c78d

reckenrode force-pushed the stdenv-clang15 branch from 455c7ad to 7a4c78d Compare May 26, 2023 20:09

This was referenced May 27, 2023

llvmPackages_latest: 14 -> 15 #213202

Closed

Tracking issue for Darwin stdenv LLVM update #234710

Closed

reckenrode closed this May 29, 2023

rrbutani added the 6.topic: llvm/clang Issues related to llvmPackages, clangStdenv and related label May 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

darwin.stdenv: Update to LLVM 15 and clang to 15.0.7 #229786

darwin.stdenv: Update to LLVM 15 and clang to 15.0.7 #229786

reckenrode commented May 4, 2023 •

edited

Loading

RaitoBezarius commented May 4, 2023

eliasnaur May 10, 2023

reckenrode May 10, 2023

eliasnaur May 10, 2023 •

edited

Loading

reckenrode commented May 20, 2023

reckenrode commented May 26, 2023

reckenrode commented May 26, 2023

cole-h commented May 26, 2023 •

edited

Loading

reckenrode commented May 29, 2023

	# --with-cpu on aarch64-darwin fails with "Unknown cpu used in --with-cpu=apple-a13".
	(lib.optional (!isAarch64Darwin && p ? cpu) "--with-cpu=${p.cpu}")

		# Having this set when trying to cross-compile to Linux causes problems, so only set it
		# when building for Darwin.

darwin.stdenv: Update to LLVM 15 and clang to 15.0.7 #229786

darwin.stdenv: Update to LLVM 15 and clang to 15.0.7 #229786

Conversation

reckenrode commented May 4, 2023 • edited Loading

Description of changes

Issues to be resolved

Testing Done

Related PRs

Things done

RaitoBezarius commented May 4, 2023

eliasnaur May 10, 2023

Choose a reason for hiding this comment

reckenrode May 10, 2023

Choose a reason for hiding this comment

eliasnaur May 10, 2023 • edited Loading

Choose a reason for hiding this comment

reckenrode commented May 20, 2023

reckenrode commented May 26, 2023

reckenrode commented May 26, 2023

cole-h commented May 26, 2023 • edited Loading

reckenrode commented May 29, 2023

reckenrode commented May 4, 2023 •

edited

Loading

eliasnaur May 10, 2023 •

edited

Loading

cole-h commented May 26, 2023 •

edited

Loading