compiler-rt: Provide __cpu_indicator_init, __cpu_model and __cpu_features2 #20081

lacc97 · 2024-05-26T20:25:36Z

Fixes #18074, original PR was #18193.

This PR adds support for X86/X86_64 CPU model detection and function multiversioning in C/C++ via the target and target_clones attributes (compatible with LLVM's, which in turn is compatible with libgcc's). It adds __cpu_indicator_init, __cpu_model and __cpu_features2 symbols to compiler-rt for x86/x86_64 which reuse the CPU detection logic under std/zig/system/x86.zig and transform the detected features into the values expected by LLVM-generated code. These values are kept in sync with LLVM using a new tool (tools/update_x86_cpu_model_enums.zig) which parses llvm/include/llvm/TargetParser/X86TargetParser.def in LLVM to generate the corresponding Zig enums.

This PR also adds a standalone test that the symbols are emitted when the target attribute is used in C.

While this functionality could in principle contribute to solving ~~issue #4591~~ (edit: wrong issue, its #1018), I think it would be best to use a Zig-specific solution for richer CPU detection capabilities, leveraging std.Target.* instead of relying only on the feature set supported by LLVM/libgcc. The changes in this PR would therefore only be useful for C/C++ compatibility with Clang and gcc.

daurnimator · 2024-05-28T12:38:35Z

I could have sworn there was an issue about function multiversioning/selection based on runtime-detected CPU in Zig itself but I can't find it now

#1018 ?

lacc97 · 2024-05-29T10:40:16Z

I could have sworn there was an issue about function multiversioning/selection based on runtime-detected CPU in Zig itself but I can't find it now

#1018 ?

Yep that was it. Thanks.

theseyan · 2024-11-03T06:41:05Z

Missing support for these symbols is a blocker for compiling libmdbx (a fork of LMDB) with Zig:

error: ld.lld: undefined symbol: __cpu_indicator_init
    note: referenced by gc-get.c:584 (src/gc-get.c:584)
    note:               mdbx-static.o:(scan4seq_resolver) in archive [...]/libmdbx.a
error: ld.lld: undefined symbol: __cpu_model
    note: referenced by gc-get.c:587 (src/gc-get.c:587)
    note:               mdbx-static.o:(scan4seq_resolver) in archive [...]/libmdbx.a

I really appreciate this PR and hope it gets merged soon. 🫡

slyshykO · 2024-11-03T09:22:46Z

As a temporary solution, you can use cpu_model, which provides __cpu_indicator_init and __cpu_model symbols.

lacc97 · 2025-01-19T02:32:45Z

Rebased to latest master.

lacc97 · 2025-01-19T17:02:28Z

Rebased again . I'm not entirely happy with getAmdTypeAndSubtype() and getIntelTypeAndSubtype() as it seems to duplicate similar logic as in lib/std/zig/system/x86.zig. It's not clear to me how to resolve this without bringing in the generated enum types inside the standard library, which don't really fit there.

alexrp · 2025-01-26T20:19:40Z

It's not clear to me how to resolve this without bringing in the generated enum types inside the standard library, which don't really fit there.

Is it actually problematic if the standard library @imports a file from compiler-rt or vice versa? If it just contains data, as seems to be the case, that seems to me like it'd be harmless?

lacc97 · 2025-02-02T22:30:15Z

It's not clear to me how to resolve this without bringing in the generated enum types inside the standard library, which don't really fit there.

Is it actually problematic if the standard library @imports a file from compiler-rt or vice versa? If it just contains data, as seems to be the case, that seems to me like it'd be harmless?

Made a new commit with regards to this. I moved the enum definition file into lib/std/zig/system and modified some of lib/std/zig/system/x86.zig functions to return the Vendor, Type and Subtype enums as part of detection in addition to the pointer to Target.Cpu.Model. Some of these decls are also exposed so that the compiler-rt code can use them. Now, the only logic in the compiler-rt file simply converts between Target.Cpu.Feature.Set and the actual values expected by LLVM for the target/target_clones attributes to work.

andrewrk · 2025-02-23T04:44:13Z

@alexrp are you interested in shepherding this one?

lib/compiler_rt/x86_cpu_model.zig

andrewrk · 2025-02-23T04:59:46Z

I do want a final review because it looks very messy

lib/compiler_rt/x86_cpu_model.zig

alexrp · 2025-02-23T05:09:15Z

test/standalone/x86_cpu_model/build.zig

+    if (!arch.isX86()) return;
+
+    const abi = target.result.abi;
+    if (target.result.ofmt != .elf or !(abi.isMusl() or abi.isGnu())) return;


Why limit to these targets?

I don't exactly remember why. It must have been from whatever standalone test I originally copied from. I'll remove it and see if anything breaks.

Oh, I see. On macOS:

/Users/runner/work/zig/zig/build/../lib/std/Build/Step/CheckObject.zig:506:17: 0x10a7671e1 in checkInDynamicSymtab (build) else => @panic("Unsupported target platform"),

But really, since the test is now actually testing multiversioning at runtime, is there any value in doing these symbol checks anyway?

test/standalone/x86_cpu_model/build.zig

These match what LLVM expects and generates when building multiversioned functions (e.g. functions tagged with attributes target or target_clones). The actual CPU detection reuses the logic under std/zig/system/x86.zig, and transforms it into the values that LLVM-generated code expects. These values are auto-generated using a new tool (tools/update_x86_cpu_model_enums.zig) which parses the relevant file in the LLVM codebase and generates Zig enums out of it.

- Use pointers in export builtin - Update names for Type enum tag

The new logic is closer to how it's done in C where any unspecified enum simply increments the previous value.

This lets us reuse the existing cpu detection in std.zig.system.x86. Hence, lib/compiler_rt/x86_cpu_model.zig simply translates the detected Target.Cpu.Feature.Set into the features set expected by LLVM.

Checks the exit code which depends on the running cpu feature set.

Move the functionality and code to update_cpu_features.

Instead of checking this at comptime, they are checked at the point of generation in tools/update_cpu_features.zig.

alexrp · 2025-02-24T01:48:16Z

test/standalone/x86_cpu_model/build.zig

+    if (std.Target.x86.featureSetHas(cpu.features, .avx512vnni)) {
+        run_exe.expectExitCode(3);
+    } else if (std.Target.x86.featureSetHas(cpu.features, .avx2)) {
+        run_exe.expectExitCode(2);
+    } else {
+        run_exe.expectExitCode(1);
+    }


This'll work for most cases but it isn't quite correct: Multiversioning will dynamically detect the features of the system the program is run on. The current approach breaks if I invoke this test manually with -Dcpu=.... So really, you should be checking b.graph.host features here.

lacc97 mentioned this pull request May 26, 2024

compiler-rt: Provide __cpu_indicator_init, __cpu_model and __cpu_features2 #18193

Closed

lacc97 force-pushed the compiler-rt-x86-cpu branch from 577394c to 167cb40 Compare January 19, 2025 02:27

lacc97 force-pushed the compiler-rt-x86-cpu branch from 167cb40 to a481fa8 Compare January 19, 2025 17:00

lacc97 force-pushed the compiler-rt-x86-cpu branch from 172a879 to edca94a Compare February 2, 2025 18:45

alexrp force-pushed the compiler-rt-x86-cpu branch from 44ae1fb to d793f18 Compare February 6, 2025 01:07

alexrp reviewed Feb 23, 2025

View reviewed changes

lib/compiler_rt/x86_cpu_model.zig Outdated Show resolved Hide resolved

andrewrk self-requested a review February 23, 2025 04:59

alexrp reviewed Feb 23, 2025

View reviewed changes

lib/compiler_rt/x86_cpu_model.zig Outdated Show resolved Hide resolved

alexrp reviewed Feb 23, 2025

View reviewed changes

lib/compiler_rt/x86_cpu_model.zig Outdated Show resolved Hide resolved

alexrp reviewed Feb 23, 2025

View reviewed changes

test/standalone/x86_cpu_model/build.zig Show resolved Hide resolved

alexrp self-assigned this Feb 23, 2025

lacc97 added 8 commits February 23, 2025 18:58

update_x86_cpu_model_enums: Fix eating first character in enum name

adbe3c7

compiler_rt: Cleanup compile error output in x86_cpu_model

cddd5f3

compiler_rt: Update x86 cpu enums for LLVM 18

3ecbd0f

compiler_rt: Support x86_64{,_v2,_v3,_v4} feature detection for LLVM

a71bf50

compiler_rt: Documentation

94b1860

compiler_rt: Add Intel Clearwaterforest and clarify AMD detection

a5d678f

compiler_rt: Update autogenerated file

10c7172

lacc97 added 10 commits February 23, 2025 18:58

compiler_rt: Update x86_cpu_model.zig to latest zig master

fd6dd88

- Use pointers in export builtin - Update names for Type enum tag

compiler_rt: Update x86 cpu model enums to LLVM 19

b005a52

tools: Fix update_x86_cpu_model_enums logic for calculating enum value

5c20066

The new logic is closer to how it's done in C where any unspecified enum simply increments the previous value.

compiler_rt: Move generated x86 cpu enums under std

a67fe35

This lets us reuse the existing cpu detection in std.zig.system.x86. Hence, lib/compiler_rt/x86_cpu_model.zig simply translates the detected Target.Cpu.Feature.Set into the features set expected by LLVM.

compiler_rt: Remove TODO

8f593ac

compiler_rt: Use atomics for loading cpu model

6f3a0c9

compiler_rt: Check x86_cpu_model standalone test binary

a03ac37

Checks the exit code which depends on the running cpu feature set.

tools: Remove update_x86_cpu_model

6f417da

Move the functionality and code to update_cpu_features.

tools: Reorganize x86 enum processing in update_cpu_features

0eacef7

Move x86 cpu feature compat check to tools/update_cpu_features

530463d

Instead of checking this at comptime, they are checked at the point of generation in tools/update_cpu_features.zig.

lacc97 force-pushed the compiler-rt-x86-cpu branch from d793f18 to 530463d Compare February 23, 2025 23:47

alexrp reviewed Feb 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

compiler-rt: Provide __cpu_indicator_init, __cpu_model and __cpu_features2 #20081

compiler-rt: Provide __cpu_indicator_init, __cpu_model and __cpu_features2 #20081

lacc97 commented May 26, 2024 •

edited

Loading

daurnimator commented May 28, 2024

lacc97 commented May 29, 2024

theseyan commented Nov 3, 2024

slyshykO commented Nov 3, 2024 •

edited

Loading

lacc97 commented Jan 19, 2025

lacc97 commented Jan 19, 2025

alexrp commented Jan 26, 2025

lacc97 commented Feb 2, 2025

andrewrk commented Feb 23, 2025

andrewrk commented Feb 23, 2025

alexrp Feb 23, 2025

lacc97 Feb 23, 2025

alexrp Feb 24, 2025

alexrp Feb 24, 2025

compiler-rt: Provide __cpu_indicator_init, __cpu_model and __cpu_features2 #20081

Are you sure you want to change the base?

compiler-rt: Provide __cpu_indicator_init, __cpu_model and __cpu_features2 #20081

Conversation

lacc97 commented May 26, 2024 • edited Loading

daurnimator commented May 28, 2024

lacc97 commented May 29, 2024

theseyan commented Nov 3, 2024

slyshykO commented Nov 3, 2024 • edited Loading

lacc97 commented Jan 19, 2025

lacc97 commented Jan 19, 2025

alexrp commented Jan 26, 2025

lacc97 commented Feb 2, 2025

andrewrk commented Feb 23, 2025

andrewrk commented Feb 23, 2025

alexrp Feb 23, 2025

Choose a reason for hiding this comment

lacc97 Feb 23, 2025

Choose a reason for hiding this comment

alexrp Feb 24, 2025

Choose a reason for hiding this comment

alexrp Feb 24, 2025

Choose a reason for hiding this comment

lacc97 commented May 26, 2024 •

edited

Loading

slyshykO commented Nov 3, 2024 •

edited

Loading