Comparing instruction sets with Hwloc.jl, ccall(:jl_dump_host_cpu, etc) and CpuId.cpufeatures()

[miguelraz]

Hello!
This package was presented at the Parallel workshop at JuliaCon and I am trying to see the consistency of different Julia packages to show cpu instruction sets so as to dispatch on them.

So far, I have 3 ways of procuring them.

using CpuId, Hwloc
# cat /proc/info
proccpuinfo() = read(pipeline(`cat /proc/cpuinfo`, `grep flags`, `uniq`, `cut -c 10-`, `tr ' ' '\n'`, `tr a-z A-Z`, `sort`),String) |> chomp |> x -> split(x, '\n')
fs_proc = proccpuinfo()

# jl_dump_host_cpu
function jldumphost()
	read(`julia -e "ccall(:jl_dump_host_cpu,Cvoid,())"`, String) # FIX PLZ IZ BROKEN
	jldump = "sse3, pclmul, ssse3, fma, cx16, sse4.1, sse4.2, movbe, popcnt, aes, xsave, avx, f16c, rdrnd, fsgsbase, bmi, avx2, bmi2, sahf, lzcnt, xsaveopt" |> x -> split(x, ", ") |> sort .|> uppercase;
end
fs_jldump = jldumphost()

# CpuId
fs_cpuid = String.(cpuidfeatures());

Although Hwloc.jl does provide a binary for obtaining this information (hwloc-gather-cpuid), I would like to PR it into the repo if possible.

Howerver, I wasn't able to find information in the manual about how to unmask the flags and know which instruction sets are actually available.

Any pointers?

Here is my result for ./hwloc-gather-cpuid

mrg@mrg~/.j/p/H/1/d/u/b/cpuid> cat pu1
# mask e[abcd]x => e[abcd]x
1 0 0 0 0 => d 756e6547 6c65746e 49656e69
1 1 0 0 0 => 306c3 1100800 7ffafbbf bfebfbff
1 2 0 0 0 => 76036301 f0b5ff 0 c10000
1 3 0 0 0 => 0 0 0 0
5 4 0 0 0 => 1c004121 1c0003f 3f 0
5 4 0 1 0 => 1c004122 1c0003f 3f 0
5 4 0 2 0 => 1c004143 1c0003f 1ff 0
5 4 0 3 0 => 1c03c163 2c0003f 1fff 6
5 4 0 4 0 => 0 0 0 0
1 5 0 0 0 => 40 40 3 42120
1 6 0 0 0 => 77 2 9 0
5 7 0 0 0 => 0 27ab 0 9c000400
1 9 0 0 0 => 0 0 0 0
1 a 0 0 0 => 7300403 0 0 603
5 b 0 0 0 => 1 2 100 1
5 b 0 1 0 => 4 8 201 1
5 b 0 2 0 => 0 0 2 1
5 d 0 0 0 => 7 340 340 0
5 d 0 1 0 => 1 0 0 0
5 d 0 2 0 => 100 240 0 0
1 80000000 0 0 0 => 80000008 0 0 0
1 80000001 0 0 0 => 0 0 21 2c100800
1 80000002 0 0 0 => 65746e49 2952286c 726f4320 4d542865
1 80000003 0 0 0 => 37692029 3137342d 20514830 20555043
1 80000004 0 0 0 => 2e322040 48473035 7a 0
1 80000005 0 0 0 => 0 0 0 0
1 80000006 0 0 0 => 0 0 1006040 0
1 80000007 0 0 0 => 0 0 0 100
1 80000008 0 0 0 => 3027 0 0 0

[eschnett]

The official documentation for this is the "Intel 64 and IA-32 Architectures Software Developer’s Manual", available e.g. here.

Decoding the CPU flags is very tedious. I'd go for read("/proc/cpuinfo", String).

If your intent is to use this information for code generation, the you'll have to rely on LLVM in the end, thus asking LLVM what it thinks about the CPU might in the end be the best way:

julia> ccall(:jl_dump_host_cpu, Cvoid, ())
CPU: skylake
Features: sse3, pclmul, ssse3, fma, cx16, sse4.1, sse4.2, movbe, popcnt, aes, xsave, avx, f16c, rdrnd, fsgsbase, bmi, avx2, bmi2, rtm, mpx, avx512f, avx512dq, rdseed, adx, clflushopt, clwb, avx512cd, avx512bw, avx512vl, pku, sahf, lzcnt, prfchw, xsaveopt, xsavec, xsaves

[miguelraz]

Sounds great!
I will close the issue then.