Build error while cmake arguments ARM_SOFTFP=1 and CLR_ARM_FPU_TYPE=none

[ivanprysyazhnyk-GL]

Device has processor Armel without any FPU features:

cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 1 (v7l)
BogoMIPS : 1397.55
Features : half thumb fastmult edsp tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x4
CPU part : 0xc09
CPU revision : 1

if dotnet is built with next command line:

./build.sh -arch armel -c release -cross -cmakeargs -DARM_SOFTFP=1 -cmakeargs -DCLR_ARM_FPU_TYPE=none

next errors appear that are required to FPU instructions:

.....
[ 25%] Built target coreconsole
/home/glip/projectdata/dotnetcore/runtime/src/coreclr/src/pal/src/arch/arm/context2.S:77:5: error: instruction requires: VFP2
vmrsne r3, fpscr
^
/home/glip/projectdata/dotnetcore/runtime/src/coreclr/src/pal/src/arch/arm/context2.S:80:5: error: instructions in IT block must be predicable
itttt ne
^
.....

Such FPU instructions exist in several *.S files.

In case dotnet is built without cmakeargs:

./build.sh -arch armel -c release -cross

compilation finishes without error but Error "Illegal instruction" appears while running on device.

[jkotas]

cc @franksinankaya @am11

[franksinankaya]

You should still be compiling with FPU instructions enabled. Kernel does FPU emulation with armel builds. Code assumes FPU presence.

[ivanprysyazhnyk-GL]

You should still be compiling with FPU instructions enabled. Kernel does FPU emulation with armel builds. Code assumes FPU presence.

In case dotnet is built without -cmakeargs -DARM_SOFTFP=1 -cmakeargs -DCLR_ARM_FPU_TYPE=none error "Illegal instruction" appears while executing it on device. However if I comment FPU instructions "Hello world" runs successfully but data abort appears in libraries when I run my application (this data abort do not exits when I execute on armhf processor).

[franksinankaya]

Yeah, I was referring to DCLR_ARM_FPU_TYPE section only.

You should not use that option on armel builds.

@janvorli thoughts?

You can maybe get the callstack in disassembly?

[ivanprysyazhnyk-GL]

Next error appears if all FPU instructions in *.S files is commented:
root@xts:~/netcoreapp5.0# dotnet TestApplication.dll
Fatal error. System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
at NLog.LogFactory..cctor()
at NLog.LogFactory..ctor()
at NLog.LogManager..cctor()
at NLog.LogManager.GetCurrentClassLogger()
at TestApplication.TestApplication..ctor()
at TestApplication.TestApplication.Main(System.String[])
Aborted

Empty application just with printing finishes successfully:
root@cube:~/netcoreapp_empty5.0# dotnet SolutionEmpty.dll
Hello World!

[franksinankaya]

You really shouldn't be commenting out the FPU instructions. Kernel is supposed to emulate the FPU instructions for you. Can you please check your kernel configuration?

I don't think you can go anywhere by commenting out FPU code.

[ivanprysyazhnyk-GL]

You really shouldn't be commenting out the FPU instructions. Kernel is supposed to emulate the FPU instructions for you. Can you please check your kernel configuration?

I don't think you can go anywhere by commenting out FPU code.

In case FPU instructions are commented out Error "Illegal instruction" appears while running "dotnet TestApplication.dll" or "dotnet --info" on device.

[franksinankaya]

Apparently, you need a new option

ARM EABI:

ABI flags passed by gcc to binutils: -mabi=aapcs-linux -mfloat-abi=soft -meabi=4

https://wiki.debian.org/ArmEabiPort

[franksinankaya]

I also learnt that kernel fpu emulation has been removed in favor of the code generation options reported on the previous message.

[smx-smx]

Hello
I'm in the same situation. I made some modifications starting from the tizen case here: https://github.com/smx-smx/runtime/commits/armel-fixes-rebase

I changed the floating point abi to "soft" (instead of "softfp"), so that no hardware floating point registers are used
The issue is that the arm argument handling code in coreclr expects d0-d7 to be always available

There are 3 kind of uses of those registers that i noticed:

• saving and restoring the registers content on the stack (can be ifdeffed, and i fixed most of those cases locally)
• pushing floating point arguments (i removed CALLDESCR_FPARGREGS in cgencpu and fixed usages, but is that enough?)
• returning float and double argument types: how can it be done without FPU registers?

[smx-smx]

I'm not sure how feasible this is tho...the JIT also assumes we have an FPU and emits FPU registers...

[franksinankaya]

according to this page, you need softfp in order to do register emulation.

https://embeddedartistry.com/blog/2017/10/11/demystifying-arm-floating-point-compiler-options/
float-abi
The -mfloat-abi= option is used to select which ARM ABI is used. This option also controls whether floating-point instructions may be used.

Here are your float-abi options:

soft: full software floating-point support
softfp: Allows use of floating-point instructions but maintains compatibility with the soft-float ABI
hard: Uses floating-point instructions and the floating-point ABI.

[smx-smx]

I think the FPU emulation is not enabled in the kernel currently running on the target, and i'm not sure if i can feasibly enable it (it's kernel 3.4.11-rt19 and has proprietary blobs and modifications that limit the kernels you can use).

Using softfp results in illegal instructions exceptions even in pure C/C++ sources (where no assembly is used explicitly)

For example, i get the following crash (gdb output)

#0  0xb6b0c98a in IpcStreamFactory::GetNextAvailableStream(void (*)(char const*, unsigned int)) () from target:/publish/libcoreclr.so

=> 0xb6b0c98a <_ZN16IpcStreamFactory22GetNextAvailableStreamEPFvPKcjE+10>:      vpush   {d8}

[franksinankaya]

Interesting, it is almost impossible to run dotnet without fpu emulation.

Looks like fpu emulation was removed in later versions of the kernel due to licensing issues

https://lwn.net/Articles/546840/

[franksinankaya]

This is about to hit dead end.

You can try qemu with binary emulation mode. I don’t know how good if would run on your platform. Performance should be 15-20% slow

[smx-smx]

Let's be positive 👍 , i think i have an alternative
Mono runs on this platform when cross compiled, but it uses the .NET 4.x profile

I previously tried to use the libraries from the linux-arm runtime but they are unsuitable (wrong libc, uses VFP, etc..)
With my patches i cannot compile coreclr itself, but i can compile the native libraries that make up the runtime (e.g System.Native) and mono by using the subset Libs+Mono.

I will try to replace the files in a published application to see if i can use mono-sgen with the netcore profile

I have this PR open: #38798 but i'll probably rebase it on master.
I think it could be a good idea to discuss about certain pieces of that PR such as

• ability to supply external toolchain file, so we're not bound to Tizen and we can use any additional platform specific option
• build fixes for linux 3.x
• build fixes for uclibc

In general i find the Runtime ID a bit ambiguous because linux-arm sounds generic but it's actually bound to a specific C library and makes assumptions about the CPU and the ABI

[smx-smx]

I'm getting this error (DOTNET_SYSTEM_GLOBALIZATION_INVARIANT required to avoid this: dotnet/macios#8906)

# DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 MONOPATH=$PWD ../netcore/bin/mono-sgen ./uclibc-arm.dll
The entry point method could not be loaded due to Could not load file or assembly 'System.Runtime, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
* Assertion: should not be reached at assembly-load-context.c:64

Aborted

The library is there, but i'm not sure if it's a version issue (5.0.0.0 vs 5.0.0-dev).
I checked the publickeytoken of System.Runtime, and it seems to match

Edit: it's related to this

Mono: DllImport attempting to load: 'QCall'.
Mono: DllImport error loading library 'QCall': 'File not found'.
Mono: DllImport error loading library 'QCall.so': 'File not found'.
Mono: DllImport error loading library 'libQCall.so': 'File not found'.
Mono: DllImport unable to load library 'QCall'.
Mono: The following assembly referenced from /publish2/uclibc-arm.dll could not be loaded:
     Assembly:   System.Runtime    (assemblyref_index=0)
     Version:    5.0.0.0
     Public Key: b03f5f7f11d50a3a

[franksinankaya]

Mono is supposed to work in this environment if that works for you.

[smx-smx]

It would have been cool to get dotnet core to work
Technically i could try to make coreclr run FPU-less, but would that be something that the coreclr team is willing to consider?

Otherwise, mono will be the only solution for constrained devices.
I'm currently looking into why libQCall is required and how i can build/provide it

[franksinankaya]

I have looked at removing FPU access before. There are also other issues where this was discussed. Biggest challenge is JIT. It assumes floating point register presence.

You can disable JIT and run with interpreter but interpreter was broken and unmaintained.

Even if you get interpreter working, performance would be terrible.

I agree that this would be a nice change. It looks like a project to me.

Feel free to go to existing issues and probe interest if you are willing to take on this work.

[smx-smx]

I'll think about it, as it's not a tiny task

As for mono, i fixed it. The official released mono doesn't support QCalls. HEAD does, but i had problems building it (has dependencies on the coreclr repository even when built outside of it)

So i used the version shipped in this repository (had to fix some symlinks first as there are issues with Makefile.am creating relative symlinks for icushim) and i created a runtime by merging CoreCLR libs, System.Private.CoreLib and mono-sgen

Finally

# DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 MONOPATH=$PWD LD_LIBRARY_PATH=$PWD /mono-sgen ./uclibc-arm.dll
Hello World!

Note that this involves pieces of CoreCLR that are built FPU-less, so the ability to build the runtime with soft instead of softfp is relevant even if you don't use CoreCLR, for the mono + libs usecase

[jkotas]

@akoeplinger Is it possible to build Mono-based runtime stack for older generation ARM processor from dotnet/runtime repo?

[akoeplinger]

@jkotas it should be possible, but might need some tweaks in mono.proj which hooks msbuild with mono's autotools-based build.

[trylek]

Can someone please confirm whether this issue can be scheduled for the .NET 6 release - or is this something we need to continue tracking in our current .NET 5 backlog?

[jkotas]

This can be closed. The initial question that started this issue was answered.