Default MemoryAllocator creation crashes on Raspberry Pi OS 11

[mcrossley]

ImageSharp version

2.0.0.0

Other ImageSharp packages and versions

None

Environment (Operating system, version and so on)

Raspberry Pi OS 11 - Bullseye - 5.10.92.7

.NET Framework version

6.0.1

Description

A memory allocation error during Image.Load() when running on Raspberry Pi OS 11 Bullseye 5.10.92

Note that the same code works without error on Raspberry Pi OS 10 Buster 5.10.63.

Version 1 of ImageSharp works OK on both OS versions.

As ImageSharp only processes that one image periodically I also tried globally fixing the MaximumPoolSizeMegabytes to 1 MB, this made no difference.

Steps to Reproduce

Simply calling...
using var bmp = Image.Load<Rgba32>("./web/MoonBaseImage.png");

gives...

Unhandled exception. System.TypeInitializationException: The type initializer for 'SixLabors.ImageSharp.Configuration' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'SixLabors.ImageSharp.Memory.MemoryAllocator' threw an exception.
 ---> System.OverflowException: Arithmetic operation resulted in an overflow.
   at SixLabors.ImageSharp.Memory.Internals.UniformUnmanagedMemoryPool..ctor(Int32 bufferLength, Int32 capacity, TrimSettings trimSettings)
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Int32 sharedArrayPoolThresholdInBytes, Int32 poolBufferSizeInBytes, Int64 maxPoolSizeInBytes, Int32 unmanagedBufferSizeInBytes, TrimSettings trimSettings)
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Int32 sharedArrayPoolThresholdInBytes, Int32 poolBufferSizeInBytes, Int64 maxPoolSizeInBytes, Int32 unmanagedBufferSizeInBytes)
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Int32 poolBufferSizeInBytes, Int64 maxPoolSizeInBytes, Int32 unmanagedBufferSizeInBytes)
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Nullable`1 maxPoolSizeMegabytes)
   at SixLabors.ImageSharp.Memory.MemoryAllocator.Create()
   at SixLabors.ImageSharp.Memory.MemoryAllocator..cctor()
   --- End of inner exception stack trace ---
   at SixLabors.ImageSharp.Configuration..ctor(IConfigurationModule[] configurationModules)
   at SixLabors.ImageSharp.Configuration.CreateDefaultInstance()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at SixLabors.ImageSharp.Configuration..cctor()
   --- End of inner exception stack trace ---
   at SixLabors.ImageSharp.Configuration.get_Default()
   at SixLabors.ImageSharp.Image.Load[TPixel](String path)
   at CumulusMX.MoonriseMoonset.CreateMoonImage(Double phaseAngle, Double latitude, Int32 size, Boolean transparent) in C:\Code\CumulusMX-Core6\CumulusMX\MoonriseMoonSet.cs:line 1530
   at CumulusMX.Cumulus.DoMoonImage() in C:\Code\CumulusMX-Core6\CumulusMX\Cumulus.cs:line 2665
   at CumulusMX.Cumulus.Initialise(Int32 HTTPport, Boolean DebugEnabled, String startParms) in C:\Code\CumulusMX-Core6\CumulusMX\Cumulus.cs:line 1283
   at CumulusMX.Program.RunAsAConsole(Int32 port, Boolean debug) in C:\Code\CumulusMX-Core6\CumulusMX\Program.cs:line 320
   at CumulusMX.Program.Main(String[] args) in C:\Code\CumulusMX-Core6\CumulusMX\Program.cs:line 271
   at CumulusMX.Program.<Main>(String[] args)

If I catch the Image.Load() error and bypass the image processing completely, then a short while later whist the program is running I get another exception...

Unhandled exception. System.ArgumentNullException: Value cannot be null.
   at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
   at SixLabors.ImageSharp.Memory.Internals.UniformUnmanagedMemoryPool.TrimAll(UnmanagedMemoryHandle[] buffersLocal)
   at SixLabors.ImageSharp.Memory.Internals.UniformUnmanagedMemoryPool.Finalize()

[antonfirsov]

Based on the stack trace, the issue is not with Image.Load, touching Configuration.Default should be enough to repro this.

@mcrossley any chance you can pull down the ImageSharp source and try a debug build on the device to see which line fails exactly? I don't have a PI, and I don't see which arithmetic operations can overflow in UniformUnmanagedMemoryPool constructor, so this is going to be hard to address without community help.

[JimBobSquarePants]

@antonfirsov My money is here.

ImageSharp/src/ImageSharp/Memory/Allocators/UniformUnmanagedMemoryPoolMemoryAllocator.cs

Lines 70 to 72 in f24b02f

	this.poolBufferSizeInBytes = poolBufferSizeInBytes;
	this.poolCapacity = (int)(maxPoolSizeInBytes / poolBufferSizeInBytes);
	this.trimSettings = trimSettings;

Since GetDefaultMaxPoolSizeBytes can return values we don't control.

[antonfirsov]

Yeah but that would mean that the stack trace is lying for some reason. Nothing is impossible though.

[antonfirsov]

@mcrossley what is the value returned by GC.GetGCMemoryInfo().TotalAvailableMemoryBytes on your hardware?

[mcrossley]

Thanks folks.

Here is the position, the error is occurring with one of my testers, he has a latest model 4 rPi with Bullseye.
My dev environment is on Windows.
I do have a spare rPi 3 that I can load Bullseye on and see if I can reproduce the problem.

My main program is a bit hefty at 90k lines of source, and it only calls ImageSharp once an hour. So I will create a test harness, download the ImageSharp source, add a test for TotalAvailableMemoryBytes and run the debug version on my pi. If I cannot reproduce on a rPi 3, then it will have to go to my test user - don't you love debugging by email!

That will take me a a couple of days or so to achieve, so do not worry if I go quiet for a little while - I have a day job!

[JimBobSquarePants]

Good luck!

[mcrossley]

OK, I made a simple test program, and built a debug version of ImageSharp (as of commit v2.0.0)...

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Memory;
using SixLabors.ImageSharp.PixelFormats;

Console.WriteLine("Hello, World!");

Console.WriteLine("TotalAvailableMemoryBytes = " + GC.GetGCMemoryInfo().TotalAvailableMemoryBytes);


Configuration.Default.MemoryAllocator = MemoryAllocator.Create(new MemoryAllocatorOptions()
{
	MaximumPoolSizeMegabytes = 1
});

var bmp = Image.Load<Rgba32>("MoonBaseImage.png");

Console.WriteLine("Image Loaded OK");

Console.WriteLine("\npress any key to exit...");
Console.ReadKey();

I loaded Bullseye on my pi 3 and the the test program ran OK.

Hello, World!
TotalAvailableMemoryBytes = 969113600
Image Loaded OK

press any key to exit...

I got my test user to run it on his pi4 and it crashed...

Hello, World!
TotalAvailableMemoryBytes = -456847360
Unhandled exception. System.TypeInitializationException: The type initializer for 'SixLabors.ImageSharp.Configuration' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'SixLabors.ImageSharp.Memory.MemoryAllocator' threw an exception.
 ---> System.OverflowException: Arithmetic operation resulted in an overflow.
   at SixLabors.ImageSharp.Memory.Internals.UniformUnmanagedMemoryPool..ctor(Int32 bufferLength, Int32 capacity, TrimSettings trimSettings) in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\Internals\UniformUnmanagedMemoryPool.cs:line 38
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Int32 sharedArrayPoolThresholdInBytes, Int32 poolBufferSizeInBytes, Int64 maxPoolSizeInBytes, Int32 unmanagedBufferSizeInBytes, TrimSettings trimSettings) in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\UniformUnmanagedMemoryPoolMemoryAllocator.cs:line 73
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Int32 sharedArrayPoolThresholdInBytes, Int32 poolBufferSizeInBytes, Int64 maxPoolSizeInBytes, Int32 unmanagedBufferSizeInBytes) in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\UniformUnmanagedMemoryPoolMemoryAllocator.cs:line 53
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Int32 poolBufferSizeInBytes, Int64 maxPoolSizeInBytes, Int32 unmanagedBufferSizeInBytes) in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\UniformUnmanagedMemoryPoolMemoryAllocator.cs:line 40
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator..ctor(Nullable`1 maxPoolSizeMegabytes) in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\UniformUnmanagedMemoryPoolMemoryAllocator.cs:line 29
   at SixLabors.ImageSharp.Memory.MemoryAllocator.Create() in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\MemoryAllocator.cs:line 35
   at SixLabors.ImageSharp.Memory.MemoryAllocator..cctor() in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\MemoryAllocator.cs:line 22
   --- End of inner exception stack trace ---
   at SixLabors.ImageSharp.Memory.MemoryAllocator.get_Default() in C:\Code\ImageSharp\src\ImageSharp\Memory\Allocators\MemoryAllocator.cs:line 22
   at SixLabors.ImageSharp.Configuration..ctor(IConfigurationModule[] configurationModules) in C:\Code\ImageSharp\src\ImageSharp\Configuration.cs:line 34
   at SixLabors.ImageSharp.Configuration.CreateDefaultInstance() in C:\Code\ImageSharp\src\ImageSharp\Configuration.cs:line 219
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at SixLabors.ImageSharp.Configuration..cctor() in C:\Code\ImageSharp\src\ImageSharp\Configuration.cs:line 61
   --- End of inner exception stack trace ---
   at SixLabors.ImageSharp.Configuration.get_Default() in C:\Code\ImageSharp\src\ImageSharp\Configuration.cs:line 61
   at Program.<Main>$(String[] args) in C:\Code\TestIS\Program.cs:line 10
Aborted

Looks like TotalAvailableMemoryBytes is overflowing somewhere where they haven't used a long.

[mcrossley]

Also this closed bug on dotnet running on ARM32:
dotnet/runtime#13115

[brianpopow]

@mcrossley is this Pi4 with the issue ARM32 or ARM64? In the dotnet runtime issue they state its only happening on ARM32.

[mcrossley]

Looking at the last log file from my app that he sent me - ARM32...
2022-02-15 13:15:14.987 OS version: Unix 5.10.92.7, 64bit OS: False

[mcrossley]

As it works on my ARM32 pi3 with 2GB memory, it looks like an overflow in the OS at 4GB of memory?

[mcrossley]

Sorry, I do not know how to link to the source, but is UniformUnmangedPoolMemoryAllocator.cs the source of the issue within IS, obviously it is getting duff data from the OS, but...

        private static long GetDefaultMaxPoolSizeBytes()
        {
#if NETCOREAPP3_1_OR_GREATER
            // On 64 bit .NET Core 3.1+, set the pool size to a portion of the total available memory.
            // There is a bug in GC.GetGCMemoryInfo() on .NET 5 + 32 bit, making TotalAvailableMemoryBytes unreliable:
            // https://github.com/dotnet/runtime/issues/55126#issuecomment-876779327
            if (Environment.Is64BitProcess || !RuntimeInformation.FrameworkDescription.StartsWith(".NET 5.0"))
            {
                GCMemoryInfo info = GC.GetGCMemoryInfo();
                return info.TotalAvailableMemoryBytes / 8;
            }
#endif

            // Stick to a conservative value of 128 Megabytes on other platforms and 32 bit .NET 5.0:
            return 128 * OneMegabyte;
        }

As we are now running .NET 6.0

[brianpopow]

I was able to reproduce the issue now with docker. I tried it yesterday, but with ARM64 images, which do not seem to be affected.

I tried the following official dotnet sdk images:

• mcr.microsoft.com/dotnet/sdk:6.0.102-bullseye-slim-arm32v7
• mcr.microsoft.com/dotnet/sdk:6.0.102-focal-arm32v7
• mcr.microsoft.com/dotnet/sdk:5.0.405-buster-slim-arm32v7

(note: running those images does not work when the host CPU is x64 as far as i know)

All report TotalAvailableMemoryBytes = -402354176

Not sure why DotNetIssue13115 was closed. It seems now locked, I cannot comment there anymore.

[antonfirsov]

Seems like they closed it because the test failure somehow stopped to occur in dotnet/runtime CI. @brianpopow if you can provide very clear repro steps I think we are a good position to open a new issue against the runtime. We can link both this issue and dotnet/runtime#13115. Can you do it?

In the meanwhile, I believe we can easily workaround this in ImageSharp.

[brianpopow]

I think we are a good position to open a new issue against the runtime. We can link both this issue and dotnet/runtime#13115. Can you do it?

Yes, I can do that. Will do that tomorrow.

edit: issue opened: dotnet/runtime#65466