On arm64, I get "cannot allocate memory (os error 12)" on executing spin up command

[theagilecoder]

This is regarding the quickstart from the docs.
It runs on my X86 laptops just fine. But on arm64 platforms such as Raspberry Pi 3 (1 gb ram) or Arm based EC2 instances(with 2 GB ram), spin build works without any issues but on spin up I get the following error:

Error: failed to create stack pool mapping
caused by:
0: mmap failed to allocate 0x7d3e8888 bytes
1: Cannot allocate memory (os error 12)

Am I low on ram ? do I need more for Arm ?
Image attached.

[itowlson]

@lann @dicej Is one of you the best person for this?

[dicej]

@alexcrichton would probably know where to look first.

Wasmtime does do a 2GB allocation for guard pages (see https://docs.wasmtime.dev/contributing-architecture.html#linear-memory for details), which is what seems to be failing here. That should only require virtual address space, not physical memory, though. There may be a way to disable the guard region in exchange for slower runtime performance (e.g. Wasmtime will need to do explicit checks for each memory access), but I'd be surprised if that's necessary -- even on a Raspberry Pi.

cc @jpflueger since I believe he's run Spin on Raspberry Pi in the past.

[alexcrichton]

This is probably one of two things happening:

1. The alloctable virtual address space is being exhausted. This may be lower on a Raspberry Pi than normal server-style systems, not sure.
2. The kernel ran out of memory tracking VMAs for this virtual memory allocation.

I dunno what to do if it's (2) but you're probably running into (1). You can test this out by frobbing some env vars configured here, the main one being SPIN_WASMTIME_TOTAL_MEMORIES and turning that down to be less than 1000 (e.g. 100).

The error here is about the "stack pool mapping" which happens here, notably after the memory/table pools were already configured. That means that most of virtual memory has already been reserved which is what leads me to think that this is address space exhaustion and turning down the knobs may be able to help. If this does help it's something that can be exposed programmatically or perhaps through auto-detection too.

[theagilecoder]

how can I make a build of spin where the value of SPIN_WASMTIME_TOTAL_MEMORIES is modified ? Do I use the build.rs file ?

[rylev]

@theagilecoder these are environment variables, so you just need to run spin with the SPIN_WASMTIME_TOTAL_MEMORIES environment variable set (e.g., SPIN_WASMTIME_TOTAL_MEMORIES=100 spin up).

[ashishsinghin]

@rylev I try above suggestion with 1gb RAM aws amazon linux, build successful but SPIN_WASMTIME_TOTAL_MEMORIES=100 spin up still get the same error. below see the attached screenshot.

[tschneidereit]

@theagilecoder or @siashish can you post the output of running ulimit -a on the system you're testing on?

[radu-matei]

To validate an assumption here, could you test by passing the --disable-pooling flag as well, please?
(spin up --disable-pooling)

Thanks!

[seungjin]

https://discord.com/channels/926888690310053918/1215979463918096424/1216015906941964418
SPIN_WASMTIME_TOTAL_MEMORIES=100 didnot help.

[seungjin]

Is running with --disable-pooling recommended?

[hffmnn]

I did some experiments on AWS where I had the same issues as described above and found the following:

• Setting SPIN_WASMTIME_TOTAL_MEMORIES env to 50 allowed me to run on arm64 with 2048MB RAM. Without the env it crashes.
• On x86_64 it runs without the SPIN_WASMTIME_TOTAL_MEMORIES on 2GB
• Adding the --disable-pooling lets me run on 128MB on both CPU architectures

But as @seungjin already mentioned: What are the implications of running with --disable-pooling?

[seungjin]

My workaround for now is giving additional 2G swapfile.
Now spin up works without any trick.
My VM is Oracle freetier(X86_64) that comes with 1G memory, 1G swap(partition), 2cores.

Since I am purposing my app for self-hosting, I am targeting mostly provided freetire VMs(AWS, Google, Azure) which is 2cores(X86_64), 1G ram machine.

[lann]

But as @seungjin already mentioned: What are the implications of running with --disable-pooling?

Under some circumstances you would expect reduced performance under high load. That is probably not a major concern on these "small" systems so disabling pooling is appropriate.

I think it would be reasonable to detect systems with <2GB (?) RAM and just disable pooling, assuming it would be significantly more effort to fine-tune pooling settings.

[tschneidereit]

I find it a bit puzzling that the pooling allocator fails here: it really shouldn't cause all that much additional real memory usage over the on-demand allocator.

I wonder if these systems also restrict the amount of virtual memory available? And if so, is that something we can detect?

[lann]

I wonder if these systems also restrict the amount of virtual memory available?

I think this is likely, via either the vm.overcommit_memory kernel param or maybe ulimit -v.

[alexcrichton]

This came up in bytecodealliance/wasmtime#8607 and on the bytecodealliance Zulip and I think I got a fix for this in Wasmtime which I've ported over to Spin as well