MemoryError with simple imports on Pico W on 9.0.0-alpha6

[michalpokusa]

CircuitPython version

Adafruit CircuitPython 9.0.0-alpha.6 on 2023-12-12; Raspberry Pi Pico W with rp2040

Code/REPL

import gc

print("Before import", gc.mem_free())
try:
    import adafruit_datetime
    # from adafruit_datetime import datetime
except Exception as e:
    print(e)
print("After import", gc.mem_free())

Behavior

Importing some libs on 8.2.9 works as expected, and allocates only a small portion of memory, the same code/imports on 9.0.0-alpha6 errors with MemoryError, and despite not importing the module uses nearly all available memory.

From my testing this does not happen on ESP32-S2, but maybe this is due to the higher amount of memory on these boards.

Description

Importing whole library on 8.2.9.

Importing only datetime class on 8.2.9.

Importing whole library on 9.0.0-alpha6.

Importing only datetime class on 9.0.0-alpha6.
Notice how despite not importing the lib successfully, the gc.mem_free() returns a 4k (I guess bytes).

Additional information

My guess is, CircuitPython 9.x.x changed something regarding memory allocation. I was able to find this live from Adafruits YT channel, when they show something that looks like the problem presented here.

[RetiredWizard]

I'm thinking I'm missing something obvious, I can't seem to reproduce this.
my code.py

import gc

print("Before import", gc.mem_free())
try:
    import adafruit_datetime
    # from adafruit_datetime import datetime
except:
    pass
print("After import", gc.mem_free())

soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Before import 130224
After import 130064

Code done running.

EDIT: I just noticed that I missed pasting in the device/version banner somehow...

>>> 
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Before import 130224
After import 130064

Code done running.

Press any key to enter the REPL. Use CTRL-D to reload.

Adafruit CircuitPython 9.0.0-alpha.6 on 2023-12-12; Raspberry Pi Pico W with rp2040
>>> 

[michalpokusa]

That is strange, there is literally nothing else on my Pico unit that ran before code I posted. I don't think it can be caused by some physical damage to my Pico, can it?

I used the latest build, downloaded jsut before posting Issue, and latest version of adafruit_datetime, I can't think of any other varaible here. 🤔

[deshipu]

Anything in settings.toml?

[dhalbert]

Are you using the .py or the .mpy version of adafruit_datetime?

[michalpokusa]

@deshipu Empty settings.toml.

@dhalbert I use .py version of adafruit_datetime.

EDIT 1:
When using the .mpy version, it seems to work (same code as above):

@RetiredWizard are you sure you have adafruit_datetime in your /lib folder? I just removed everything from the board to start from scratch and got same results as you, but noticed I forgot to install the lib. Please try running:

import gc

print("Before import", gc.mem_free())
try:
    import adafruit_datetime
    # from adafruit_datetime import datetime
except Exception as e:
    print(e)
print("After import", gc.mem_free())

[dhalbert]

The .py version has to be compiled, and that can cause fragmentation, even if the RAM eventually used is about the same.

[michalpokusa]

The .py version has to be compiled, and that can cause fragmentation, even if the RAM eventually used is about the same.

I believe this could also happen when trying to execute user code, not only when importing libraries.

Also, which is important, it used to work back in 8.2.9 and Pico W is a very popular board. I think it will happen to a lot of people when 9.0.0 comes out of alpha.

I guess one could avoid that by compiling user code, but if I am correct it is not a intended use case.

[RetiredWizard]

I was not importing the .py version of the library. I had considered it might have been a python library issue and tried using the .mpy. With the .py library in /lib/, I don't get the memory allocation error but I do see the high memory use: After import 4080.

[dhalbert]

MicroPython is doing some split-heap tuning, which we could test out: #8729

[tannewt]

The after number after the exception is a red herring because gc.mem_free() does not ensure that gc.collect() is run first. gc.collect() is run after a successful import.

I'm looking into why the import process doesn't have enough memory.

[tannewt]

So, I think the addition of USB host in 9.x is the main cause of this regression (not the split heap stuff). It takes ~12k extra static RAM. At least 5k of that is code that needs to run even when the flash is being accessed.

CIRCUITPY_USB_HOST=0:
RAM: 91956 B 256 KB 35.08%

CIRCUITPY_USB_HOST=1:
RAM: 104392 B 256 KB 39.82%

LWIP and picodvi also have large RAM footprints even when they aren't in use as well.

I think that using mpy files when this happens is a good work around. I could try to tune RAM use for this specific case but I'm sure there are others that would still fail. We should more broadly audit static RAM use and move more stuff to dynamic allocation now that we always have an outer heap.