Custom probe breaks on second boot

[TimGoll]

We designed a custom pico debug probe with a design very similar to the original debug probe. The design can be seen in the attached schematic: LibroPicoProbe.pdf

We ordered them from JLC about a year ago (09.23) and all of them worked just fine (we flashed the firmware from this repository via USB). Recently some of these boards stared dying. We first assumed that this was a hardware defect, but then investigated things further. We are still able to flash the board via USB and via SWD. After flashing it, the board automatically starts the flashed firmware and the firmware works without any issues.

However, in both cases a power cycle breaks the board and it fails to start up. Later tests with probe-rs (gdb based) showed that the memory address returned by the step command is within reasonable ranges for the first power up after the flashing process. But for all subsequent attempts, it is 0xfffffffe, which seems to be some kind of error state.

Our research hinted that this could be due to bad oscillators. This seems unlikely because these boards worked at some point, work in the initial run and the other boards from the same batch work as well. However we still tested it with two other programs. A small blinky and a multi megabyte project done by us. The first being smaller than the flasher binary, the second being larger. Both use the external crystal and both work fine. On the same hardware where the debugprobe binary breaks.

At this point we're at loss. We can only reproduce this issue with this specific binary, everything else seems to work well. Additionally we are using the same RP2040-flash-quartz combo since 2023 in multiple projects without any issues so far.

One thing that we want to verify is that the flash is not broken. As soon as we receive new flash ICs, I will desolder them from the "broken" boards and replace them with the new ones. I doubt this will fix anything, but it is worth a try.

Maybe it is something obvious that we are missing, I'm not sure. Maybe someone else has a similar problem, I at least found a few posts in some forums without any solutions:

1. https://forums.raspberrypi.com/viewtopic.php?t=342212
2. Bebug RP2040 not working (i use J-link EDU) platformio/platform-raspberrypi#27
3. https://forums.raspberrypi.com/viewtopic.php?t=325897

UF2 Bootloader v3.0
Model: Raspberry Pi RP2
Board-ID: RPI-RP2

[lurch]

Our research hinted that this could be due to bad oscillators.

You might want to try creating a custom header file for your board, and experiment with different values for PICO_XOSC_STARTUP_DELAY_MULTIPLIER? See e.g. raspberrypi/pico-sdk#457 and the various other issues that it links to.

[TimGoll]

Our research hinted that this could be due to bad oscillators.

You might want to try creating a custom header file for your board, and experiment with different values for PICO_XOSC_STARTUP_DELAY_MULTIPLIER? See e.g. raspberrypi/pico-sdk#457 and the various other issues that it links to.

That's worth a try. Thank you. I do wonder though why the problem only occurred after a year of use. Maybe some degradation of the MLCCs or the crystal over time?

[terminar]

Same problem here with a RP2040-Zero ordered from Aliexpress. Programming the debugprobe_pico.uf2 works fine, after a reboot it doesn't work anymore, I have to recopy it.

[terminar]

Update: @TimGoll in my case it was correct, I added PICO_XOSC_STARTUP_DELAY_MULTIPLIER=64 to target_compile_definitions in CMakeLists.txt, recompiled and now everything works fine. Same problem seems to be with some Adafruit feather RP2040 (I took 64 from a discussion about it).

@lurch thanks for help!

[TimGoll]

Update: @TimGoll in my case it was correct, I added PICO_XOSC_STARTUP_DELAY_MULTIPLIER=64 to target_compile_definitions in CMakeLists.txt, recompiled and now everything works fine. Same problem seems to be with some Adafruit feather RP2040 (I took 64 from a discussion about it).

@lurch thanks for help!

Awesome! Now I wonder why the board worked for about a year. I did not have the time yet to test this. Next week probably

[TimGoll]

I can confirm, this resolved our issue. It works now without any hardware changes, we used the same values @terminar did.

I'll keep this issue open because I think the default value should be increased to prevent such issues. A startup multiplier of 64 also did not add any human measurable delay to the startup.

[lurch]

I can confirm, this resolved our issue. It works now without any hardware changes

Hooray! 🎉

I'll keep this issue open because I think the default value should be increased to prevent such issues.

The default value works fine with the official https://www.raspberrypi.com/products/debug-probe/ and that's obviously the main hardware that this repo targets. So I'm going to close this issue.

[TimGoll]

that's obviously the main hardware that this repo targets

Understandable. Maybe a note in the readme would help then. Because as you can see from the comments in this issue and my linked threads in the main post, we're not alone here