Spontaneous Power Cycles and Intermittent Boot Loops #249
Comments
|
We have a Sat1 in our possession that reproduces this issue. We're investigating. Stay tuned! |
|
Connected to the computer and I got these logs. 12:56:05]ESP-ROM:esp32s3-20210327 |
|
Another one... This shows a panic! [13:03:57][W][esp-idf:000][ann_read]: I (460720) HTTP_CLIENT: Body received in fetch header state, 0x3fcda5af, 305 |
|
Just wanted to share the results of all the customer troubleshooting polls here. Here is link to actual polls.
|
These logs are really helpful @philgrocks. Thank you! Please let me know if any of the information below is not accurate. For the record - below are details on the customer's device that is panicking:
|
|
@philgrocks can you tell me if this panic occurs when you've unplugged the speaker from the built-in amplifier and are just running a 3.5mm external speaker with a USB-C power adapter? Also, does the panic occur when you plug into a 5v USB-C computer or laptop connection? NOTE: You must plug in an external 3.5mm speaker as 5v will not power the amplifier and built-in speaker. |
|
I think the bellow is all correct. --PhilOn Feb 16, 2025, at 8:57 PM, Brad Davis ***@***.***> wrote:
Another one... This shows a panic!
These logs are really helpful @philgrocks. Thank you! Please let me know if any of the information below is not accurate.
For the record - below are details on the customer's device that is panicking:
The panic typically occurs immediately AFTER the customer utters the wake word. However, the customer has also seen the device fall into boot loops immediately upon powering up the device.
ESP firmware version: 0.0.4 (original factory firmware)
ESP Dashboard has not "taken control" of the device
Customer is not running any mmWave sensors.
Anker 60W USB-C power adapter reproduces the panic consistently, but this could change over time, we think.
Apple branded 29W USB-C power adapter reproduces the panic intermittently.
Another Anker 60W usb c adapter which DOES not reproduce the panic issue, but again this could change over time, we think.
Another Anker Nano 30W usb c adapter which DOES not reproduce the panic issue.
|
Using external powered speaker off of 3.5mm Jack
The panic has occurred with my 2018 MacBook Pro as power source or with USB-C power supplies. I'm always using 3.5mm speaker connection. |
|
Here is another log from this morning. A bit different (not a panic, but "abort() was called...". I was asking for the time three times over about 10 minutes. Third time it crashed. Let me know if you want me to keep posting these. [09:03:51][W][esp-idf:000][ann_read]: I (26147090) HTTP_CLIENT: Body received in fetch header state, 0x3fcca8df, 305 |
that1guy
changed the title
|
Thank you for the logs, We are investigating this issue, and have been preforming multiple tests to track down the source of the issue causing the reboots. From lowering the voltage to heating the unit up until it becomes unstable, we have managed to identify that there are a few different ways to achieve the crash logs similar to the ones philgrocks posted Specifically, if the device is powered by a USB 5V supply (as would be coming from a mackbook i believe) the buck regulator and backfeed protection diode drop the voltage to a point where the voltage is close to the limit of what the ESP32 can handle for input. We believe adding an increased capacitance on the 3.3V rail of the Core board can greatly improve system stability at the 5V or over 75C ranges |
|
A suggestion: Back in the day (80' and 90's) we used to do 4 corner testing... One axis was temperature and the other voltage. Taking the device to the limits of the specs for the most picky component in a device. Now in a residential environment I double temps are ever below 50 degrees F or more than 100... But if the device was used in an outdoor barn or something the ranges could exceed this. We would also construct a test program that kept I/O, Memory, CPU all very busy looking for glitches (memory corruption in RAM, bad data reads from ROM, etc). -- It is possible my cheap USB C meters are not very capable, but I never saw them getting in the way of 5V to 9V negotiation before. |
|
We have new hardware incoming from manufacturer and we continue running tests to squash this bug. More info incoming. |
|
Migrating notes from internal ticket, for transparency. From testing it looks like the voltage fluctuations on the 3.3v rail cause instability in the ESP32, resulting in odd behaviour. This is made worse by thermals inside the enclosure reaching 70+C causing the linear regulator to not be able to smooth out the current draw while transmitting on 2.4Ghz. Reboot at 72C ESP32 internal temp, 74.35C on AHT20. Board was problem Rev4.1 board recieved from as a return from customer experiencing reboots. This behaviour was recreatable multiple times on this Core Board Voltage trace taken at point of reboot, the voltage droped to 2.48V and resulted in a "Core Panic" |
|
Using the same setup as above, adding a 100uF Electrolytic capacitor to the 3.3v rail, pushed the temperature at which the Core board became unstable. Reboot at 100C ESP32 internal temp, 94.41C on AHT20. Board was problem Rev4.1 board returned by a Customer, with addition of 100uF on the 3.3v rail I believe i missed the exact point where the reset happened in the scope trace |
|
Second test. Aborted due to passing 110C |
|
Other boards we tested performed better, but all exibited some form of issues with fluctuations on the 3.3v rail, If the voltage droped below 2.48V odd issues where likeley to occur. We have fabricated a new Core board to test the solution, the results can be seen below. Test aborted at 111.4C(232.52F) ESP32 internal temp, 101.3C(214.34F) on AHT20. Board is Rev6, with addition of 200uF on the 3.3v rail, change of 3.3v regulator to LM1117, and 100uF added to 5v Rail. 
|
|
FYI... I saw a note about the thermal testing and the the results were worse when the board was mounted in an enclosure. My board is not in any enclosure and it is at room temperature. Around 72 degrees F. |
|
I have not been able to pull any logs--and it sounds like the culprit has been identified and this is just further confirmation--but my Sat1 went into an enclosure for the first time today and I pretty immediately saw this boot loop behavior with no change to software or use. So that would lend credence to the idea that this is a thermal issue. |
|
our best current theory of what is happening is that at one point the voltage to the SDRAM chip is low enough for it refuse data transmission. since the esp chip itself is less sensitive to low voltage than the SDRAM chip, it detects this and reboots. depending on what the esp has been doing it reports a different reset reason (can be read in home assistant) and/or error logs. as for causes, as far as we can tell this sort of reboot can be forced by either running the whole system on under voltage, running it at high temperatures (above ~70C) or a combination of both. our guess is that the higher temperatures seem to increase noise on the power rails which in turn produces spikes low enough to cause the SDRAM chip to fail. in the long term this will be fixed by reevaluating the power circuit. our current short term mitigation strategy is to run the system at 9V and add capacitors to the 3.3V and 5V rails of the core board to stabilize both the power for esp32 and the SDRAM chip. a similar setup is in testing with 7 devkits currently running for 5 days. test6 all new sat1s 9v, all enclosures, leds 100%, media_player pink noise, added 3.3Vcore caps
|
All I get is solid white LED with clicking sound from the speaker. Did factory reset, reloaded the firmware. Logs attached.
esp-web-tools-logs.txt
The text was updated successfully, but these errors were encountered: