-
Notifications
You must be signed in to change notification settings - Fork 7.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ESP32-S2 I2C: lengthy gaps (about 10 ms) between transactions. (IDFGH-7169) #8770
Comments
I believe this is related to If I raise the I2C frequency only slightly, then the large gap goes away. For instance, at 125000, the gap reduces from 10ms to 600-700ish microseconds. So there is something unusual about the 100kHz rate compared to higher rates. |
Hi @dhalbert , I tried on my side with the following parameters:
The code I used is heavily inspired by the one you linked from ESP Forum. So I don't know if this problem only hits the ESP32-S2, I don't have one right now to test. Please, could you try to reproduce it with IDF and share your example with me? |
@o-marshmallow What I2C peripherals do you have available? I would like a real I2C device to communicate with, so I can get ACKs. I'll do a simple example with an MSA301 or PCA9685 or something like that. Thanks. |
Hi @dhalbert , |
I wrote a short test but could not reproduce the problem. |
@dhalbert, you are right, if you have ay other task that has the same or higher priority than the one performing the I2C one, it is possible that FreeRTOS pre-empts your I2C task to let another one, of same of higher priority, run. In this case, you will see a delay between the two transfers. Regarding the frequency, my hypothesis is that increasing the I2C clock frequency will let the transfer finish earlier, thus you'll still have time to start the second transfer before being pre-empted by another task. So, what you can do is raise the priority of the I2C task before initiating both your transfers, thanks to |
@o-marshmallow I have managed to reproduce this with a very simple example. It is not a task issue, as far as I can tell. The key is that I set the clock speed to 240 MHz instead of 160 MHz. Here is the simple ESP-IDF example, which is a cut-down version of the I2C example in the esp-idf. This example builds for an Adafruit Metro ESP32-S2 board, which is based on the WROVER module, but with only slight changes it should be usable on an Espressif or other dev board. It writes 4 bytes (could be less) to a register on an I2C device over and over again. I picked an arbitrary device, an MSA311, but it does not really matter what device it is, as long as it will ACK writes. When the clock frequency is 240 MHz (the choice in When the clock frequency is 160 MHz, the gaps disappear: I also discovered that adding a very small delay at a crucial point in the ESP-IDF I2C code will fix the problem. The very end of esp-idf/components/driver/i2c.c Lines 1371 to 1374 in b8050b3
If I add a small delay immediately after that, the problem goes away: i2c_hal_update_config(&(i2c_context[i2c_num].hal));
i2c_hal_trans_start(&(i2c_context[i2c_num].hal));
for (size_t i = 0; i < 50; i++) { // ADD DELAY HERE
asm("");
}
return;
} The delay must be after So, the key to producing the problem is the higher clock speed, and the key to the fix is a slight delay, which perhaps is related to the I2C hardware in some way. |
@dhalbert Thanks for your example! I was able to reproduce the issue on an ESP32-S2 I just got. This issue also happens on the S3 and C3even with a CPU frequency of 160MHz. But your fix doesn't work though. Adding a delay at the end of After investigating, I have found that the problem is not linked to the I2C hardware IP but to FreeRTOS. In fact, after starting an I2C transfer, we wait for any I2C event (NACK/END/etc...), you can see this on the line 1490:
The counterpart of it is in the interrupt handler, line 527:
but also, and mainly, line 1318 and 1327:
For some reasons, the function I have tried to get rid of the You can apply it with:
I haven't checked in depth yet why |
Thank you @o-marshmallow! Were you testing against I was testing against c29343e, the tip of I experimented with lowering the CPU frequency to 160 MHz to see if that was the issue. That did not help. With your patch, re-adding my small delay (as above) also did not help. But when I went back to the original I tried advancing circuitpython to So I am not sure how to continue testing in the short run. Your patch works on v4.4 on the You might test my Other minor notesYour patch did not quite apply, because the BTW, I got this compilation warning, which you probably know about:
|
Hi @dhalbert , Yesterday I did my test on Unfortunately, I can't help with In order to track this issue in IDF, I toggled a GPIO as a way to "see" what is going on before and during the I2C transfer. When I toggle the GPIO after the In any case, I am going to create an internal fix for this bug in |
I did almost exactly the same thing: I turned a pin on before the gpio_set_direction(5, GPIO_MODE_DEF_OUTPUT); // Redundant after the first time.
gpio_set_level(5, true);
portBASE_TYPE evt_res = xQueueReceive(p_i2c->cmd_evt_queue, &evt, wait_time);
gpio_set_level(5, false); If I removed the I also found that running at 160 MHz instead of 240 MHz fixed the problem. This is why I suspected a hw timing problem. I just want to make sure that you tested your fix at 240 MHz. I am wondering if there are two bugs here: one hw timing, and one task-related. I am one of the core developers of CircuitPython, so I opened an issue already and assigned it to myself. 🙂 A user found this problem originally, while trying to use an I2C motor controller at high speed. |
Hi @dhalbert , I confirm I tested with both 160MHz and 240MHz, the delay between two transfers is slightly less with the latter. The device is still the 24C16 EEPROM. As you said, maybe there are two bugs here and we only solved one but I can't get the second one to occur with IDF only. |
Apologies, I missed your reply until today. Which ESP32-S2 board(s) do you have, and are they WROVER or WROOM? I can give you a .bin or similar with the problem, and also a .zip or similar of the repo snapshot. |
@dhalbert Currently, I only have a single one, it's an ESP32-S2-WROOM. |
I think I have reproduced the problem I was seeing in CircuitPython by modifying my original example to add another task. This should be a lot easier to debug. Test programThe test program is now in this public repo: https://github.com/dhalbert/i2c_delay. I added a second task to this program. It doesn't do anything but delay for a very short time and then yield for 1 tick. It runs at priority 5, though I think that is not essential. This roughly mimics a high-priority task CircuitPython uses to do TinyUSB work servicing. I am building https://github.com/dhalbert/i2c_delay with Below is what I see without and with your patch. I am not including my delay patch at all. Without your patch
With your patch
These timeouts are like the errors I was seeing in CircuitPython.
So it appears your patch works in the simplest case, but not when there is another task present that is doing some work. If I reduce the timeout in the other task to 0, by doing A note on I2C devicesI was able to reproduce your results of not being able to get rid of the original 10 ms delays by using a different I2C device, a 24LC32, which is an EEPROM very similar to your 24C16: same commands, same I2C address. I think these I2C EEPROM's may have unusual timing characteristics. For instance, the data sheets say they are non-responsive when an EEPROM write is in progress. I think it might be better to test with the TCA8418, and just do a simple write to some innocuous state register (like setting pullups or something)., kind of like what I am doing with the MSA311. If you can get some typical sensor like an MSA301 or MSA311 eventually, that would also be a good test. I realize you may be at home with little opportunity to get more devices at the moment. In the long run, it would also be good to test with devices that are known to do clock stretching. We have a short list here. Thank you for working on this. |
I am also seeing this issue. Maybe some extra data points will help. IDF 4.4.1 There is an approximate 335 us delay between the call to start the I2C transaction and the actual start of the bits on the wire. That is around 80,000 CPU clocks of doing nothing. In case anyone is tempted to suggest that 335 us is not a lot of time, I would point out that on a battery powered application that is a truly massive waste of power. It's enough of a problem that I may have to consider bit banging directly (which might also correct the incorrect I2C clock rate, which is arguably a second bug). @o-marshmallow I tried your patch, but it kernel panics on the idle task. |
Hi @readmodifywrite , your problem is not linked to the one @dhalbert is encountering. In fact, ESP32 doesn't have this bug. While checking your issue, I've found a potential bug for the frequency issue you are encountering. One part of the fix is to use a smaller pull-up resistor. When I did my test locally, I also got a frequency of 357KHz, I was using a 4.7KOhm resistor, I switched to a 1KOhm one and now I get 370KHz. Regarding your 335us delay I suspect you are using |
Hi @dhalbert , Sorry for the delay. Thank you very much for your example, I was able to reproduce the issue and as you said, using the 24C16 I first tested with the fix I gave earlier and I also got the timeouts. These issue are really volatile, as soon as I add a small delay, everything goes back to normal... My hypothesis is that there was a race condition between the moment the I2C task is about to enter the critical section inside I didn't dig too long this fix and instead I got back to the original code that is on Here is the patch you can apply directly on top of master: |
@o-marshmallow: i2c_delay.patch.txt works for me! I tested it in CircuitPython, altering the patch slightly to make it work on v4.4: Thank you very much! I think we will incorporate this into our fork of |
@o-marshmallow I'll try your patch out when I can get around to it. But could we just get that fixed in an official release? I'd like to avoid doing manual patching in my builds if I can avoid it. You got me on the i2c_cmd_link_create, etc! That's exactly how I was doing it. I didn't realize that was doing a heap allocation each time, that's definitely not what I want. Thanks very much! |
@readmodifywrite Indeed, it will be fixed officially. The patch is already under review, I only provided it to you so you could test right now without waiting mster branch to be updated 😄 No problem, happy to help you! Note: If your transactions don't differ a lot from one to the other (only the data buffer changes), you can even re-use the command link, so no need to free and re-create. In that case it would be even faster than
In such cases, you can simply modify the pointer of data, the I2C command link can be reused. This is feature available since IDF v4.4 (commit 7dd499d) |
@dhalbert Awesome! I am happy to hear that! |
I2C delay fix from espressif#8770
Thanks @o-marshmallow! |
@o-marshmallow BTW, as I mentioned before that there are some missing i2c fixes in stable branches so it's difficult to trace the issues. |
Interesting. Just stumbled upon this thread. I admit I'm new to low level troubleshooting of I2C and don't fully understand the cause, but I've found that simply increasing the ESP32 I2C timeout to around 66% of the maximum (easier to talk in % than numbers as different chip revisions store the timeout in different numbers of bits) seems to guarantee success in my applications. Explanation in my PR, but this does not require an esp-idf change if anyone needs a fix in the meantime. adafruit/circuitpython#6594 |
Hi @litui , This issue didn't concern ESP32 boards, only newer versions such as the ESP32-S2, ESP32-S3, etc. I checked your issue, PR and repo and I am not sure to undertstand the problem you were having. |
@o-marshmallow yes, the timeout provided to each of those functions did not register a change. It was only by changing the overall timeout value that I saw success with my devices. |
@o-marshmallow |
I do not believe this issue is resolved. Testing 4.4 on ESP32 and I'm getting 19ms between master writes. I'm recycling the same cmd link and just starting it again. If there's something I'm missing I'm open to suggestions. |
The fix cb411a8 is committed after v4.4.2. Update to latest v4.4 branch to verify it. |
Yes, I am using the latest 4.4 branch and getting the same results. Not 4.4.2. As previously mentioned, changing the Freertos tickrate has an impact on this. I'm trying to get 1ms intervals. Changing the tickrate from 100hz to 1000hz gets me there, but ideally this would not be the case. I've opened a new issue as this may be a separate issue with the code impacting ESP32 and not ESP32-S2? |
v4.4-351-g121ddb87e5
xtensa-esp32-elf-gcc (crosstool-NG esp-2021r2-patch3) 8.4.0
Problem Description
Using I2C at 100_000 Hz (standard clock rate) cause ~10ms delays between I2C transactions. Raising to 125_000 causes the delays to go away. In between, only some I2C speeds work; others produce glitches that cause the I2C device to fail. Testing on a PCA9685. A colleague tried another device and was able to get it to work at 110_000.
Expected Behavior
There should not be excessive delays at 100kHz. I was suspicious that 10ms is close to the FreeRTOS tick interval, but the gaps vary sometimes between 9.something and occasionally as low as 7.something, so that may be a red herring.
Steps to reproduce
Tested in CircuitPython, which currently uses the ESP-IDF version shown above.
Code to reproduce this issue
See adafruit/circuitpython#6263. These forum posts report what sounds like the same issue:
https://www.esp32.com/viewtopic.php?p=95404
https://www.esp32.com/viewtopic.php?p=95405
I have Saleae traces if that is of interest.
I spent quite a bit of time inside the IDF I2C code, adjusting timeouts (
ticks_to_wait
), printing things, and trying various tweaks, and I cannot locate the source of this issue. I am wondering if it is hardware or hardware-setup related.The text was updated successfully, but these errors were encountered: