[L01 with 1.14.0.b1] UART issues (buffer overflow?)

[dmartauz]

In my application object I have a test() method making use of UART bus 1. CO2 sensor transmitting messages (with format " Z 01540 z 01575\r\n") every 500ms is connected to this bus via pins P3 and P4.
Simplified code:

from machine import UART

class Sensor():
	def __init__(self):
		self.uart2 = UART(1, baudrate=9600, pins=('P3', 'P4'))

	def test(self):
		print("uart2.any: %s" % self.uart2.any())
		while self.uart2.any():
			print(self.uart2.readline())

node = Sensor()

I am calling node.test() method manually from REPL. It works well until I keep the L01 receving data from the sensor a little longer without calling the node.test() method. It seems that in case UART buffer gets full some characters are lost or the order is mixed. This behavior keeps going on until next reset.

MicroPython v1.8.6-849-86da809 on 2018-01-17; LoPy with ESP32
Type "help()" for more information.
>>> node.test()
uart2.any: 144
b' Z 01540 z 01575\r\n'
b' Z 01514 z 01418\r\n'
b' Z 01492 z 01385\r\n'
b' Z 01514 z 01621\r\n'
b' Z 01547 z 01666\r\n'
b' Z 01544 z 01532\r\n'
b' Z 01555 z 01601\r\n'
b' Z 01589 z 01722\r\n'
>>> node.test()
uart2.any: 108
b' Z 01613 z 01710\r\n'
b' Z 01636 z 01741\r\n'
b' Z 01643 z 01666\r\n'
b' Z 01624 z 01544\r\n'
b' Z 01605 z 01530\r\n'
b' Z 01597 z 01574\r\n'
>>> node.test()
uart2.any: 486
b' Z 01593 z 01574\r\n'
b' Z 01593 z 01586\r\n'
b' Z 01586 z 01555\r\n'
b' Z 01574 z 01530\r\n'
b' Z 01578 z 01597\r\n'
b' Z 01601 z 01698\r\n'
b' Z 01624 z 01702\r\n'
b' Z 01632 z 01678\r\n'
b' Z 01624 z 01586\r\n'
b' Z 01589 z 01443\r\n'
b' Z 01570 z 01499\r\n'
b' Z 01566 z 01544\r\n'
b' Z 01559 z 01544\r\n'
b' Z 01555 z 01529\r\n'
b' Z 01566 z 01616\r\n'
b' Z 01574 z 01616\r\n'
b' Z 01586 z 01620\r\n'
b' Z 01597 z 01647\r\n'
b' Z 01597 z 01593\r\n'
b' Z 01601 z 01620\r\n'
b' Z 01609 z 01632\r\n'
b' Z 01613 z 01624\r\n'
b' Z 01617 z 01632\r\n'
b' Z 01628 z 01682\r\n'
b' Z 01624 z 01598\r\n'
b' Z 01621 z 01602\r\n'
b' Z 01624 z 01647\r\n'
b' Z 01644 z 01722\r\n'
b' Z 01640 z 01621\r\n'
b' Z 01594 z 01421\r\n'
b' Z 01567 z 01448\r\n'
b' Z 01552 z 01481\r\n'
b' Z 01567 z 01636\r\n'
b' Z 01586 z 01659\r\n'
b' Z 01590 z 01602\r\n'
>>> node.test()
uart2.any: 126
b' Z 01609 z 01698\r\n'
b' Z 01621 z 01655\r\n'
b' Z 01621 z 01632\r\n'
b' Z 01624 z 01636\r\n'
b' Z 01636 z 01682\r\n'
b' Z 01647 z 01686\r\n'
b' Z 01636 z 01598\r\n'
>>> node.test()
uart2.any: 486
b' Z 01628 z 01597\r\n'
b' Z 01640 z 01682\r\n'
b' Z 01651 z 01698\r\n'
b' Z 01655 z 01663\r\n'
b' Z 01651 z 01637\r\n'
b' Z 01663 z 01715\r\n'
b' Z 01659 z 01629\r\n'
b' Z 01663 z 01687\r\n'
b' Z 01690 z 01798\r\n'
b' Z 01690 z 01687\r\n'
b' Z 01666 z 01590\r\n'
b' Z 01640 z 01530\r\n'
b' Z 01651 z 01682\r\n'
b' Z 01686 z 01839\r\n'
b' Z 01686 z 01691\r\n'
b' Z 01659 z 01545\r\n'
b' Z 01637 z 01545\r\n'
b' Z 01644 z 01675\r\n'
b' Z 01679 z 01819\r\n'
b' Z 01674 z 01651\r\n'
b' Z 01636 z 01500\r\n'
b' Z 01648 z 01679\r\n'
b' Z 01640 z 01617\r\n'
b' Z 01628 z 01586\r\n'
b' Z 01636 z 01670\r\n'
b' Z 01644 z 01667\r\n'
b' Z 01663 z 01731\r\n'
b' Z 01682 z 01757\r\n'
b'1563 z 01526\r\n'
b' Z 01541 z 01455\r\n'
b' Z 01545 z 01560'
>>> node.test()
uart2.any: 360
b'\r\n'
b' Z 01556 z 01590\r\n'
b' Z 01571 z 01629\r\n'
b' Z 01590 z 01663\r\n'
b' Z 01587 z 01579\r\n'
b' Z 01602 z 01671\r\n'
b' Z 01621 z 01691\r\n'
b' Z 01629 z 01663\r\n'
b' Z 01629 z 01629\r\n'
b' Z 01621 z 01575\r\n'
b' Z 01621 z 01633\r\n'
b' Z 01610 z 01560\r\n'
b' Z 01590 z 01507\r\n'
b' Z 01575 z 01533\r\n'
b' Z 01587 z 01602\r\n'
b' Z 01617 z 01746\r\n'
b' Z 01633 z 01699\r\n'
b' Z 01644 z 01699\r\n'
b' Z 01644 z 01637\r\n'
b' Z 01598 z 01411\r\n'
b' Z 01556 z 01386'
>>> node.test()
uart2.any: 162
b'\r\n'
b' Z 01533 z 01463\r\n'
b' Z 01526 z 01478\r\n'
b' Z 01530 z 01545\r\n'
b' Z 01567 z 01715\r\n'
b' Z 01571 z 01602\r\n'
b' Z 01567 z 01545\r\n'
b' Z 01560 z 01526\r\n'
b' Z 01545 z 01481\r\n'
b' Z 01560 z 01617'
>>>

[robert-hh]

I do not see an issue here: if the buffer is full, the device can either stop receiving or overwrite old content. The buffer is there for ... buffering. It is up to the application to consume the data at the average rate, it arrives, and soon enough. Buffering supports the application, in that soon enough does not mean: 1 * character time, but buffer size * character time .

[livius2]

like @robert-hh i do not see issue here
If you do not need to loose some characters you must use fast read or RTS/CTS

[dmartauz]

Ok, I understand that the buffer size is limited.

Let's say that "error" in bold line of following reading is caused by buffer being full:

node.test()
uart2.any: 486
b' Z 01628 z 01597\r\n'
b' Z 01640 z 01682\r\n'
b' Z 01651 z 01698\r\n'
b' Z 01655 z 01663\r\n'
b' Z 01651 z 01637\r\n'
b' Z 01663 z 01715\r\n'
b' Z 01659 z 01629\r\n'
b' Z 01663 z 01687\r\n'
b' Z 01690 z 01798\r\n'
b' Z 01690 z 01687\r\n'
b' Z 01666 z 01590\r\n'
b' Z 01640 z 01530\r\n'
b' Z 01651 z 01682\r\n'
b' Z 01686 z 01839\r\n'
b' Z 01686 z 01691\r\n'
b' Z 01659 z 01545\r\n'
b' Z 01637 z 01545\r\n'
b' Z 01644 z 01675\r\n'
b' Z 01679 z 01819\r\n'
b' Z 01674 z 01651\r\n'
b' Z 01636 z 01500\r\n'
b' Z 01648 z 01679\r\n'
b' Z 01640 z 01617\r\n'
b' Z 01628 z 01586\r\n'
b' Z 01636 z 01670\r\n'
b' Z 01644 z 01667\r\n'
b' Z 01663 z 01731\r\n'
b' Z 01682 z 01757\r\n'
b'1563 z 01526\r\n'
b' Z 01541 z 01455\r\n'
b' Z 01545 z 01560'

What I do not understand is why following readings have '\r\n' in the first line and in last line '\r\n' is missing?

node.test()
uart2.any: 360
b'\r\n'
b' Z 01556 z 01590\r\n'
b' Z 01571 z 01629\r\n'
b' Z 01590 z 01663\r\n'
b' Z 01587 z 01579\r\n'
b' Z 01602 z 01671\r\n'
b' Z 01621 z 01691\r\n'
b' Z 01629 z 01663\r\n'
b' Z 01629 z 01629\r\n'
b' Z 01621 z 01575\r\n'
b' Z 01621 z 01633\r\n'
b' Z 01610 z 01560\r\n'
b' Z 01590 z 01507\r\n'
b' Z 01575 z 01533\r\n'
b' Z 01587 z 01602\r\n'
b' Z 01617 z 01746\r\n'
b' Z 01633 z 01699\r\n'
b' Z 01644 z 01699\r\n'
b' Z 01644 z 01637\r\n'
b' Z 01598 z 01411\r\n'
b' Z 01556 z 01386'
node.test()
uart2.any: 162
b'\r\n'
b' Z 01533 z 01463\r\n'
b' Z 01526 z 01478\r\n'
b' Z 01530 z 01545\r\n'
b' Z 01567 z 01715\r\n'
b' Z 01571 z 01602\r\n'
b' Z 01567 z 01545\r\n'
b' Z 01560 z 01526\r\n'
b' Z 01545 z 01481\r\n'
b' Z 01560 z 01617'

[livius2]

This is because \r\n is from you last read (missing)
i you read docs there is

uart.readline()
Read a line, ending in a newline character. If such a line exists, return is immediate. If the timeout elapses, all available data is returned regardless of whether a newline exists.
Return value: the line read or None on timeout if no data is available.

maybe it's not entirely comfortable that readline() read also when no \r\n exists but this is how it work now

[dmartauz]

Ok, that's explanation for '\r\n' in the first line but why is last line always without it?

[robert-hh]

If the timeout elapses, all available data is returned regardless of whether a newline exists.

[dmartauz]

I don't think that it should timeout because '\r\n' characters are transmitted continuously after the numbers.

I left method test2() from this code running for an hour and it never happened. And I doubt that my reading algorithm was in perfect sync with the transmitting sensor.

from machine import UART
import time

class Sensor():
	def __init__(self):
		self.uart2 = UART(1, baudrate=9600, pins=('P3', 'P4'))

	def test(self):
		print("uart2.any: %s" % self.uart2.any())
		while self.uart2.any():
			print(self.uart2.readline())

	def test2(self):
		while True:
			self.test()
			time.sleep(10)	

node = Sensor()
node.test2()

[robert-hh]

The mechanics are hard to know without knowing the mechanism for the serial buffer. It could be a timeout, if the buffer was full and the last \r\n did not fit into the buffer any more. If you are continuously reading, then the buffer will not overflow, and you will always get full ines. Besides that, it something you could do: set up a timer wich reads the data continously and keep the last line read. Or, when you want to read, first flush the buffer and then read until you get a full line. Or read lines and just keep the last full line.

[livius2]

Maybe there is really some problem
https://forum.pycom.io/post/15390

[dmartauz]

Other users are experiencing similar issue:
https://forum.pycom.io/topic/2595/uart-rx-buffer-issue-after-buffer-overflow

Please take a look at it.