end to end latency over 50ms #53
Comments
|
FYI: I tried reducing the latency for a project as well and best I got was ~50ms with my setup using this driver. |
|
Thanks , interesting - not sure what they have done (if anything) to the aes67 stuff to make it lower latency. I can hack together simple TX/RX command line stuff test stuff and send raw stereo PCM audio over UDP point to point with |
So you have 1.2s latency when you play on the local audio card too ? |
|
The expected audio latency when running the default configuration on bare metal (no containers/VMs} with a 1ms Source is about 1.5ms. See picture from Dante Controller included in #17 |
|
playing on the local PI audio card is fine. |
|
ok I'm trying with another PI and building AES67 from scratch. However, the driver fails to build now: |
|
It seems the playback issue is the buffering for alsa plughw:RAVENNA. I tried another setup with a modern desktop PC with Lubuntu 64 bit and still got the same delay when it was a player.
|
|
ok, I see: when you measure the latency you include the time spent in the ALSA playback buffer and this depends on the size of such buffer. Please consider that the shorter the buffer the higher the chance of under-run errors on the ALSA playback device. The driver architecture was meant to minimize the network latency: the whole process of RTP packet creation, timestamping, scheduling and delivery to the network interface is fully implemented in the kernel and scheduled by a kernel high resolution timer whose period is set by the tic_frame_size_at_1fs daemon parameter. |
|
any update on this ? What end-to-end latency could you reach and how do you measure it ? Thanks |
|
Did a quick test with some tone bursts and an oscilloscope. send sound to local DAC at the same time as sending to AES67 alsa device. Using small buffer sizes I could get down to around 20ms but as you mention small buffer sizes cause under runs. Have other issues somewhere with slow sync drift maybe related to PTP so trying to solve those first. Unfortunately run out of time for a while on this. Hope to revisit it in a week or so. |
|
OK thanks for the info. |
|
any news on this ? |
|
It looks like it might be more like 50-100mS for best reliability on a normal Linux system. Thats just about usable for audio playout. |
|
I have improved latency a little bit on the receiver by running a jack server to handle playback on this device. This gives a latency of around 70mS. However it seems this driver does not work with jack as an input device. (also doesn't work as out put device) Also the latency slowly increases - after half an hour it crept up by 30mS. Using this results in an input device error:
|
|
what about trying with (corrections in bold): |
|
Regarding the drift I don't know how jack handles the audio clock drift between the capture and the playback devices. |
|
thanks for the suggestions. On my receiver the jack server runs and I can play a test sound through the output: I can also test without jack and use alsa which works as before (but not what I need): However I can't loop jack like this - no sound! - it is patching in jack though. I recorded the stream as a wav file: AES67-daemon all ok no errors and it clearly works with plain alsa. Any clues? I really need it working this way full-duplex with jack, then I can test properly. Re:drift - understood I can see this might happen. Maybe there is a way to get the aes67-ptp clock into gstreamer? |
|
apparently jackd doesn't do any clock drift correction so my suggestion is to not use it for this use case. |
|
Thanks looks like that is a possible solution |
|
good news on this issue. |
|
Excellent - look forward to testing the next version. |
…n to investigate #53 The script run_latency_test.sh can be used to run the test. The latency utility measures the end to end latency by adding timestamps in the stream at playback time and checking the timestamps when the packets are received on the capture device. The average measured end to end latency is printed at the end of the test. The latency utility is based on a modified version of the original ALSA latency tool.
|
if you checkout the end_to_end_latency branch you can run a test to measure the real end to end latency introduced by RAVENNA playback and capture devices.
To run a 10 seconds latency test.
A patch to the driver to fix this will follow soon. |
|
I tried the alsaloop playback method which works, but there is a drift over a few minutes that slowly increases then an under-run then the process repeats as show below: `sudo nice -n -10 alsaloop -c 2 -r 48000 -f S24_LE -P hw:sndrpihifiberry -C hw:RAVENNA -S 5 -v New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 1.00000000 (min/max samples = 0/0) |
|
I will be releasing a patch for the problem with the high latency. |
|
The different sync options appear to make no difference. The resampler option does not run at all. |
|
I have just pushed a change to the driver repo to allow the usage of smaller ALSA buffer periods. This happens because we can now program the driver to use a smaller number of periods. Indeed the RAVENNA buffer size is now set 128 frames. @steeley My suggestion is to specify a latency > 4ms when using alsaloop (-t 4000) and to use a buffer size > 512 frames when using aplay ( --buffer-size 512). Thanks ! |
This requires some investigations on alsaloop |
|
To test with the new driver checkout the end_to_end_latency branch cleanup the repo using the cleanup.sh and rebuild using build.sh |
|
thanks - will look at this next week. |
|
Hi -- very interested in this work on lower latency. I just finished compiling the 5.16-rc3 kernel for rpi4B, with the PREEMPT_RT patch and the fully preemptible kernel enabled. I was able to compile the end_to_end_latency branch daemon and driver against this. The platform test is running fine. However, the latency test yields an error about setting Round Robin, and I'm not sure the results are valid. Is there something I can do to troubleshoot this?
|
|
Not sure about the result. I would try to test with the real setup and check with the oscilloscope. |
|
Pretty sure it's flawed, perhaps due to the problem setting the Round Robin scheduler? |
|
No idea at the moment. The fact that you see the End to end latency print it means some samples were captured but I never had a 0 ms as result. |
|
solved the Round Robin error -- I ran the test script using sudo. Still getting 0 ms latency, though, so something isn't quite right. I'll try a different method. |
|
I did a few changes to the latency application. |
|
BTW the patch to the driver is working since I see in your logs that the RAVENNA buffer size is set to 128 frame: |
|
Had a quick run of the new latency test, results below:
|
|
ok, it looks good then. I see the end to end latency you have is similar to mine now. |
|
With the new latency test, I no longer need to run via sudo. Running on the stock Raspberry Pi OS kernel for now. I got the essentially the same results as steeley -- end-to-end latency was 7.997 msecs -- but just like in his results, I get a line stating "Failure" after "Trying latency 128 frames..." Also, both Playback and Capture show a state of XRUN after around 45000 frames. Repeating the test up to -M 1024 -m 1024, I still get failures at the same point. The end to end latency reads 63.993 secs What kind of ARM boards are you guys using? I'm on a 4GB RPi4B -- perhaps the board Andrea is using has much better latency? -N. |
|
I'm using a RPI cm4 compute module running at 1.8Ghz. My audio test now seems instant, and looks like latency is well below 10mS .When I hit 'play' the sound replays Play: Listen1: shows lots of pitch messages, then a small glitch and underrun after a few minutes like this: It seems these new pitch messages occur even when I'm not sending any audio. Play2, using gstreamer as player, clocking from input source and resampling to output: I think the resampling keeps the sync/drift issue to. a minimum, but ideally I'd rather not use it. It would be good if it was possible to make the master AES67 device also the PTP clock master. |
|
Steeley -- If you run ptp4l on the master AES67 device on both the ethernet and loopback interfaces, the daemon can lock to it. I had this problem, too, before. |
I am using a Nano Pi Neo 2 running Ubuntu 20.04.1 LTS |
You can try to use option -t 4000 when running alsaloop to reduce the latency. |
Interesting. You're running arm64...I wonder if running the standard Raspberry Pi OS 32-bit is hurting mine and Steeley's performance. Note the substantial improvement in latency sysbench cpu tests using a 64-bit system: https://forums.raspberrypi.com/viewtopic.php?t=243567 I know it's not exactly the same thing, but the difference is quite dramatic. Will test this a bit and get back to you. |
This is confirmed my side too. |
|
Working on this. Built a 64-bit image, as identically as possible to the 32-bit image I was using. Same hardware. sysbench shows cpu latency less than 1/20th of the 32-bit platform. The Daemon platform test runs well, tested up to 4 channels for 5 min at 24bit/48kHz. The latency test only runs at 2 channels of 24/48 if I set the buffer to >=3072 frames, and I get the following errors no matter what the testing parameters are:
Full report below.
|
|
This execution is OK You can ignore the errors. |
Interesting - I've used ptp4l before for a few things - didn't notice it can handle 2 interfaces at the same time. thanks |
|
I believe that you can use |
|
are ok that makes sense, thanks |
|
changes to the Ravenna driver plus latency tests and documentation merged to main branches. |
Have a system playing from a Raspberry PI CM4 with cm4 I/o board for ethernet. AesS67 daemon 1.31.
I ran the auto test routine on this AES67 cm4 and it passed ok.
Using a second PI as PTP clock via new d-link gigabit switch. Using MacBook Pro (Big Sur) with Merging/Ravenna Virtual audio device. Everything is locked together and speaker test works, and I can play audio files and RX the streams on the MacBook without clicks or dropouts.
However, the latency is about 1.2 sec!
SDP file from the AES57 sink/playout seems to work:
`v=0
o=- 2831165442 2831165444 IN IP4 192.168.2.40
s=AES67 daemon a8c02802 ALSA Source 0
c=IN IP4 239.255.0.10/15
t=0 0
a=clock-domain:PTPv2 0
m=audio 5004 RTP/AVP 98
c=IN IP4 239.255.0.10/15
a=rtpmap:98 L24/48000/2
a=sync-time:0
a=framecount:48
a=ptime:1
a=mediaclk:direct=0
a=ts-refclk:ptp=IEEE1588-2008:E4-5F-01-FF-FE-23-C2-20:0
a=recvonly
`
I tried not using Merging/Ravenna VAD and tried some command line FFMPEG and gstreamer stuff which works fine
but I always get circa 1.2 sec playback delay.
I also tried using a gstreamer command on the CM4 that is the player and sending to both the local audio output and plughw:RAVENNA at the same time. This works but the latency here is 1.2s, so maybe thats the problem?
Thanks.
The text was updated successfully, but these errors were encountered: