Old performance issues, new plan of action ? #22
Comments
|
Performance on the latest Matrix 19.4 is still quite poor, I'm barely breaking 30MB/s on a 1Gbps capable network. On the same system using SSHFS-Win the speeds reach 90MB/s. |
|
Tagging @AlwinEsch, not sure if there is any low hanging fruit in terms of optimization? I'm on Windows 10. |
|
This is dog slow on a new firestick, and two other amlogic boxes. Running sshfs independently worked around the bug on those other boxes, but this is completely unusable on Android. |
|
I can confirm this issue. On a 300mbps capable network, sftp transfer within kodi reaches 5-8mbps at most, nit enough to play HD videos. However, if I mount a folder using sshfs externally, the performance is more than 10 times faster. Sadly, on Android devices like a firetv stick, there's no option to externally add a sftp mountpoint. |
|
Yeah, the sftp performance is pretty poor :( |
|
Same issue on 20.2 using the VFS plugin. Not able to achieve anything over 20 megabits/s. Vanilla SSHFS using Debian on the same hardware target/ yields over 300 Megabits. |
|
There's no prefetching which is the problem. They're not using the async API, and the video buffering code is just terrible. so; latency + low throughput = bad news. It's very stupid and silly, but if you stream over http it'll improve the situation. The devs on a Shield use the native SMB functionality to the OS to work around the bug. |
Why is the video buffering code "terrible" ? There's no argument to back that statement up (and I think it's not true either). It seems to work pretty well with other protocols, right? |
Agreed here, does not seem to be effecting other protocols. UPNP, SMB, work seamlessly. HTTPS is an option, but when you're streaming over the internet, SSH is just such an easy and versatile protocol you can do so many things over a single port. Wonder what the effort would be to fix this. |
Very small, just change the call-sites to async_read. Directory prefetching and caching will help with library import.
xbmc/xbmc#22799 Not really sure where the breakdown here is. |
|
I face the same issue, limiting download speeds (max. 30 Mbit/s) for links with ~20ms round trip time so I dug into it. IDK if Kodi tries to use multiple threads, but this add-on uses a mutex on all reads, preventing such optimizations. Libssh says you shouldn't share sessions for thread safety, so this synchronization might be required. PR #114 tried to address this problem. Besides this point I have two optimization ideas: 1. Increase the chunk size by implementing |
To improve the performance. The current implementation severely underperforms, reaching at maximum 60 Mbit/s on a local 1 Gbit/s link (where the cmd tool `scp` performs at over 800 Mbit/s). With video bit rates reaching 60 Mbit/s, videos can no longer be streamed without occasional interruptions. This problem severely worsens if the network latency is higher. One situation with about 40 ms RTT latency only reached about 10 Mbit/s, even though the network could handle more throughput. See xbmc#22 for more discussion. This work always initiates the next read at the end of each previous one. This however requires to specify a length, which might not match with the length Kodi requests next (e.g. at the end of file). To avoid memory leaks, we have to call `sftp_async_read` with a valid buffer of the correct size. This means a temporary buffer is needed. If too much was actually read, this work simply rewinds the current file offset to the Kodi-expected place. This might also mess with other/threaded usage, as the async work can run concurrently with normally mutexed areas, and even is expected to change the file position during retrieval, leading to inconsistent results when calling `GetPosition` at the same time.
To improve the performance. The current implementation severely underperforms, reaching at maximum 60 Mbit/s on a local 1 Gbit/s link (where the cmd tool `scp` performs at over 800 Mbit/s). With video bit rates reaching 60 Mbit/s, videos can no longer be streamed without occasional interruptions. This problem severely worsens if the network latency is higher. One situation with about 40 ms RTT latency only reached about 10 Mbit/s, even though the network could handle more throughput. See xbmc#22 for more discussion. This work always initiates the next read at the end of each previous one. This however requires to specify a length, which might not match with the length Kodi requests next (e.g. at the end of file). To avoid memory leaks, we have to call `sftp_async_read` with a valid buffer of the correct size. This means a temporary buffer is needed. If too much was actually read, this work simply rewinds the current file offset to the Kodi-expected place. This might also mess with other/threaded usage, as the async work can run concurrently with normally mutexed areas, and even is expected to change the file position during retrieval, leading to inconsistent results when calling `GetPosition` at the same time.
To improve the performance. The current implementation severely underperforms, reaching at maximum 60 Mbit/s on a local 1 Gbit/s link (where the cmd tool `scp` performs at over 800 Mbit/s). With video bit rates reaching 60 Mbit/s, videos can no longer be streamed without occasional interruptions. This problem severely worsens if the network latency is higher. One situation with about 40 ms RTT latency only reached about 10 Mbit/s, even though the network could handle more throughput. See xbmc#22 for more discussion. This work always initiates the next read at the end of each previous one. This however requires to specify a length, which might not match with the length Kodi requests next (e.g. at the end of file). To avoid memory leaks, we have to call `sftp_async_read` with a valid buffer of the correct size. This means a temporary buffer is needed. If too much was actually read, this work simply rewinds the current file offset to the Kodi-expected place. This might also mess with other/threaded usage, as the async work can run concurrently with normally mutexed areas, and even is expected to change the file position during retrieval, leading to inconsistent results when calling `GetPosition` at the same time.
To improve the performance. The current implementation severely underperforms, reaching at maximum 60 Mbit/s on a local 1 Gbit/s link (where the cmd tool `scp` performs at over 800 Mbit/s). With video bit rates reaching 60 Mbit/s, videos can no longer be streamed without occasional interruptions. This problem severely worsens if the network latency is higher. One situation with about 40 ms RTT latency only reached about 10 Mbit/s, even though the network could handle more throughput. See xbmc#22 for more discussion. This work always initiates the next read at the end of each previous one. This however requires to specify a length, which might not match with the length Kodi requests next (e.g. at the end of file). To avoid memory leaks, we have to call `sftp_async_read` with a valid buffer of the correct size. This means a temporary buffer is needed. If too much was actually read, this work simply rewinds the current file offset to the Kodi-expected place. This might also mess with other/threaded usage, as the async work can run concurrently with normally mutexed areas, and even is expected to change the file position during retrieval, leading to inconsistent results when calling `GetPosition` at the same time.
Hi,
I've been using XBMC and Kodi for quite a long time and i'm sure im not the only one interested and affected by the old performances issues on SFTP remote sources.
As a personnal example, i frequently use a dedicated server that can push a little over 1Gbps on internet (filezilla, python simpleHTTPserver, nginx, rsync ...) to a client fiber that is also capable of receiving 1Gbps.
Also the ping -seen on the client side- between the client and the dedicated server is typically around 13-16ms.
And when im browsing a BIG SFTP folder through Kodi (17.6, not tested 18.0 until now) on my Kodi "dedicated laptop" (win10 pro, i5 520m,, 8GB RAM and 1Gbps RJ45 network), Windows Task Manager network graph does not go over 8Mbps !
I think im not alone with this kind of problems -https://trac.kodi.tv/ticket/14038- (very low performance/usage relative to real capacity of the network and computers involved), there were numerous topics about HPN-SSH implementation for example ...
Now that there is an addon (independent of main app source code), could it be thinkable to look into solution for those kind of inefficiencies ?
Or if the problems are already resolved, it would be a good -public- place to make it known loud and clear !
The text was updated successfully, but these errors were encountered: