Slow download speed when streaming through the server

[jeffli678]

New issue

Your environment

Please answer these questions when reporting a new issue:

What is your operating system (Windows, Linux, OSX, etc.)? Ubuntu 16.04 Server

What is your web server (Apache, IIS, etc.)? Nginx

What version of AllTube are you using? 1.2.5 (Latest as of today)

How did you install AllTube (with Git or with a release package)? release package

What version of PHP are you using? PHP 7.0.33-0ubuntu0.16.04.3 (cli) ( NTS )

What version of Python are you using? Python 2.7.12 (default, Nov 12 2018, 14:36:49)

What version of youtube-dl are you using? 2019.03.18

Do you get any PHP-related errors in your webserver's logs? No.

What is the content of your config/config.yml file?

---

# Path to your youtube-dl binary
youtubedl: /usr/local/bin/youtube-dl

# Path to your python binary
python: /usr/bin/python

# An array of parameters to pass to youtube-dl
params:
- --no-warnings
- --ignore-errors
- --flat-playlist
- --restrict-filenames
- --no-playlist

# True to enable audio conversion
convert: false

# True to enable advanced conversion mode
convertAdvanced: false

# List of formats available in advanced conversion mode
convertAdvancedFormats: [mp3, avi, flv, wav]

# Path to your avconv or ffmpeg binary
avconv: /usr/bin/ffmpeg

# avconv/ffmpeg logging level.
avconvVerbosity: error

# Path to the directory that contains the phantomjs binary.
phantomjsDir: vendor/bin/

# True to disable URL rewriting
uglyUrls: false

# True to stream videos through server
stream: true

# True to enable remux mode (merge best audio and best video)
remux: true

# MP3 bitrate when converting (in kbit/s)
audioBitrate: 128

# App name
appName: AllTube Download

Please provide the URL of a video that causes the issue.

https://www.youtube.com/watch?v=gXkSF0RWauA

Describe your issue

I enabled the "stream through the server" option and it works. I can get the video correctly. If I download the "default" one (for youtube, it is called "Best(mp4)"), the speed is very fast, ranging from 5MB-15MB/s. However, when I try to download anything other than the default one, the download speed gets as slow as 500KB/s. Let me give more details.

I deployed the alltube on my vps. I tried to run youtube-dl from it directly and the download speed is 40-50MB/s. My browser's connection to my vps is also very fast, so I think the ~10MB/s download speed is expected.

Now, when I download anything other than the default one, the download speed is 1-2MB/s in the beginning, but quickly drops to 500KB/s until it finishes. The CPU and memory consumption is both low, so I think it is not a performance issue.

Note this also has nothing to do with the remux option. I experienced slow download speed on those formats in "detailed formats", not the remux version. Nevertheless, I found the download speed for the remux version is also 500KB/s. So I think remux is OK and it is not related to my problem.

I suggest you check the code and find what is different when downloading the default one and other formats.

If you have some difficulties pinpointing the problem, I can let you access my server from the public Web and you can at least reproduce the problem exactly. Though I would not post any information about the server in this issue publicly. Please contact me privately if necessary.

[jeffli678]

To be more specific, take the format code 135 of the video at https://www.youtube.com/watch?v=gXkSF0RWauA. If I directly download it from my vps with youtube-dl, it finishes in one second. If I download through alltube, its speed is roughly 500KB/s.

[jeffli678]

I also used Wireshark to inspect the traffic between my browser and the alltube. All seems very good. All green, almost no re-transmission or other alerting things.

[jeffli678]

One last word: I tried to download through alltube from my vps itself, i.e., wget https://my-server-ip/redirect?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DgXkSF0RWauA&format=135.

The download speed is roughly 1MB/s, which is significantly slower than the ~50MB/s download speed if I simply run youtube-dl on my vps.

I also find if I download using wget or curl, then regardless of whether I download the best format, it is slow.

[Rudloff]

Hello,
Thanks for the detailed information. I can reproduce the issue locally but I am not sure what causes it (AFAIK the code does the exact same thing for every format).

PS: You can edit your posts if you want to add new information without spamming the issue.

[Rudloff]

OK, I found the explanation.
YouTube now slows down requests that try to download the whole video in one request (but only for some formats apparently?). To work around that, youtube-dl actually downloads the video in several smaller chunks: ytdl-org/youtube-dl@e4a6091
I am not sure Alltube can do the same thing, but I will think about it.

[jeffli678]

You are right! The server throttles the download. The youtube-dl requests the video chunk-by-chunk using the Range header. This should not be very difficult to implement. The max chunk size for youtube is 10MB. Although it seems to me that Dailymotion imposes a smaller allowed chunk size.

So if you want a quick fix, just put the 10MB chunk size in the config file and most likely things will be fine. Otherwise, you should allow the user to select a chunk size from the web interface.

[Rudloff]

I found a solution; it will be in the next release 🙂