[yandexmusic:playlist] argument of type 'bool' is not iterable after calling _download_webpage_handle

[mvoidex]

Checklist

• I'm reporting a bug unrelated to a specific site
• I've verified that I'm running yt-dlp version 2022.07.18 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

While downloading large Yandex Music playlist I get error argument of type 'bool' is not iterable

The reason is that:

• InfoExtractor._download_webpage_handle can return False
• Derived class tests for some string within result:
  • Yandex Music affected: link
  • Also Xiaomi may be affected: link
  • And Adobe possibly too: link, unpacks result of _download_webpage_handle, which calls base version

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

yt-dlp --cookies ..\..\yandex.music.txt https://music.yandex.ru/users/VoidEx/playlists/1031 -vU
[debug] Command-line config: ['--cookies', '..\\..\\yandex.music.txt', 'https://music.yandex.ru/users/VoidEx/playlists/1031', '-vU']
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.07.18 [135f05e] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.07.18, Current version: 2022.07.18
yt-dlp is up to date (2022.07.18)
[debug] [yandexmusic:playlist] Extracting URL: https://music.yandex.ru/users/VoidEx/playlists/1031
[yandexmusic:playlist] 1031: Downloading playlist JSON
[yandexmusic:playlist] 1031: Downloading missing tracks JSON chunk 1
WARNING: [yandexmusic:playlist] Unable to download JSON metadata: HTTP Error 431: Request Header Fields Too Large
ERROR: argument of type 'bool' is not iterable
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1431, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1506, in __extract_info
  File "yt_dlp\extractor\common.py", line 644, in extract
  File "yt_dlp\extractor\yandexmusic.py", line 356, in _real_extract
  File "yt_dlp\extractor\yandexmusic.py", line 217, in _extract_tracks
  File "yt_dlp\extractor\yandexmusic.py", line 49, in _call_api
  File "yt_dlp\extractor\yandexmusic.py", line 44, in _download_json
  File "yt_dlp\extractor\common.py", line 1004, in download_content
  File "yt_dlp\extractor\common.py", line 968, in download_handle
  File "yt_dlp\extractor\yandexmusic.py", line 39, in _download_webpage_handle
TypeError: argument of type 'bool' is not iterable

[mvoidex]

However, problem that causing this issue is "HTTP Error 431: Request Header Fields Too Large", and i don't know how to workaround it :(
Fixing this issue can make just better error message

[pukkandan]

It is likely an issue either with cookies or with how yt-dlp is processing them. First try by re-exporting the cookies. If that doesn't help, I can take a look if you are willing to share login details with me

[pukkandan]

nvm, issue seems related to ytdl-org/youtube-dl#27355

[mvoidex]

nvm, issue seems related to ytdl-org/youtube-dl#27355

Yes, but as I can see, it's fixed there and also this commit merged here as long as there's message about chunks while reading playlist

[mvoidex]

It is likely an issue either with cookies or with how yt-dlp is processing them. First try by re-exporting the cookies. If that doesn't help, I can take a look if you are willing to share login details with me

I've tried reexporting cookies and it didn't help. I can share login details, if it helps, where I can do it?

[pukkandan]

Contact: discord (pukkandan#4207) / reddit (/u/pukkandan) / email ([email protected])

[mvoidex]

Let's try discord, I'm voidex#9676

[dirkf]

The issue is twofold:

• _call_api() has _download_json(..., fatal=False, ...), which can cause _download_webpage_handle() to return False instead of raising
• the specialisation of _download_webpage_handle() doesn't take account of that, but just obscures the error, and in any case doesn't actually understand what the normal return value should be (ie the original captcha check never worked).

     def _download_webpage_handle(self, *args, **kwargs):
         webpage = super()._download_webpage_handle(*args, **kwargs)
+        if not webpage:
+            return None
+        if 'Нам очень жаль, но запросы, поступившие с вашего IP-адреса, похожи на автоматические.' in webpage[0]:
-        if 'Нам очень жаль, но запросы, поступившие с вашего IP-адреса, похожи на автоматические.' in webpage:
            self._raise_captcha()
        return webpage

Edited: now even more similar to #4432 (comment)

[bashonly]

     def _download_webpage_handle(self, *args, **kwargs):
         webpage = super()._download_webpage_handle(*args, **kwargs)
+        if not webpage:
+            return None
+        webpage = webpage[0]
         if 'Нам очень жаль, но запросы, поступившие с вашего IP-адреса, похожи на автоматические.' in webpage:
            self._raise_captcha()
        return webpage

This would still be wrong, since the expected result from a successful _download_webpage_handle call is a tuple, (webpage, urlh)

It seems like the mistake is the decision to redefine _download_webpage_handle() instead of _download_webpage()

[dirkf]

The result is only a tuple if the webpage was fetched. When fatal is falsy, False can be returned if the request failed.

If it was only intended to affect webpage downloads rather than JSON as well, _download_webpage() would have been the right target and the method suite might have been correct.

[bashonly]

The result is only a tuple if the webpage was fetched. When fatal is falsy, False can be returned if the request failed.

That's what I was going for with "successful call." Your patch only returns the 1st character of the webpage (res[0]) if the fetching is successful

If it was only intended to affect webpage downloads rather than JSON as well, _download_webpage() would have been the right target and the method suite might have been correct.

True, making this _download_webpage() instead wouldn't work either. I guess we'd have to do this:

    def _download_webpage_handle(self, *args, **kwargs):
        res = super()._download_webpage_handle(*args, **kwargs)
        if res is False:
            return False
        if 'Нам очень жаль, но запросы, поступившие с вашего IP-адреса, похожи на автоматические.' in res[0]:
            self._raise_captcha()
        return res

[dirkf]

Not all true at the time

That's equivalent to the patch that I posted, given that res (called webpage in the original code) is guaranteed to be either False or a tuple whose head is the webpage. As they say, "ask me how I know".

It was perhaps a poor design decision to return False rather than returning (None, None).

Fancier:

class HandleResult(collections.namedtuple(['webpage', 'urlh'])):
    def __bool__(self):
        return self.urlh is not None

    __nonzero__ = __bool__  # Python 2.7

Then:

>>> x = HandleResult(None, None)
>>> not x
True
>>> bool(x)
False
>>> x or 42
42
>>> 42 and x
HandleResult(webpage=None, urlh=None)
>>> 

But that requires callers who are testing for a non-tuple to be using if res: rather than the pettier if res is not False:.

Not crashing on the patched code would reveal whether the underlying issue, which was made a warning by fatal=False, was recoverable:

WARNING: [yandexmusic:playlist] Unable to download JSON metadata: HTTP Error 431: Request Header Fields Too Large

[bashonly]

That's equivalent to the patch that I posted, given that res (called webpage in the original code) is guaranteed to be either False or a tuple whose head is the webpage. As they say, "ask me how I know".

No, it's not. Your patch only returns the 1st character of a webpage upon a successful request

[dirkf]

Oh sure, edited. Current yt-dlp would do that because the network rework now indexes the content after the "petty" test, while formerly and upstream the content is destructured, which would have been an error except with a 2-character webpage.