Infinite scroll partly broken

[bastimeyer]

Looks like the infinite scroll mechanism is currently broken due to another bug on the Twitch API.

I've already mentioned it yesterday in #706 that Twitch has just made some changes to streams related endpoints on the kraken API without properly notifying 3rd party app developers. They just posted a thread on the dev announcements forums (https://discuss.dev.twitch.tv/t/upcoming-changes-to-the-streams-v5-endpoints/23926) and gave app developers merely 20 days to apply changes/fixes to their apps before deploying the API changes. What they did was remove metadata from API responses that described how many overall entries there are for a certain query.

In my first response to #706, I said that this wouldn't affect Streamlink Twitch GUI because it already had a workaround implemented due to the metadata being inconsistent between certain API endpoints. This is true, but this also means that all API responses have to return expected results for each query being made, or else it won't work.

The infinite scroll mechanism works like this:
It first calculates how many records/items in a list of streams (or games, etc) are needed, so that the entire page of the current route can be filled with just one API query without querying for more records than needed. If the user then scrolls down and reaches a certain calculated threshold, it re-queries the API in order to get more fresh data records. It only does this though if it thinks that it hasn't reached the end of all available records yet. That's why it keeps track of the number of records already fetched from the API and the (until recently) included metadata or if the previous query included the same number of records (or more) than it has asked for. The logic is very simple: if there are fewer records in a response than queried for, it means that the end was reached and no more additional records are available.

Unfortunately though, there seems to be an issue on the streams API endpoint and the infinite scroll mechanism breaks after fetching records once.

offset=0, limit=12

$ curl -s \
  -H "Accept: application/vnd.twitchtv.v5+json" \
  -H "Client-ID: phiay4sq36lfv9zu7cbqwz2ndnesfd8" \
  "https://api.twitch.tv/kraken/streams?offset=0&limit=12" \
  | jq "[.streams[].channel.name] | length"
12

offset=12, limit=12

$ curl -s \
  -H "Accept: application/vnd.twitchtv.v5+json" \
  -H "Client-ID: phiay4sq36lfv9zu7cbqwz2ndnesfd8" \
  "https://api.twitch.tv/kraken/streams?offset=12&limit=12" \
  | jq "[.streams[].channel.name] | length"
11

As you can see, if the offset is greater than 0, the API response includes fewer records than requested. This breaks the infinite scroll's "hasFetchedAll" implementation.

A workaround for this could be fetching records from the API no matter what, until the first response includes zero records. This is bad however, because it means that the API is being queried once unnecessarily and it is especially bad because the helix API is rate-limited and kraken may be soon too (they currently backpedaled with that change though). It would also introduce a delay, which means that the "fetch more" button would be visible and then disappear while being in sight, and that's bad. What's also bad about this bug is that due to the response lacking one record, the route's list doesn't get filled properly. Sigh 😩