oneBehindFetch does not always fetch

[ithinkihaveacat]

This is quite bizarre. For large files, oneBehindFetch() does not return any response:

const fetch = require("@ampproject/toolbox-core").oneBehindFetch;

fetch("https://amp.dev/manifest.json")
  .then((res) => res.text())
  .then((text) => console.log(`length of manifest.json = ${text.length}`));

fetch("https://amp.dev/")
  .then((res) => res.text())
  .then((text) => console.log(`length of index.html = ${text.length}`));

On node 12.16.1, this outputs:

length of manifest.json = 1136

(That is, the length of index.html is not output.) Removing and reordering requests, etc. does not help.

The problem here seems to be that Response.clone() (as indirectly used by oneBehindFetch) is essentially broken for rather complicated reasons).

The recommended way of fixing this is to resolve the cloned responses in parallel, however this is not really suitable for oneBehindFetch's use case, since it endeavours to be a drop-in replacement for fetch().

This is related to #712 and #378. It may also be a factor in #650, since one of the consequences of this is that responses are not cached as expected. (Although that involved a small file, so maybe not.)

[sebastianbenz]

Thanks for investigating. I think we need to completely re-think how we want to perform caching. Currently, I'm leaning towards an explicit file-system based caching interface.