bug: streaming NDJSON does not work at https://delegated-ipfs.dev/

[SgtPooki]

Summary

I was looking into ipfs/service-worker-gateway#391 and ran into some issues with delegated-ipfs.dev queries.

I think passing accept: application/x-ndjson with filters breaks something in someguy. I was able to reproduce this without filters as well, but not as consistently.

Details

When I call https://delegated-ipfs.dev/routing/v1/providers/bafybeif3ie6d3yf2orkshi6xahlmwqudxst6db7mefkw6rcwanxwsksqou?filter-addrs=https%2Cwss&filter-protocols=unknown%2Ctransport-bitswap%2Ctransport-ipfs-gateway-http directly, I seem to get a response pretty quick. But when I call it from a locally running sw-gateway, it always times out. I can reproduce this with a curl call:

# No accept header
> time timeout 30s curl 'https://delegated-ipfs.dev/routing/v1/providers/bafybeif3ie6d3yf2orkshi6xahlmwqudxst6db7mefkw6rcwanxwsksqou?filter-addrs=https%2Cwss&filter-protocols=unknown%2Ctransport-bitswap%2Ctransport-ipfs-gateway-http' || echo "Failed to fetch data"
{"Providers":[{"Addrs":["/dns4/bitswap.filebase.io/tcp/443/wss"],"ID":"12D3KooWGtYkBAaqJMJEmywMxaCiNP7LCEFUAFiLEBASe232c2VH","Protocols":["transport-bitswap"],"Schema":"peer","transport-bitswap":"gBI="}]}timeout 30s curl   0.01s user 0.01s system 15% cpu 0.086 total

# accept header
> time timeout 30s curl -H 'accept: application/x-ndjson' 'https://delegated-ipfs.dev/routing/v1/providers/bafybeif3ie6d3yf2orkshi6xahlmwqudxst6db7mefkw6rcwanxwsksqou?filter-addrs=https%2Cwss&filter-protocols=unknown%2Ctransport-bitswap%2Ctransport-ipfs-gateway-http' || echo "Failed to fetch data"
timeout 30s curl -H 'accept: application/x-ndjson'   0.01s user 0.01s system 0% cpu 30.015 total
Failed to fetch data


# accept header without filters
> time timeout 30s curl -H 'accept: application/x-ndjson' 'https://delegated-ipfs.dev/routing/v1/providers/bafybeif3ie6d3yf2orkshi6xahlmwqudxst6db7mefkw6rcwanxwsksqou' || echo "Failed to fetch data"

# ... lots of providers

[lidel]

This is not a bug in someguy. If you run the same test against local someguy streaming works fine locally.

You can compare:

• curl "http://127.0.0.1:8190/routing/v1/providers/bafyreicpca3o3lcyhusmfhmq34h44l26vvekqwdw5npwo3jplaeh3uimpa" -H "Accept: application/x-ndjson"
• curl https://delegated-ipfs.dev/routing/v1/providers/bafyreicpca3o3lcyhusmfhmq34h44l26vvekqwdw5npwo3jplaeh3uimpa -H "Accept: application/x-ndjson"

IMO this is Nginx from https://github.com/ipshipyard/waterworks-infra or Cloudflare breaking streaming.
(The fact that you added ?filter to URL makes unique HTTP cache key, so you hit Nginx, and not get fast response from Cloudflare). Let's discuss internally in #waterworks.

[gammazero]

This was an issue with nginx and Cloudflair setup specific to delegated-ipfs-dev.