[BUG] Some requests not intercepted without waitFor(2000)

[dialupdev]

Context:

• Playwright Version: 0.10.0
• Operating System: macOS
• Extra: Node 12.14.1

Code Snippet
Here's a minimal repro of the behavior I'm seeing.

const pw = require('playwright');

(async () => {
  const browser = await pw.chromium.launch({ headless: false });
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.setRequestInterception(true);

  const pattern = /^https:\/\/www\.behance\.net/;

  page.on('request', request => {
    if (request.url().match(pattern)) {
      console.log('request', request.url());
    }

    request.continue();
  });

  page.on('requestfailed', (request) => {
    if (request.url().match(pattern)) {
      console.log('requestfailed', request.url());
    }
  });

  page.on('requestfinished', (request) => {
    if (request.url().match(pattern)) {
      console.log('requestfinished', request.url());
    }
  });

  await page.goto('https://www.behance.net/search');

  // await page.waitFor(1000);
  await page.waitFor(2000);

  await page.click('[data-ut="moodboards-tab"]');
  // ^ this click fires the XHR
  await page.waitFor(5000);
  
  await browser.close();
})();

Describe the bug

Hey there. Thanks for building this great tool!
I'm having an issue with request interception not working all the time. If I don't wait for 2s or more after loading my first page, I can't see any XHRs that that page makes. However, if I do wait for 2s or more, I can see the XHR.

Here's the output the script generates:

# With 2s wait
request https://www.behance.net/search
requestfinished https://www.behance.net/search
request https://www.behance.net/search
requestfinished https://www.behance.net/search
request https://www.behance.net/assets/img/footer/icon-ad-choices.png
requestfinished https://www.behance.net/assets/img/footer/icon-ad-choices.png
request https://www.behance.net/v2/an/pv
requestfinished https://www.behance.net/v2/an/pv
request https://www.behance.net/v2/an/pv
request https://www.behance.net/search?content=moodboards
# ^ this is the XHR I care about
requestfinished https://www.behance.net/v2/an/pv
requestfinished https://www.behance.net/search?content=moodboards

# Without 2s wait
request https://www.behance.net/search
requestfinished https://www.behance.net/search
request https://www.behance.net/search
requestfinished https://www.behance.net/search
request https://www.behance.net/assets/img/footer/icon-ad-choices.png
requestfinished https://www.behance.net/assets/img/footer/icon-ad-choices.png
request https://www.behance.net/v2/an/pv
request https://www.behance.net/search/moodboards
requestfinished https://www.behance.net/v2/an/pv
request https://www.behance.net/assets/img/footer/icon-ad-choices.png
requestfinished https://www.behance.net/search/moodboards
requestfinished https://www.behance.net/assets/img/footer/icon-ad-choices.png
request https://www.behance.net/v2/an/pv
requestfinished https://www.behance.net/v2/an/pv

[pavelfeldman]

I'm using 0.11, so I've rewritten your script using page.route.

const { chromium, webkit, firefox } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.route(/^https:\/\/www\.behance\.net/, request => {
    console.log('request', request.url());
    request.continue();
  });

  await page.goto('https://www.behance.net/search');

  // If looks like page-side router is installed in JavaScript lazily.
  // - If you don't wait, click produces the navigation to `/search/moodboard` that
  //   contains all the data required to render the page.
  // - if you do wait, `history.pushState` is used to navigate and `XHR` is issued
  //   to fetch the data.
  // You either need to wait for the router to be installed or to intercept both XHR
  // and `/search/moodboard`.
  // 
  // Btw, it does XHR for me on Chromium, bug WebKit and Firefox do the
  // navigation if I don't wait here. I believe that is a timing issue.

  await Promise.all([
    page.waitForRequest(/search\?content\=moodboards/),
    page.click('[data-ut="moodboards-tab"]'),
  ]);
  await browser.close();
})();

[dialupdev]

Thanks very much @pavelfeldman. This explanation is super helpful. In my case I was able to make it work simply by doing this:

await Promise.all([
  page.click('[data-ut="moodboards-tab"]'),
  page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);

I'm not sure this is the very best way, since JS bindings may be ready well before all network requests are complete, but this does the job.