[BUG] Reopen #10150 testProject.testMatch |Array<string>| not respecting the order of file names in the array during execution

[vsravuri]

Context:

• Playwright Version: [1.16.3]
• Operating System: [Mac]
• Node.js version: [16.12.0]
• Browser: [Chrome]
• Extra: [[BUG] testProject.testMatch |Array<string>| not respecting the order of file names in the array during execution #10150]

This bug is follow up of #10150

I ran into issues while doing further testing with the suggested approach.

Code Snippet

Issue #1

If i have same title in multiple spec files, i get warning message, although i have different spec files, i see this warning when i run the test with suggested approach.
` duplicate test titles are not allowed.

title: pw1_test test
    test/pw1_test_spec.js:6
    test/pw2_test_spec.js:6
    `

Issue #2

storageState in test.use is not respected, if i have different storage states in each spec file, it doesn't load the storageState when the test runs

playwright.config

// playwright.config.js
// @ts-check

/** @type {import('@playwright/test').PlaywrightTestConfig} */
   const config = {
        workers: 1,
        globalTimeout: 0,
        globalSetup: './global-setup.js',
        reporter: [
            ['list']
        ],
        reportSlowTests: null,
        projects: [
            {
            name: 'Test1',
            testDir: './test',
            testMatch: 'sequence1.spec.js',
            retries: 0,
            use: {
                browserName: 'chromium',
                channel: 'chrome',
                headless: false,
                viewport: {width: 1600, height: 825},
                ignoreHTTPSErrors: true,
                trace: "off",
                screenshot: "only-on-failure",
                acceptDownloads:true,
            },
        },
        ]

    };
    module.exports = config;

global-setup.js

const { chromium, Browser, BrowserContext } = require('@playwright/test');

module.exports = async () => {

        const browser = await chromium.launch({
            headless: false
        });
        const context = await browser.newContext();

        // Open new page
        const page = await context.newPage();
        await page.context().storageState({path: 'empty_storage.json'});

        // Go to https://github.com/
        await page.goto('https://github.com/');

        // Click text=Sign in
        await page.click('text=Sign in');
        // assert.equal(page.url(), 'https://github.com/login');

        // Fill input[name="login"]
        await page.fill('input[name="login"]', '');

        // Fill input[name="password"]
        await page.fill('input[name="password"]', '');

        // Click input:has-text("Sign in")
        await page.click('input:has-text("Sign in")');
        // assert.equal(page.url(), 'https://github.com/');

        await page.context().storageState({path: 'login_storage.json'});

        // ---------------------
        await context.close();
        await browser.close();

}

sequence1.spec.js

require('./pw1_test_spec.js');
require('./pw2_test_spec.js');

pw1_test_spec.js

const {test,expect} = require("@playwright/test");

test.use({storageState: 'login_storage.json'});
test.describe('pw1_test', () => {

    test('test', async ({ page }) => {

        // Go to https://github.com/
        await page.goto('https://github.com/');

        // Click text=Pull requests
        await page.click('text=Pull requests');
        await expect(page).toHaveURL('https://github.com/pulls');

    });
});

pw2_test_spec.js

const {test,expect} = require("@playwright/test");

test.use({storageState: 'empty_storage.json'});
test.describe('pw2_test', () => {

    test('test', async ({ page }) => {

        // Go to https://github.com/
        await page.goto('https://gmail.com/');

    });
});

Describe the bug

• Provide login credentials to GitHub in global-setup.js
• global-setup will create two storageState files, one for GitHub and other for empty storageState
• When pw1_test_spec.js is executed, it will redirect to login page instead of using the storageState
• If i exclude pw2_test_spec.js from the sequence file, i won't be redirected to login page

Kindly let me know if you need any other information.

[vsravuri]

@aslushnikov

Appreciate if you could have a look at this.

Thanks

[yury-s]

i see this warning when i run the test with suggested approach.
` duplicate test titles are not allowed.

I don't see such warning when running npx playwright test with 1.16.3, how do you run your tests?

• Provide login credentials to GitHub in global-setup.js
• global-setup will create two storageState files, one for GitHub and other for empty storageState
• When pw1_test_spec.js is executed, it will redirect to login page instead of using the storageState
  It may
• If i exclude pw2_test_spec.js from the sequence file, i won't be redirected to login page

Both tests use the most recently configured storage state file, which in this case is empty_storage.json, so it's empty in both tests.

[vsravuri]

@yury-s

Please find the details of actual issue reported in #10150 and side affects of using the suggestion provided in #10150

Config and spec files.

global-setup.js

const { chromium, Browser, BrowserContext } = require('@playwright/test');

module.exports = async () => {
        const browser = await chromium.launch({
            headless: true
        });
        const context = await browser.newContext();
        const page = await context.newPage();
        await page.context().storageState({path: 'empty_storage.json'});
        await page.goto('https://github.com/');
        await page.click('text=Sign in');
        await page.fill('input[name="login"]', '<place_holder>');
        await page.fill('input[name="password"]', '<place_holder>');
        await page.click('input:has-text("Sign in")');
        await page.context().storageState({path: 'login_storage.json'});
        await context.close();
        await browser.close();
}

pw-config-demo.js

// playwright.config.js
// @ts-check

/** @type {import('@playwright/test').PlaywrightTestConfig} */
const config = {
    workers: 1,
    globalSetup: './global-setup.js',
    reporter: [
        ['list']
    ],
    projects: [
        {
            name: 'Test1',
            testDir: './test/bug',
            testMatch: '*_spec.js',
            retries: 0,
            use: {
                browserName: 'chromium',
                channel: 'chrome',
                headless: false,
                viewport: {width: 1600, height: 825},
                ignoreHTTPSErrors: true,
                trace: "off",
                screenshot: "only-on-failure",
            },
        },
    ]
};
module.exports = config;

pw1_test_spec.js

const {test,expect} = require("@playwright/test");
test.use({storageState: 'login_storage.json'});
test.describe('pw1_test', () => {
    test('test', async ({ page }) => {
        await page.goto('https://github.com/');
        await page.click('text=Pull requests');
        await expect(page).toHaveURL('https://github.com/pulls');
    });
});

pw2_test_spec.js

const {test,expect} = require("@playwright/test");
test.use({storageState: 'empty_storage.json'});
test.describe('pw1_test', () => {
    test('test', async ({ page }) => {
        await page.goto('https://gmail.com/');
    });
});

pw10_test_spec.js

const {test,expect} = require("@playwright/test");
test.use({storageState: 'empty_storage.json'});
test.describe('pw1_test', () => {
    test('test', async ({ page }) => {
        await page.goto('https://gmail.com/');
    });
});

I have three spec files in a folder as displayed here

I expect the spec files to be executed in order (with worker=1)
/pw1_test_spec.js
/pw2_test_spec.js
/pw10_test_spec.js

Actual, Spec files are executed in the order pw1, pw10 & pw2

Running 3 tests using 1 worker

As suggested in #10150, i made the following changes.
Created a new file sequence1.spec.js

require('./pw1_test_spec.js');
require('./pw2_test_spec.js');
require('./pw10_test_spec.js');

Modified the config file to use the sequence1.spec.js

// playwright.config.js
// @ts-check

/** @type {import('@playwright/test').PlaywrightTestConfig} */
const config = {
    workers: 1,
    globalSetup: './global-setup.js',
    reporter: [
        ['list']
    ],
    projects: [
        {
            name: 'Test1',
            testDir: './test/bug',
            testMatch: 'sequence1.spec.js',
            retries: 0,
            use: {
                browserName: 'chromium',
                channel: 'chrome',
                headless: false,
                viewport: {width: 1600, height: 825},
                ignoreHTTPSErrors: true,
                trace: "off",
                screenshot: "only-on-failure",
            },
        },
    ]
};
module.exports = config;

When i run the test using this command, i get error message
PLAYWRIGHT_BROWSERS_PATH=$HOME/pw-browsers playwright test --config=./pw-config-demo.js --workers 1

Using config at /Users/ravuri/Documents/kb-auto-pw-final/pw-config-demo.js
=================
 duplicate test titles are not allowed.
 - title: pw1_test test
   - test/bug/pw1_test_spec.js:4
   - test/bug/pw2_test_spec.js:4
   - test/bug/pw10_test_spec.js:4
=================

To overcome the duplicate test titles, i only modified the test titles in each spec file.

pw1_test_spec.js

const {test,expect} = require("@playwright/test");
test.use({storageState: 'login_storage.json'});
test.describe('pw1_test', () => {
    test('test1', async ({ page }) => {
        await page.goto('https://github.com/');
        await page.click('text=Pull requests');
        await expect(page).toHaveURL('https://github.com/pulls');
    });
});

pw2_test_spec.js

const {test,expect} = require("@playwright/test");
test.use({storageState: 'empty_storage.json'});
test.describe('pw1_test', () => {
    test('test2', async ({ page }) => {
        await page.goto('https://gmail.com/');
    });
});

pw10_test_spec.js

const {test,expect} = require("@playwright/test");
test.use({storageState: 'empty_storage.json'});
test.describe('pw1_test', () => {
    test('test3', async ({ page }) => {
        await page.goto('https://gmail.com/');
    });
});

When i run the test, i don't see duplicate titles error message but spec pw1_test_spec.js fails as the test is stuck at login page, it seems login_storage.json is not loaded.

Here is the output

> PLAYWRIGHT_BROWSERS_PATH=$HOME/pw-browsers playwright test --config=./pw-config-demo.js  --workers 1

Using config at /Users/ravuri/Documents/kb-auto-pw-final/pw-config-demo.js

Running 3 tests using 1 worker

  ✘  [Test1] › test/bug/pw1_test_spec.js:4:5 › pw1_test › test1 (13s)
  ✓  [Test1] › test/bug/pw2_test_spec.js:4:5 › pw1_test › test2 (3s)
  ✓  [Test1] › test/bug/pw10_test_spec.js:4:5 › pw1_test › test3 (2s)


  1) [Test1] › test/bug/pw1_test_spec.js:4:5 › pw1_test › test1 ====================================

    Error: expect(received).toHaveURL(expected)

    Expected string: "https://github.com/pulls"
    Received string: "https://github.com/"

    Call log:
      - waiting for selector ":root"
      -   selector resolved to <html lang="en">…</html>
      -   unexpected value "https://github.com/"
      -   selector resolved to <html lang="en">…</html>
      -   unexpected value "https://github.com/"
      -   selector resolved to <html lang="en">…</html>
      -   unexpected value "https://github.com/"
      -   selector resolved to <html lang="en">…</html>
      -   unexpected value "https://github.com/"
      -   selector resolved to <html lang="en">…</html>
      -   unexpected value "https://github.com/"
      -   selector resolved to <html lang="en">…</html>
      -   unexpected value "https://github.com/"
      -   selector resolved to <html lang="en">…</html>
      -   unexpected value "https://github.com/"


       5 |         await page.goto('https://github.com/');
       6 |         await page.click('text=Pull requests');
    >  7 |         await expect(page).toHaveURL('https://github.com/pulls');
         |                            ^
       8 |     });
       9 | });
      10 |

        at /Users/ravuri/Documents/kb-auto-pw-final/test/bug/pw1_test_spec.js:7:28
        at WorkerRunner._runTestWithBeforeHooks (/Users/ravuri/Documents/kb-auto-pw-final/node_modules/@playwright/test/lib/workerRunner.js:478:7)

    attachment #1: screenshot (image/png) ----------------------------------------------------------
    test-results/sequence1-pw1-test-test1-Test1/test-failed-1.png
    ------------------------------------------------------------------------------------------------


  1 failed
    [Test1] › test/bug/pw1_test_spec.js:4:5 › pw1_test › test1 =====================================
  2 passed (19s)

Please let me know if you need more information.

[dgozman]

@vsravuri Currently, Playwright Test executes files in alphabetical order (when it uses just a single worker). Overall, the order of execution is not guaranteed. However, if you'd name your files pw01_test_spec.js, pw02_test_spec.js, ..., pw10_test_spec.js, it will most likely work today.

I'll see what we can do.

[dgozman]

I've added a section to the docs. I don't think we'll do any changes in the test runner here, so closing the issue.