[BUG] testProject.testMatch |Array<string>| not respecting the order of file names in the array during execution

[vsravuri]

Context:

• Playwright Version: [1.16.3]
• Operating System: [Mac]
• Node.js version: [16.12.0]
• Browser: [All]
• Extra: [N/A]

Describe the bug
I have 3 spec files in test folder

Here is my config file

When i run the test with one worker, files are not executed in the order mentioned in testProject.testMatch Array

As per documentation, testMatch support Array of files
https://playwright.dev/docs/next/api/class-testproject#test-project-test-match

Actual:

pw1_test_spec.js
pw10_test_spec.js
pw2_test_spec.js

Expected:

pw1_test_spec.js
pw2_test_spec.js
pw10_test_spec.js

Noticed the same behavior if i use

testMatch: '*_test_spec.js'

Kindly let me know if i am doing something wrong here.

[aslushnikov]

@vsravuri The order is not guaranteed. Moreover, it's generally advised against relying on the test order.

Why would you like to have a specific order of your test files?

[vsravuri]

@aslushnikov

I would expect specific order if worker = 1.

Here is the reason..
My e2e test cases are built on business workflow.

• Create knowledge base
• Create articles
• Review articles
• Publish articles
• Categorize articles
• Create users
• Create portal
• Publish portal
• Users view the articles in portal

If the order is not maintained, whole e2e execution is broken.

In Cypress and other automation tools (Protractor), the order is maintained.

Using the cypress configuration we can execute cypress tests in a sequence we desire. Simply write test file names under the testFiles property in your cypress.json file, like below.

"testFiles": [
    "TC01.spec.js",
    "TC02_fixtures.spec.js",
    "TC03_writeReadFile.spec.js",
    "TC04_each.spec.js",
    "TC05_conditionalTesting.spec.js",
    ]

Please let me know if you need further clarification.

[aslushnikov]

Well, let it be, then! It'll come in the next release.

[vsravuri]

Thanks @aslushnikov

[aslushnikov]

@vsravuri More discussion happened about this.

Here's a nice approach for you to build any sequence:

1. create a new file sequence.spec.js
2. inside the sequence, require spec files in whatever order you'd like:

   require('./pw1_test_spec.js');
   require('./pw2_test_spec.js');
   require('./pw10_test_spec.js');

3. Run the sequence in your project config:

   module.exports = {
     projects: [{
       name: 'chrome',
       testMatch: 'sequence.spec.js',
       use: { browserName: 'chromium' },
     }],
   };

With this approach, you have unlimited configurability of your tests!
Let me know if this works for you.

[vsravuri]

@aslushnikov

I tested the approach, it works for me. Thanks for quick solution.

[aslushnikov]

Awesome, I'll close this then.

[vsravuri]

@aslushnikov @dgozman

Request to reopen this bug. I ran into issues while doing further testing with the suggested approach.

Issue #1

If i have same title in multiple spec files, i get warning message, although i have different spec files, i see this warning when i run the test with suggested approach.
` duplicate test titles are not allowed.

• title: pw1_test test
  • test/pw1_test_spec.js:6
  • test/pw2_test_spec.js:6
    `

Issue #2

storageState in test.use is not respected, if i have different storage states in each spec file, it doesn't load the storageState when the test runs

Here is the repro

1. Provide login credentials to GitHub in global-setup.js
2. global-setup will create two storageState files, one for GitHub and other for empty storageState
3. When pw1_test_spec.js is executed, it will redirect to login page instead of using the storageState
4. If i exclude pw2_test_spec.js from the sequence file, i won't be redirected to login page

playwright.config

// playwright.config.js
// @ts-check

/** @type {import('@playwright/test').PlaywrightTestConfig} */
   const config = {
        workers: 1,
        globalTimeout: 0,
        globalSetup: './global-setup.js',
        reporter: [
            ['list']
        ],
        reportSlowTests: null,
        projects: [
            {
            name: 'Test1',
            testDir: './test',
            testMatch: 'sequence1.spec.js',
            retries: 0,
            use: {
                browserName: 'chromium',
                channel: 'chrome',
                headless: false,
                viewport: {width: 1600, height: 825},
                ignoreHTTPSErrors: true,
                trace: "off",
                screenshot: "only-on-failure",
                acceptDownloads:true,
            },
        },
        ]

    };
    module.exports = config;

global-setup.js

const { chromium, Browser, BrowserContext } = require('@playwright/test');

module.exports = async () => {

        const browser = await chromium.launch({
            headless: false
        });
        const context = await browser.newContext();

        // Open new page
        const page = await context.newPage();
        await page.context().storageState({path: 'empty_storage.json'});

        // Go to https://github.com/
        await page.goto('https://github.com/');

        // Click text=Sign in
        await page.click('text=Sign in');
        // assert.equal(page.url(), 'https://github.com/login');

        // Fill input[name="login"]
        await page.fill('input[name="login"]', '');

        // Fill input[name="password"]
        await page.fill('input[name="password"]', '');

        // Click input:has-text("Sign in")
        await page.click('input:has-text("Sign in")');
        // assert.equal(page.url(), 'https://github.com/');

        await page.context().storageState({path: 'login_storage.json'});

        // ---------------------
        await context.close();
        await browser.close();

}

sequence1.spec.js

require('./pw1_test_spec.js');
require('./pw2_test_spec.js');

pw1_test_spec.js

const {test,expect} = require("@playwright/test");

test.use({storageState: 'login_storage.json'});
test.describe('pw1_test', () => {

    test('test', async ({ page }) => {

        // Go to https://github.com/
        await page.goto('https://github.com/');

        // Click text=Pull requests
        await page.click('text=Pull requests');
        await expect(page).toHaveURL('https://github.com/pulls');

    });
});

pw2_test_spec.js

const {test,expect} = require("@playwright/test");

test.use({storageState: 'empty_storage.json'});
test.describe('pw2_test', () => {

    test('test', async ({ page }) => {

        // Go to https://github.com/
        await page.goto('https://gmail.com/');

    });
});

Kindly let me know if you need any other information.

[vsravuri]

I filed a new bug #10290