MidsceneJS integration

[RussellCanfield]

MidsceneJS is an AI powered e2e test framework. This discussion will talk through integration and how it could potentially work.

Problem statements

• I don't know which directory the tests are in
• I don't know the command to execute e2e tests
• I don't know which tests cover what
• I don't know the import for importing the test fixture Midscene has you create
• Limit integrations to playwright

Suggestions

For determining the directory of tests, add a new section to configuration to allow the user to set the root directory of the test project, and the command to execute the e2e test suite. This has limitations in monorepos, its possible to maybe create another map here as well. Map projects to e2e tests.

For determining which tests to cover, when the user configures the test directory - kick off a "crawl" of the directory and create a simple map of files -> intent, for example:

"example.spec.ts" -> Covers testing the home page and searches for titles. Has a test for products on sale, etc.

Use this intent map, along with Composer results to find a test file that may fit appropriately or create one if there is no reasonable match.

Test File Generation

This has a few challenges, say you have a React component that has another component it renders as a child. If the AI has to generate a test for it, or generate a new test file. How does it know, how to test the child component? This may warrant its own refinement loop in order to determine how to test those files - and if the files are overly complex it may need modifications; for instance the css selectors are difficult.

Another potential problem is screen size and resolution. It may regularly fail because it didn't scroll enough or didn't take into account screen size/dimensions. This is a hard problem to solve and could be solved with another feedback loop that can analyze a screenshot of the HTML output - although this is not ideal.