Improve AI usage of newly open tabs #254
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
app/(dashboard)/page.test.ts
Outdated
| shortest("Verify that buttons on the landing page are rounded"); | ||
| shortest("Login using this email: [email protected]"); | ||
| shortest("Log in", { email: loginEmail }); | ||
| shortest("Verify that the user can access the /dashboard page"); |
There was a problem hiding this comment.
This test doesn't seem to work as the user with Mailosaur email doesn't have GitHub connected, and the AI is confused by the Dashboard page:
Unsure what would be a good way to fix it. Maybe change the test, or just delete for now?
There was a problem hiding this comment.
We can delete the test for now.
|
The test is failing as AI seems to be confused on using the new tab after clicking the During the run below, Attempting to adjust the system prompt to handle this better.
|
|
Weird as it was working before with the old prompt I believe |
|
I've executed the same test from Run 1: passed once it checked the email was received and had a link (did not click on it)
Runs 2&3: failed because couldn't switch to the new tab after clicking the email link (kept retrying by opening new tabs 5 times)All the screenshots taken were of the email tab.
Run 4: passed after clicking the email link, didn't check the new page loaded
|
|
|
||
| const loginEmail = `shortest@${process.env.MAILOSAUR_SERVER_ID}.mailosaur.net`; | ||
| shortest("Log in", { email: loginEmail }).expect( | ||
| "Check Manage Account page from user icon menu", |
There was a problem hiding this comment.
Added a chained test to test that the user is actually logged in.
| - If you need to test a condition that involves seeing the contents of an email, use the "check_email" tool. | ||
| - For email validation, you MUST always use 'Click' and 'Mouse' action instead of using keyboard shortcuts. | ||
| - This tool will grab the latest email from the email address given to you and will render it in a new tab for you to see. | ||
| - Once you are done with validating the email, navigate back to the original tab. |
There was a problem hiding this comment.
Since now we're automatically focusing on the new tab when opening a link, this is not needed. In my tests, AI couldn't navigate between tabs.
| @@ -62,6 +62,12 @@ export class BrowserTool extends BaseBrowserTool { | |||
| this.viewport = { width: config.width, height: config.height }; | |||
| this.testContext = config.testContext; | |||
|
|
|||
| // Update active page reference to a newly opened tab | |||
| this.page.context().on("page", async (newPage) => { | |||
There was a problem hiding this comment.
This fixes the issue reported, where the AI didn't have the new tab, after clicking on the email link, as active.
@m2rads do you think this could cause any possible side effects?
rmarescu
changed the title
shorters@<MAILOSAUR_SERVER_ID>.mailosaur.net