Axe API integration with Jest and Puppeteer

[gziolo]

Description

Implementation based on #12743.

Hopefully this is th final try to include axe integration with out e2e tests setup. This PR introduces new Jest matcher which takes a Puppeteer page instance and check if it is accesible as shown below:

await expect( page ).toPassAxeTests( {
	include: '.block-editor',
	exclude: '#metaboxes, #post-title-0, .editor-default-block-appender__content',
} );

How has this been tested?

npm run test-e2e with the following patch applied in addition to those changes:

diff --git a/packages/tests-e2e/support/setup-test-framework.js b/packages/tests-e2e/support/setup-test-framework.js
index cc8168644da..89cfc37b127 100644
--- a/packages/tests-e2e/support/setup-test-framework.js
+++ b/packages/tests-e2e/support/setup-test-framework.js
@@ -162,6 +162,7 @@ beforeAll( async () => {
 } );
 
 afterEach( async () => {
+	await expect( page ).toPassAxeTests();
 	await setupBrowser();
 } );

Types of changes

New feature for e2e testing.

Known issues

There is a bug in Jest with async matcher which makes them print differently than regular matchers. See jestjs/jest#7066 for more details. They are working on a fix and it looks like it might land in the next major release of Jest.

Checklist:

• My code is tested.
• My code follows the WordPress code style.
• My code follows the accessibility standards.
• My code has proper inline documentation.
• I've included developer documentation if appropriate.

[gziolo]

Logs with accessibility violations (closer to the end of file) for jobs without popular plugins and author role:

• https://api.travis-ci.org/v3/job/478442476/log.txt (part 1)
• https://api.travis-ci.org/v3/job/478442477/log.txt (part 2)

To test it locally try the following diff:

diff --git a/packages/tests-e2e/support/setup-test-framework.js b/packages/tests-e2e/support/setup-test-framework.js
index cc8168644da..89cfc37b127 100644
--- a/packages/tests-e2e/support/setup-test-framework.js
+++ b/packages/tests-e2e/support/setup-test-framework.js
@@ -162,6 +162,7 @@ beforeAll( async () => {
 } );
 
 afterEach( async () => {
+	await expect( page ).toPassAxeTests();
 	await setupBrowser();
 } );

[gziolo]

This is ready for review. This PR in the final version adds only new package which exposes the async matcher to validate whether a given page or its part is accessible according to Axe tool analysis.

@stephenmathieson and @dylanb, I would appreciate your feedback on this one as it's based on axe-puppeteer and your great work started at WC US (#12743) :)

[greatislander]

I've proposed some changes based on concerns that the function name and documentation could lead users to think that running Axe alone can accurately determine whether or not a page is accessible. Any automated accessibility tool can only catch a certain subset of potential accessibility issues, and must be combined with manual evaluation. It is probably better not to use language like:

expect( page ).toBeAccessible();

But rather to use documentation and function names which indicates that the evaluation is based on criteria of this specific tool, e.g.:

expect( page ).toPassAxeTests();

Open to and grateful for other ideas of how to frame this.

[gziolo]

I've proposed some changes based on concerns that the function name and documentation could lead users to think that running Axe alone can accurately determine whether or not a page is accessible. Any automated accessibility tool can only catch a certain subset of potential accessibility issues, and must be combined with manual evaluation.

Totally agree with your reasoning 👍 I didn't think about it at all. I was too focused on the technical aspects. Thanks for your review. I applied all the changes you proposed as they better explain the intents of this new Jest matcher introduced. I'm not 100% sure if tests in expect( page ).toPassAxeTests(); is the best fit but I'm having hard time finding out a better alternative. Some ideas I had are:

expect( page ).toPassAxeChecks();
expect( page ).toPassAxeAnalysis();
expect( page ).toPassAxeVerification();

We can totally proceed as is if there isn't a better proposal.

[greatislander]

@gziolo thanks for accepting the suggestions! I hear the concern around using Tests in the function name. Of your suggestions I think I'd lean towards:

expect( page ).toPassAxeChecks();

I also thought of:

expect( page ).toPassAxeAudit();

Open to other suggestions too.

[aduth]

The messaging of a failing test is non-conventional, with only an "Error" text being highlighted. In direct conversation, it was mentioned this is a bug present in Jest for the moment. Pending @gziolo to share.

In the meantime, this appears good as it stands. 👍

[dylanb]

I am a little late on this but this looks like great progress

[dylanb]

@gziolo seems like a good improvement would be to publish a axe-jest-puppeteer module that implements the matcher and use that - WDYT?

cc/ @WilcoFiers @stephenmathieson

[gziolo]

This is going to be published to npm as @wordpress/jest-puppeteer-axe closer to the end of February, is it what you mean?