rule function gets called twice for each invalid test case

[shogunsea]

Issue workflow progress

Progress of the issue based on the Contributor Workflow

• 1. The issue provides a reproduction available on GitHub, Stackblitz or CodeSandbox

Please make sure the graphql-eslint version under package.json matches yours.

• 2. A failing test has been provided
• 3. A local solution has been provided
• 4. A pull request is pending review

---

Describe the bug

the create function is called twice for each invalid test cases:
1 valid + 2 invalid = 1 + 2* 2 = 5 times logging

When I only keep 1 invalid test case:

This does NOT happen in eslint or typescript-eslint rule test runner.

To Reproduce
Steps to reproduce the behavior:

This can be easily reproduced by adding a console log like here: https://github.com/B2o5T/graphql-eslint/blob/7251269d9806202a1d9d6651a809a49396656978/packages/plugin/src/rules/no-unused-fields.ts#L105-L107
then run the spec:

yarn test packages/plugin/tests/no-unused-fields.spec.ts

Expected behavior

Environment:

• OS: macOS
• @graphql-eslint/eslint-plugin:
• Node.js: 16.10

Additional context

[dimaMachina]

Hi @shogunsea, nice catch 🙈 I know about this issue 🙂, it happens because I run tests 2 times, the first one with the default eslint rule tester
https://github.com/B2o5T/graphql-eslint/blob/5336c0a0212b74bb9c333b0311ab1ce62248404f/packages/plugin/src/testkit.ts#L85

and the second with snapshot rule tester

https://github.com/B2o5T/graphql-eslint/blob/5336c0a0212b74bb9c333b0311ab1ce62248404f/packages/plugin/src/testkit.ts#L123

to ensure correct highlight cursor position, example:

https://github.com/B2o5T/graphql-eslint/blob/5336c0a0212b74bb9c333b0311ab1ce62248404f/packages/plugin/tests/__snapshots__/alphabetize.spec.md?plain=1#L3-L43

I thought about removing GraphQLRuleTester from the bundle to avoid confusing by users. What do you think?

Also, you can always use the default RuleTester by eslint, just specify a correct parser

Closing it for now because I do not plan to fix it for the current moment and still waiting for support snapshot testing out of box from eslint

The related issue from eslint about snapshots testing is here eslint/eslint#14936

[shogunsea]

SGTM, thanks for the workaround 👍