Support for jest shards

[mccraveiro]

Description

Hey @ArtiomTr, any plans to support jest 28 shards?

I tried to merge them with NYC but it seems to only work with coverage files and not report files.
Also when I tried to use the coverage file I got the same erros as #268

Thanks you :)

[ArtiomTr]

Hello @mccraveiro 👋,

Don't really know, what are the "shards", but I will try to investigate how they could be used for this action.

About the error that you're getting - have you tried the solutions, described in this comment #268 (comment)? If none of these fix your issue, please ping me, I will try to provide more solutions for that issue.

[mccraveiro]

@ArtiomTr Shards are a new way of splitting your tests and running them in multiple workers. It's pretty useful in GitHub actions with a matrix strategy.

The issue with shards is that you get n reports.json and then you need to merge them. However, I couldn't merge them in a way that works with this action.

Thank you

[brunosala]

To clarify, the issue @mccraveiro is having is related to a mismatch in instanbul/nyc versions from what is on their main release vs what is shipped with Jest. There are a handful of other threads that describe the issue and how to patch: istanbuljs/nyc#1302

As for a workaround there is this: jestjs/jest#11581 as a workaround but it wouldn't be consumable by this GH action.

[luke-lacroix-healthy]

I've been struggling with this myself. There are no tools to merge Jest's seemingly proprietary report.json files, so I have been trying to work with Istanbul's output from Jest instead - I got that merging but the available GitHub actions are lightyears behind what this action does (annotations and comments would have to be coded and added by myself).

We have a slightly different use case for shards. We shard because of memory limitations on the Github Actions workers and the fact that Jest running on recent versions of node eats up TONS of memory: jestjs/jest#11956. Even with forced garbage collection, we still hit the cap. So, we shard the larger test suites so that they will finish but the Jest tool itself does not even support code coverage reports with shards: jestjs/jest#12751.

I think that if the action allowed us to specify multiple report.json files (possibly through a glob??) and it handled merging them, then it would work beautifully for our needs.

[arminrosu]

For those also struggling with this, I managed to get it working. Here's an example of a github actions workflow:

  run-tests:
      # ...

      - name: Run Jest
        run: |
          yarn jest \
            --coverage --json --outputFile=coverage/coverage-shard-${{ matrix.shard }}.json --testLocationInResults \
            --shard=${{ matrix.shard }}/${{ strategy.job-total }} \

      - uses: actions/upload-artifact@v3
        with:
          name: coverage-artifacts
          path: coverage/

  report-coverage:
    runs-on: ubuntu-latest
    needs:
      - run-tests
    steps:
      - uses: actions/checkout@v3

      - uses: actions/download-artifact@v3
        with:
          name: coverage-artifacts
          # will cache files for all shards
          path: coverage/

      - name: Merge coverage reports
        run: |
          # Merge sharded reports into one
          jq 'reduce inputs as $i (.; . * $i)' coverage/coverage-shard-*.json > coverage-merged.json

      - name: Publish test results
        uses: ArtiomTr/jest-coverage-report-action@v2
        with:
          base-coverage-file: coverage-merged.json
          coverage-file: coverage-merged.json

[luke-lacroix-healthy]

I can confirm the above does work.

[mhd3v]

While the above comment from @arminrosu works, the merged file only reports coverage from one of the shards. Which is leading me to believe that the reduce is not working as expected.

Even when I download the artifacts and run the reduce through jq locally, I do get a merged file but the actual values inside the report are not getting combined which causes the coverage report to be incorrect. Does anyone have any ideas here what the issue could be?

jq 'reduce inputs as $i (.; . * $i)' coverage/coverage-shard-*.json > coverage-merged.json

[MorenoMdz]

While the above comment from @arminrosu works, the merged file only reports coverage from one of the shards. Which is leading me to believe that the reduce is not working as expected.

Even when I download the artifacts and run the reduce through jq locally, I do get a merged file but the actual values inside the report are not getting combined which causes the coverage report to be incorrect. Does anyone have any ideas here what the issue could be?

jq 'reduce inputs as $i (.; . * $i)' coverage/coverage-shard-*.json > coverage-merged.json

Interesting, would the approach used in this article help?

cc @arminrosu

[Dutchie1990]

any updates?

[redabacha]

here's my solution to this problem. you run jest with sharding as normal but output the report to a different file (in the format of jest-report-[shard].json) for each shard and then use the following script to merge the reports from each shard into a single report:

const fs = require('node:fs')
const path = require('node:path')
const istanbul = require('istanbul-lib-coverage')

const reportDirectory = process.cwd()
const reportFiles = fs
  .readdirSync(reportDirectory)
  .filter(file => /jest-report-[0-9]+.json$/.test(file))
console.log(`merging ${reportFiles.length} reports: ${reportFiles.join(', ')}`)

const combinedReport = {}
const combinedCoverageMap = istanbul.createCoverageMap({})

for (const reportFile of reportFiles) {
  const report = JSON.parse(
    fs.readFileSync(path.join(reportDirectory, reportFile), {
      encoding: 'utf8',
    }),
  )

  for (const key of Object.keys(report)) {
    if (key.startsWith('num')) {
      combinedReport[key] ??= 0
      combinedReport[key] += report[key]
    }
  }

  combinedReport.startTime ??= report.startTime
  if (report.startTime < combinedReport.startTime) {
    combinedReport.startTime = report.startTime
  }

  combinedReport.success ??= true
  combinedReport.success = combinedReport.success && report.success

  combinedReport.wasInterrupted ??= false
  combinedReport.wasInterrupted =
    combinedReport.wasInterrupted || report.wasInterrupted

  combinedReport.snapshot ??= {
    added: 0,
    didUpdate: false,
    failure: false,
    filesAdded: 0,
    filesRemoved: 0,
    filesRemovedList: [],
    filesUnmatched: 0,
    filesUpdated: 0,
    matched: 0,
    total: 0,
    unchecked: 0,
    uncheckedKeysByFile: [],
    unmatched: 0,
    updated: 0,
  }
  combinedReport.snapshot.added += report.snapshot.added
  combinedReport.snapshot.didUpdate =
    combinedReport.snapshot.didUpdate || report.snapshot.didUpdate
  combinedReport.snapshot.failure =
    combinedReport.snapshot.failure || report.snapshot.failure
  combinedReport.snapshot.filesAdded += report.snapshot.filesAdded
  combinedReport.snapshot.filesRemoved += report.snapshot.filesRemoved
  combinedReport.snapshot.filesUnmatched += report.snapshot.filesUnmatched
  combinedReport.snapshot.filesUpdated += report.snapshot.filesUpdated
  combinedReport.snapshot.matched += report.snapshot.matched
  combinedReport.snapshot.total += report.snapshot.total
  combinedReport.snapshot.unchecked += report.snapshot.unchecked
  combinedReport.snapshot.uncheckedKeysByFile = [
    ...combinedReport.snapshot.uncheckedKeysByFile,
    ...report.snapshot.uncheckedKeysByFile,
  ]
  combinedReport.snapshot.unmatched += report.snapshot.unmatched
  combinedReport.snapshot.updated += report.snapshot.updated

  combinedReport.openHandles = [
    ...(combinedReport.openHandles ?? []),
    ...report.openHandles,
  ]

  combinedReport.testResults = [
    ...(combinedReport.testResults ?? []),
    ...report.testResults,
  ]

  combinedCoverageMap.merge(report.coverageMap)
}

combinedReport.coverageMap = combinedCoverageMap.toJSON()

fs.writeFileSync('jest-report-combined.json', JSON.stringify(combinedReport))

the script uses the istanbul-lib-coverage package to merge the code coverage together.