--all is super broken

[jwalton]

Link to bug demonstration repository.

https://github.com/exegesis-js/exegesis

Expected Behavior

--all should include all files

Observed Behavior

--all does not include all files, and sometimes randomly drops files (sometimes even files with coverage), or (extra weirdly) parts of files.

If you run (and re-run) npx nyc mocha 'test/**/*.@(ts|js)' in the Exegesis repo, it will sometimes report:

=============================== Coverage summary ===============================
Statements   : 83.38% ( 1164/1396 )
Branches     : 74.81% ( 677/905 )
Functions    : 83.59% ( 219/262 )
Lines        : 84.14% ( 1146/1362 )
================================================================================

And sometimes report:

=============================== Coverage summary ===============================
Statements   : 83.85% ( 1064/1269 )
Branches     : 74.82% ( 624/834 )
Functions    : 82.98% ( 195/235 )
Lines        : 84.67% ( 1044/1233 )
================================================================================

Note the total number of lines has changed. Repeated runs will always generate either 1164/1396 statements covered, or 1064/1269 covered, seemingly at random. Digging a little deeper, in the "bad case" some files have most of their lines mysteriously not counted:

And some files are just missing entirely (despite the fact that I'm using --all):

Removing the "all" option (from .nycrc in this case) seems to fix the problem, and I consistently get 1164/1396 statements covered.

Forensic Information

Operating System: MacOS 10.13.6 (High Sierra)
Environment Information: https://gist.github.com/jwalton/5107d4db159ef4dcd9f1d233b92a0f4c

(And, I tried upgrading to nyc 13.1.0 after I grabbed this output.txt - no difference.)

[JaKXz]

I think I'm seeing consistent results when I add cache: false to your .nycrc. Not 100% sure that's the problem, however, and that makes this issue more interesting...

I ran npm test, npx nyc mocha 'test/**/*.@(ts|js)', and yarn run nyc mocha 'test/**/*.@(ts|js)' and got the following summary at time of writing:

=============================== Coverage summary ===============================
Statements   : 83.62% ( 1179/1410 )
Branches     : 75.46% ( 692/917 )
Functions    : 83.71% ( 221/264 )
Lines        : 84.38% ( 1161/1376 )
================================================================================

from my fork: https://github.com/JaKXz/exegesis

npx envinfo

npx: installed 1 in 2.209s

  System:
    OS: macOS 10.14.2
    CPU: (4) x64 Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz
    Memory: 179.17 MB / 8.00 GB
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 11.6.0 - /usr/local/bin/node
    Yarn: 1.12.3 - /usr/local/bin/yarn
    npm: 6.4.1 - /usr/local/bin/npm

Kudos on the repo btw! :)

[bcoe]

my bet would be that this is order of operations of how the raw coverage data is combined across multiple processes; issues along these lines have cropped up quite a bit with TypeScript, which requires several source-map transformations to map coverage.

[za-creature]

Moe:cf-emu radu$ npx nyc --require esm --all mocha
Cannot convert undefined or null to object

No, it's not

/s

Sorry, long day

[JaKXz]

I strongly encourage the use of the nyc-config-typescript package as a base config for typescript projects! It's been mostly effective in helping people with basic problems like this! @jwalton @za-creature can you try that and let me know if you're still experiencing this isssue?

[morris]

I can confirm that all: true generates largely inaccurate coverage (in our case, missing files in the report and about 15% below our expected coverage). This is with latest versions of everything and fresh installs (no package-lock). Can be easily reproduced by updating dependencies in https://github.com/philipbeel/example-typescript-nyc-mocha-coverage and running with all: true.

@JaKXz the nyc-config-typescript package doesn't help here. It's just a base config without specifying all.

[coreyfarrell]

@morris your package is using very old versions of almost everything. Simply upgrading everything causes the test to fail. [email protected] was just released which does fix some issues with ts + source-map + all. Please update your repository, if you are continuing to have trouble please post a new bug including all requested information.

[JaKXz]

@morris I fixed your example repo in a couple minutes: philipbeel/example-typescript-nyc-mocha-coverage#2

The documentation for nyc-config-typescript has been updated recently. Please follow it closely.

[morris]

Thanks @JaKXz, this works :) not my repo, though - hopefully the PR is accepted for other googlers. FWIW here's the config that lead me to believe --all is bad:

  "nyc": {
    "cache": false,
    "all": true,
    "reporter": [
      "text",
      "lcov",
      "html"
    ],
    "extension": [".js", ".ts"]
  },

When used in @JaKXz fork, and running "npm run coverage" twice without clearing .nyc_output, this gives me weird results (only on the test files). Surely a contrived example (nothing excluded etc.) but something similar happened to us in our project, and I wouldn't expect it to break.

Note the ".js" extension; it's this exact combination which breaks, and only without clearing .nyc_output, so maybe not even an issue with --all.

[JaKXz]

running "npm run coverage" twice without clearing .nyc_output

Not exactly sure how that is happening, unless you have modified the --clean flag's value, I think it's supposed to clear .nyc_output

Correct me if I'm wrong @bcoe @coreyfarrell?

[coreyfarrell]

That is correct, deleting .nyc_output is one of the first things that happens when you run nyc mocha. It is deleted before mocha is run, and before any sources are transpiled.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[electrovir]

I was just running into similar perplexing issues with --all and TypeScript that were stumping me and nyc-config-typescript fixed them all!