Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up CircleCI test summary [npmcollaborators] #2526

Merged
merged 14 commits into from
Dec 16, 2018
Merged

Set up CircleCI test summary [npmcollaborators] #2526

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
71b0465
Set up test summary
paulmelnikow Dec 13, 2018
83ef124
Add failing tests
paulmelnikow Dec 13, 2018
93304b9
Diagnose
paulmelnikow Dec 13, 2018
a76f105
Again
paulmelnikow Dec 13, 2018
7e6c327
More fixes
paulmelnikow Dec 13, 2018
a556263
Fixes / progress
paulmelnikow Dec 13, 2018
964c45b
Show service test failures
paulmelnikow Dec 13, 2018
deb63be
Reset fake failing tests
paulmelnikow Dec 13, 2018
3d6c588
Use folders
paulmelnikow Dec 13, 2018
85e0403
Merge branch 'master' into test-summary
paulmelnikow Dec 16, 2018
5e13553
Try to use xunit
paulmelnikow Dec 16, 2018
00b556f
Revert "Try to use xunit"
paulmelnikow Dec 16, 2018
0d4a6bd
Merge branch 'master' into test-summary
paulmelnikow Dec 16, 2018
a727346
Merge branch 'master' into test-summary
paulmelnikow Dec 16, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 94 additions & 21 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ jobs:

- restore_cache:
keys:
- v2-dependencies-{{ checksum "package.json" }}
# fallback to using the latest cache if no exact match is found
- v2-dependencies-
- v2-dependencies-{{ checksum "package.json" }}
# fallback to using the latest cache if no exact match is found
- v2-dependencies-

- run:
name: Install dependencies
Expand All @@ -30,9 +30,10 @@ jobs:
- checkout

- restore_cache:
key: v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
key: v2-dependencies-
keys:
- v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
- v2-dependencies-

- run:
name: Install dependencies
Expand All @@ -43,21 +44,38 @@ jobs:
when: always
command: npm run lint

- run:
name: Prepare test summary
when: always
command: mkdir junit

- run:
name: Server unit tests
when: always
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/server/results.xml
command: npm run test:js:server

- run:
name: Integration tests
when: always
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/server/integration.xml
command: npm run test:integration

- run:
name: Tests for gh-badges package
when: always
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/server/gh-badges.xml
command: npm run test:js:package

- store_test_results:
path: junit

- run:
name: 'Prettier check (quick fix: `npm run prettier`)'
when: always
Expand All @@ -71,9 +89,10 @@ jobs:
- checkout

- restore_cache:
key: v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
key: v2-dependencies-
keys:
- v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
- v2-dependencies-

- run:
name: Install dependencies
Expand All @@ -84,21 +103,38 @@ jobs:
when: always
command: npm run lint

- run:
name: Prepare test summary
when: always
command: mkdir junit

- run:
name: Server unit tests
when: always
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/server/results.xml
command: npm run test:js:server

- run:
name: Integration tests
when: always
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/integration/results.xml
command: npm run test:integration

- run:
name: Tests for gh-badges package
when: always
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/gh-badges/results.xml
command: npm run test:js:package

- store_test_results:
path: junit

- run:
name: 'Prettier check (quick fix: `npm run prettier`)'
when: always
Expand All @@ -111,9 +147,10 @@ jobs:
- checkout

- restore_cache:
key: v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
key: v2-dependencies-
keys:
- v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
- v2-dependencies-

- run:
name: Install dependencies
Expand All @@ -131,9 +168,10 @@ jobs:
- checkout

- restore_cache:
key: v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
key: v2-dependencies-
keys:
- v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
- v2-dependencies-

- run:
name: Install dependencies
Expand All @@ -143,10 +181,21 @@ jobs:
name: Prepare frontend tests
command: npm run defs && npm run features

- run:
name: Prepare test summary
when: always
command: mkdir junit

- run:
name: Frontend unit tests
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/frontend/results.xml
command: npm run test:js:frontend

- store_test_results:
path: junit

- run:
name: Frontend build completes successfully
when: always
Expand All @@ -165,9 +214,10 @@ jobs:
echo "{\"gh_token\":\"$GITHUB_TOKEN\", \"wheelmap_token\":\"$WHEELMAP_TOKEN\"}" > private/secret.json

- restore_cache:
key: v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
key: v2-dependencies-
keys:
- v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
- v2-dependencies-

- run:
name: Install dependencies
Expand All @@ -177,10 +227,21 @@ jobs:
name: Identify services tagged in the PR title
command: npm run test:services:pr:prepare

- run:
name: Prepare test summary
when: always
command: mkdir junit

- run:
name: Run tests for tagged services
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/services/results.xml
command: npm run test:services:pr:run

- store_test_results:
path: junit

services@node-latest:
docker:
- image: circleci/node:latest
Expand All @@ -194,9 +255,10 @@ jobs:
echo "{\"gh_token\":\"$GITHUB_TOKEN\"}" > private/secret.json

- restore_cache:
key: v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
key: v2-dependencies-
keys:
- v2-dependencies-{{ checksum "package.json" }}
# https://github.com/badges/shields/issues/1937
- v2-dependencies-

- run:
name: Install dependencies
Expand All @@ -206,10 +268,21 @@ jobs:
name: Identify services tagged in the PR title
command: npm run test:services:pr:prepare

- run:
name: Prepare test summary
when: always
command: mkdir junit

- run:
name: Run tests for tagged services
environment:
mocha_reporter: mocha-junit-reporter
MOCHA_FILE: junit/services/results.xml
command: npm run test:services:pr:run

- store_test_results:
path: junit

workflows:
version: 2

Expand Down
1 change: 1 addition & 0 deletions mocha.opts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
--reporter mocha-env-reporter
42 changes: 42 additions & 0 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 13 additions & 11 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@
"xpath": "~0.0.27"
},
"scripts": {
"coverage:test:frontend": "BABEL_ENV=test nyc node_modules/mocha/bin/_mocha --require @babel/polyfill --require @babel/register --require mocha-yaml-loader \"frontend/**/*.spec.js\"",
"coverage:test:server": "HANDLE_INTERNAL_ERRORS=false nyc node_modules/mocha/bin/_mocha \"*.spec.js\" \"lib/**/*.spec.js\" \"services/**/*.spec.js\"",
"coverage:test:package": "nyc node_modules/mocha/bin/_mocha \"gh-badges/**/*.spec.js\"",
"coverage:test:integration": "nyc node_modules/mocha/bin/_mocha \"lib/**/*.integration.js\" \"services/**/*.integration.js\"",
"coverage:test:services": "nyc node_modules/mocha/bin/_mocha --delay lib/service-test-runner/cli.js",
"coverage:test:frontend": "BABEL_ENV=test nyc node_modules/mocha/bin/_mocha --opts mocha.opts --require @babel/polyfill --require @babel/register --require mocha-yaml-loader \"frontend/**/*.spec.js\"",
"coverage:test:server": "HANDLE_INTERNAL_ERRORS=false nyc node_modules/mocha/bin/_mocha --opts mocha.opts \"*.spec.js\" \"lib/**/*.spec.js\" \"services/**/*.spec.js\"",
"coverage:test:package": "nyc node_modules/mocha/bin/_mocha --opts mocha.opts \"gh-badges/**/*.spec.js\"",
"coverage:test:integration": "nyc node_modules/mocha/bin/_mocha --opts mocha.opts \"lib/**/*.integration.js\" \"services/**/*.integration.js\"",
"coverage:test:services": "nyc node_modules/mocha/bin/_mocha --opts mocha.opts --delay lib/service-test-runner/cli.js",
"coverage:test": "rimraf .nyc_output coverage; npm run coverage:test:server; npm run coverage:test:package; npm run coverage:test:frontend; npm run coverage:test:integration; npm run coverage:test:services",
"coverage:report": "nyc report",
"coverage:report:reopen": "opn coverage/lcov-report/index.html",
Expand All @@ -71,14 +71,14 @@
"prettier": "prettier --write \"**/*.js\"",
"prettier-check": "prettier-check \"**/*.js\"",
"danger": "danger",
"test:js:frontend": "BABEL_ENV=test mocha --require @babel/polyfill --require @babel/register --require mocha-yaml-loader \"frontend/**/*.spec.js\"",
"test:js:server": "HANDLE_INTERNAL_ERRORS=false mocha \"*.spec.js\" \"lib/**/*.spec.js\" \"services/**/*.spec.js\"",
"test:js:package": "mocha \"gh-badges/**/*.spec.js\"",
"test:integration": "mocha \"lib/**/*.integration.js\" \"services/**/*.integration.js\"",
"test:services": "HANDLE_INTERNAL_ERRORS=false mocha --delay lib/service-test-runner/cli.js",
"test:js:frontend": "BABEL_ENV=test mocha --opts mocha.opts --require @babel/polyfill --require @babel/register --require mocha-yaml-loader \"frontend/**/*.spec.js\"",
"test:js:server": "HANDLE_INTERNAL_ERRORS=false mocha --opts mocha.opts \"*.spec.js\" \"lib/**/*.spec.js\" \"services/**/*.spec.js\"",
"test:js:package": "mocha --opts mocha.opts \"gh-badges/**/*.spec.js\"",
"test:integration": "mocha --opts mocha.opts \"lib/**/*.integration.js\" \"services/**/*.integration.js\"",
"test:services": "HANDLE_INTERNAL_ERRORS=false mocha --opts mocha.opts --delay lib/service-test-runner/cli.js",
"test:services:trace": "TRACE_SERVICES=true npm run test:services -- $*",
"test:services:pr:prepare": "node lib/service-test-runner/pull-request-services-cli.js > pull-request-services.log",
"test:services:pr:run": "HANDLE_INTERNAL_ERRORS=false mocha --delay lib/service-test-runner/cli.js --stdin < pull-request-services.log",
"test:services:pr:run": "HANDLE_INTERNAL_ERRORS=false mocha --opts mocha.opts --delay lib/service-test-runner/cli.js --stdin < pull-request-services.log",
"test:services:pr": "npm run test:services:pr:prepare && npm run test:services:pr:run",
"test": "npm run defs && npm run lint && npm run test:js:frontend && npm run test:js:package && npm run test:js:server",
"depcheck": "check-node-version --node \">= 8.0\"",
Expand Down Expand Up @@ -157,6 +157,8 @@
"minimist": "^1.2.0",
"mkdirp": "^0.5.1",
"mocha": "^5.0.0",
"mocha-env-reporter": "^3.0.0",
Copy link
Member

@calebcartwright calebcartwright Dec 14, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using one of the multi reporters, like mocha-multi-reporters or mocha-multi? That way we can still keep the terminal (spec) reporter as well for the benefit of local development and also get the results in the xml file that Circle needs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mocha-env-reporter should use the default reporter outside of CI, though if there’s some other benefit to showing both I could change it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I remember looking at that a while back but I see now where I may have gotten confused by the list of CI servers in the readme

"mocha-junit-reporter": "^1.18.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also FWIW, Circle can parse the "xunit" format that is emitted by mocha's native/bundled xunit reporter which could save having to add another dev dependency

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, nice! I’ll give that a shot.

Copy link
Member

@calebcartwright calebcartwright Dec 16, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how well the xunit reporter will play with mocha-env-reporter though. I always pass the file path for the xunit file via the reporter options --reporter-options ./foo/bar.xml but no clue if that can be specified via an env variable or some other mechanism

Copy link
Member Author

@paulmelnikow paulmelnikow Dec 16, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like with xunit there's not a way to configure the output file through the environment, whereas mocha-junit-reporter supports this. I opened mochajs/mocha#3621 with this ask.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, not worth fighting xunit here IMO :) My guess is that they will point you to one of the community reporters (like xunit-file) that provide support for env var defined output file, at which point we're right back to what the junit reporter already gives us

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, possibly. I have a soft spot for Mocha, especially after some recent unpleasant experiences with Jest doing unexpected things with setup. However I do find the way Mocha handles reporters and configuration to be frustrating, fiddly, and to require more effort than it should. I was happy with some upcoming improvements to the config which they mentioned in mochajs/mocha#3612. 🤞

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a big fan of Mocha too. I've only worked with Jest in one project, but I do hear a lot of good things about it. What sort of setup issues have you seen with Jest?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hit these issues:

"mocha-yaml-loader": "^1.0.3",
"next": "^6.1.1",
"nock": "^10.0.4",
Expand Down