Skip to content

Commit 32f0774

Browse files
authored
Infrastructure: Run regression tests in parallel in TravisCI and improve test selection logic (pull #1465)
Fixes issues #1449 and 1450 by running regression tests in parallel and improving logic that determines which tests to run. To run in parallel: 1. Updates `.travis.yml` to use the [Ava feature](https://github.com/avajs/ava#parallel-runs-in-ci) that allows tests in parallel across three VMs in the CI 2. Updates Ava version to include [this bug fix](avajs/ava#2461) -- the fix prevents a false error when Ava tries to parallelize less than three test files (it parallelizes by files, so one test file is not parallelized). 3. Updates the script that runs the regression tests in the CI to only the test files for the example or test file you are editing to take advantage of the parallelization option. Before, it filtered tests by using a regex on the name of the test. Now, it explicitly lists all the test FILES that should be run. When running `bash regression-tests.sh` locally from a branch that is not master, the script will now run as if the current branch is a pull request against master. You can now test locally that file changes in your branch will trigger the correct tests. The logic is: 1. If `package.json` has been edited, or if test utility files have been edited, or if example utility files have been edited, then all regression tests will be run in the CI. 2. Otherwise, if an example is edited, then the regression tests for all of the examples contained in the same example directory are run. For example: if you edit one of the two checkbox examples, then both the checkbox-1 and checkbox-2 example regression tests will be run in the CI. 3. All test files that have been edited will be run in the CI.
1 parent 962481a commit 32f0774

File tree

4 files changed

+260
-181
lines changed

4 files changed

+260
-181
lines changed

.travis.yml

+31-8
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ git:
77
depth: 3
88

99
stages:
10+
- Lint
1011
- Test
1112
- Deploy
1213

@@ -16,30 +17,52 @@ jobs:
1617
- env: ALLOW_FAILURE=true
1718

1819
include:
19-
- stage: Test
20+
- stage: Lint
2021
name: CSS Linting
2122
script: npm run lint:css
22-
- stage: Test
23+
- stage: Lint
2324
name: JS Linting
2425
script: npm run lint:es
25-
- stage: Test
26+
- stage: Lint
2627
name: HTML Linting
2728
script: npm run vnu-jar
28-
- stage: Test
29+
- stage: Lint
2930
name: Spellcheck
3031
script: npm run lint:spelling
3132
env: ALLOW_FAILURE=true
33+
- stage: Lint
34+
name: Regression Tests Coverage Report
35+
script: node test/util/report.js
36+
env: ALLOW_FAILURE=true
3237

3338
- stage: Test
3439
name: AVA Regression Tests
3540
addons:
3641
firefox: latest
3742
script: scripts/regression-tests.sh
38-
env: TEST_WAIT_TIME=1000
43+
env:
44+
- TEST_WAIT_TIME=1000
45+
- CI_NODE_TOTAL=3
46+
- CI_NODE_INDEX=0
3947
- stage: Test
40-
name: Regression Tests Coverage Report
41-
script: node test/util/report.js
42-
env: ALLOW_FAILURE=true
48+
name: AVA Regression Tests
49+
addons:
50+
firefox: latest
51+
script: scripts/regression-tests.sh
52+
env:
53+
- TEST_WAIT_TIME=1000
54+
- CI_NODE_TOTAL=3
55+
- CI_NODE_INDEX=1
56+
- stage: Test
57+
name: AVA Regression Tests
58+
addons:
59+
firefox: latest
60+
script: scripts/regression-tests.sh
61+
env:
62+
- TEST_WAIT_TIME=1000
63+
- CI_NODE_TOTAL=3
64+
- CI_NODE_INDEX=2
65+
4366
- stage: Deploy
4467
if: branch = master AND type != pull_request
4568
script: skip

0 commit comments

Comments
 (0)