Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bugs with regression test script: tests running when they should not, and not running when they should #1449

Closed
spectranaut opened this issue Jul 15, 2020 · 1 comment · Fixed by #1465
Closed

Bugs with regression test script: tests running when they should not, and not running when they should #1449

spectranaut opened this issue Jul 15, 2020 · 1 comment · Fixed by #1465
Assignees
@spectranaut @a11ydoer
Labels
Infrastructure Related to maintaining task force and repo operations, processes, systems, documentation regression-testing Related to AVA regression tests of example pages or AVA framework implementation within repo

Comments

@spectranaut
Copy link
Contributor

spectranaut commented Jul 15, 2020 •
edited
Loading

I noticed two weird problems with this script that handles running regression tests in travis:

  1. In PR Landmark Example pages: Make HTML techniques tab active on page load #1441, the regression tests are running although they should not. There is logic specifically to avoid running the regression tests if no examples with tests have been edited.
  2. In PR Infrastructure: Add missing test for 6 examples #1424 no regression tests are run because the script errors before getting to the ava command, and the regression check passes. We should have the check fail in this case, but we also need to debug why it errors.
@mcking65 mcking65 added Infrastructure Related to maintaining task force and repo operations, processes, systems, documentation regression-testing Related to AVA regression tests of example pages or AVA framework implementation within repo labels Jul 19, 2020
@spectranaut
Copy link
Contributor Author

I've looked into both issues and have fixes, hopefully will open a PR tomorrow!

For bug part 1: There simply needs to be a little extra logic for when there is change in the landmark directory only -- this should be straight forward.

For bug part 2: The issue turns out to be related to force pushing a branch and using the environment variable TRAVIS_COMMIT_RANGE, which no longer represents a valid commit range after a force push (see travis-ci/travis-ci#2668). I think we can replace this variable with master...$TRAVIS_BRANCH but I'll need to test that this works on the PR and branch builds.

@spectranaut spectranaut mentioned this issue Jul 24, 2020
Merged
@mcking65 mcking65 linked a pull request Aug 4, 2020 that will close this issue
Run regression tests in parallel in TravisCI and other regression tests improvements #1465
Merged
mcking65 pushed a commit that referenced this issue Aug 4, 2020
@spectranaut
Infrastructure: Run regression tests in parallel in TravisCI and impr…
32f0774 
…ove test selection logic (pull #1465)

Fixes issues #1449 and 1450 by running regression tests in parallel and improving logic that determines which tests to run.

To run in parallel:
1. Updates `.travis.yml` to use the [Ava feature](https://github.com/avajs/ava#parallel-runs-in-ci) that allows tests in parallel across three VMs in the CI
2. Updates Ava version to include [this bug fix](avajs/ava#2461) -- the fix prevents a false error when Ava tries to parallelize less than three test files (it parallelizes by files, so one test file is not parallelized).
3. Updates the script that runs the regression tests in the CI to only the test files for the example or test file you are editing to take advantage of the parallelization option. Before, it filtered tests by using a regex on the name of the test. Now, it explicitly lists all the test FILES that should be run.

When running `bash regression-tests.sh` locally from a branch that is not master, the script will now run as if the current branch is a pull request against master.
You can now test locally that file changes in your branch will trigger the correct tests.
The logic is:
1. If `package.json` has been edited, or if test utility files have been edited, or if example utility files have been edited, then all regression tests will be run in the CI.
2. Otherwise, if an example is edited, then the regression tests for all of the examples contained in the same example directory are run. For example: if you edit one of the two checkbox examples, then both the checkbox-1 and checkbox-2 example regression tests will be run in the CI.
3. All test files that have been edited will be run in the CI.
michael-n-cooper pushed a commit that referenced this issue Aug 4, 2020
Generated by TRAVIS-CI 32f0774
10ec8a7 
Infrastructure: Run regression tests in parallel in TravisCI and improve test selection logic (pull #1465)

Fixes issues #1449 and 1450 by running regression tests in parallel and improving logic that determines which tests to run.

To run in parallel:
1. Updates `.travis.yml` to use the [Ava feature](https://github.com/avajs/ava#parallel-runs-in-ci) that allows tests in parallel across three VMs in the CI
2. Updates Ava version to include [this bug fix](avajs/ava#2461) -- the fix prevents a false error when Ava tries to parallelize less than three test files (it parallelizes by files, so one test file is not parallelized).
3. Updates the script that runs the regression tests in the CI to only the test files for the example or test file you are editing to take advantage of the parallelization option. Before, it filtered tests by using a regex on the name of the test. Now, it explicitly lists all the test FILES that should be run.

When running `bash regression-tests.sh` locally from a branch that is not master, the script will now run as if the current branch is a pull request against master.
You can now test locally that file changes in your branch will trigger the correct tests.
The logic is:
1. If `package.json` has been edited, or if test utility files have been edited, or if example utility files have been edited, then all regression tests will be run in the CI.
2. Otherwise, if an example is edited, then the regression tests for all of the examples contained in the same example directory are run. For example: if you edit one of the two checkbox examples, then both the checkbox-1 and checkbox-2 example regression tests will be run in the CI.
3. All test files that have been edited will be run in the CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spectranaut spectranaut

@a11ydoer a11ydoer

Labels
Infrastructure Related to maintaining task force and repo operations, processes, systems, documentation regression-testing Related to AVA regression tests of example pages or AVA framework implementation within repo
Projects
None yet
Milestone
No milestone
3 participants
@spectranaut @mcking65 @a11ydoer