Parallel tests (UNIX)

[giampaolo]

OK, this is big. We are now able to run tests in parallel on UNIX (and we keep using unittest module). On my Linux box with 8 logical cores all 551 unit-tests complete in half the time: around 6 seconds instead of 12. Same on FreeBSD. This is great news also because hopefully the CI builds should complete sooner. Currently we have 10 builds running on each commit:

• Travis (8 builds: 5 Linux, 2 OSX, 1 for PYPY on Linux)
• Cirrus-CI (2 builds for FreeBSD)

We now depend on the super cute concurrencytest lib which takes care of the internal fork/parallelization details.

The tricky part though was the fact that some tests needed to be run serially. In order to achieve that I added a @serialrun decorator which can be used to mark TestCase classes. On start, the test runner will split the test suite in 2 based on that, and finally run parallel and serial suites separately but as a single test run.

Prior to this I also had to refactor quite a lot of stuff and get rid of TESTFN global variable (#1734), since a global test file name doesn't play nice when accessed by multiple tests at the same time (and no, including os.getpid() as part of the file name wasn't enough).

We now have:

make test
make test-parallel

And here's the test output (an actual one):

$ make test-parallel
>>> starting parallel tests using 8 workers <<<
test_testutils.TestNetUtils.test_unix_socketpair (subunit.RemotedTestCase)
test_testutils.TestNetUtils.test_unix_socketpair ... OK
test_testutils.TestRetryDecorator.test_retries_arg (subunit.RemotedTestCase)
test_testutils.TestRetryDecorator.test_retries_arg ... OK
test_system.TestCpuAPIs.test_cpu_count_logical (subunit.RemotedTestCase)
test_system.TestCpuAPIs.test_cpu_count_logical ... OK
test_system.TestCpuAPIs.test_cpu_times_percent (subunit.RemotedTestCase)
test_system.TestCpuAPIs.test_cpu_times_percent ... OK
[...]
----------------------------------------------------------------------
Ran 510 tests in 3.683s

FAILED (failures=1, skipped=146)

>>> starting serial tests <<<
psutil.tests.test_unicode.TestFSAPIsWithInvalidPath.test_proc_cwd ... OK
psutil.tests.test_unicode.TestFSAPIsWithInvalidPath.test_proc_exe ... OK
psutil.tests.test_unicode.TestFSAPIsWithInvalidPath.test_proc_name ... OK
psutil.tests.test_unicode.TestFSAPIsWithInvalidPath.test_proc_open_files ... OK
[...]
----------------------------------------------------------------------
Ran 41 tests in 2.790s

OK
----------------------------------------------------------------------

+----------+----------+----------+----------+----------+----------+
|          |    total | failures |   errors |  skipped |     time |
+----------+----------+----------+----------+----------+----------+
| parallel |      510 |        1 |        0 |      146 |    3.68s |
+----------+----------+----------+----------+----------+----------+
| serial   |       41 |        0 |        0 |        0 |    2.79s |
+----------+----------+----------+----------+----------+----------+

Ran 551 tests in 6.474s using 8 workers

[giampaolo]

It turns out on CI we save just a bunch of secs but there are occasional false positives.
Since the majority of the time is spent waiting for the CI to start, install etc, it's not worth the instability so CI services will run the test serially (done in #1741 where I added parallelization to build/install).
While developing though, parallel tests are still a game changer.