Add code coverage and integrate with Coveralls

[djkirkham]

Coveralls (http://coveralls.io) is a utility that can be integrated with a GitHub repository to display a code coverage percentage. It can be used with a variety of languages, but for Python it uses a utility called Coverage (https://coverage.readthedocs.io/en/coverage-4.2), which you can run in a terminal to produce text based reports.

This PR gets coverage/coveralls working on Iris via Travis. It's based on some initial work that @marqh did. Since I encountered a lot of obstacles trying to get it to work, and since I'm leaving for two weeks, I'll document the steps I needed to take.

Firstly, since the Iris tests run in parallel using Nose, it's necessary to set up Coverage to run on all the sub-processes. Coverage needs to be able run some set-up code for every process, and one way the docs suggest to do this is to add code to sitecustomize.py, which is run whenever the Python interpreter starts up. It's also necessary to set an environment variable COVERAGE_PROCESS_START with an absolute path to the Coverage configuration file to use. When calling the coverage command, it's necessary to specify both -p (parallel) and --concurrency=LIB, where LIB is replaced with the concurrency library used (multiprocessing for nose).

When running in parallel mode, Coverage produces a separate output file for each processor; these need to be combined using coverage combine before generating a report.

By default Coverage won't report on any libraries installed in the Python library directory. Additionally, for the web utility to be able to match the covered files to those in the GitHub repository, the files need to be run from where they are when Travis clones the repo. While it's not strictly necessary to be able to match the files, if they can't be matched one won't be able to inspect individual files at http://coveralls.io, which is quite useful. So, rather than running setup.py to install the files elsewhere and calling iris.tests.runner, I've changed the Travis script to run setup.py test.

I've added a new configuration to Travis that runs coverage with the default tests, rather than modifying the existing default configuration. This may or may not be necessary. I'm happy for it to be changed if not.

I've also restricted the version of cf_units to <1.1.1 (I tried making it =1.1.0, but it didn't work for some reason), because currently the Iris tests are failing with 1.1.2 (#2144 will fix them). Obviously the version should be unrestricted when merging.

Although the first commit shows @marqh as author, a lot of the changes in it were made by me. I only mention this so that people don't ask him questions about it expecting him to know the answer.

I've made sure certain files are omitted by Coverage when recording data (e.g. those in lib/iris/tests). Some additions could probably be made to these omissions.

[djkirkham]

Currently coverage will only run if TEST_TARGET is default, even if you set COVERAGE=true for other targets.

[djkirkham]

Annoyingly, since the code takes longer to run under coverage, it fails due to timeouts even more than it was before.

[coveralls]

Changes Unknown when pulling 69ad920 on djkirkham:coveralls into * on SciTools:master*.

[QuLogic]

All this setup stuff sounds pretty complicated. Is there a way we could enable the nose coverage plugin instead, which I'd hope takes care of all that for us?

[QuLogic]

By default Coverage won't report on any libraries installed in the Python library directory.

Adding something like /home/travis/virtualenv/python*/lib/python*/site-packages/iris/ to the sources should get that to work. You still need the coverage combine to flatten the paths as well.

[QuLogic]

You might also want to try out codecov.io; it might be a bit smarter about file paths. It's also got some sort of browser extension that allows you to see code coverage directly on GitHub, though I've not used it.

[djkirkham]

All this setup stuff sounds pretty complicated. Is there a way we could enable the nose coverage plugin instead, which I'd hope takes care of all that for us?

Initially I didn't go down that route because I didn't want to modify the Iris library code (Nose is called in lib/iris/tests/runner/_runner.py), but I now see that you can enable the behaviour by setting an environment variable, so maybe that's the way to go.

You might also want to try out codecov.io

That does look good, thanks!

I'm not able to work on this right now so if someone else wants to take over be my guest.

[pelson]

Closing for now. I'm 👍 on adding coverage, and happy for this PR to be re-opened if you can get this back up and running @djkirkham.