Enable pytest and bazel tests.

[gabrieldemarmiesse]

Used the idea of Jason and @Squadrick to make bazel run pytest.

It's possible to lauch the tests with the pytest command and with the bazel command. In the build, tests are all started by pytest but I made a single new build (py35, linux) to test with bazel and ensure we're still compatible.

Some goodies:

The 25 slowest tests are printed in the CI logs

It's just an option from the command line; --durations=25.

The code coverage is printed in the CI logs

Thanks to the --cov=tensorflow_addons command line option (needs pip install pytest-cov).
Because we're now people who like fancy things, we can even have a beautiful HTML report by just adding an option from the command line: --cov-report html.

We can use assert all the time.

No need to remember specific ones. Note that this is only available if at the bottom of the file, sys.exit(pytest.main([__file__])) is used instead of tf.test.main(). This is possible on all files, and we should enable it progressively by changing the last line. One example is in this pull request: see tensorflow_addons/seq2seq/basic_decoder_test.py.

We don't have to subclass tf.test.TestCase, but we can if we need self.

See tensorflow_addons/register_test.py where simple functions are used.

We can run individual methods instead of the whole file.

pytest ./tensorflow_addons/layers/wrappers_test.py::WeightNormalizationTest::test_removal_Conv2D

Automatic post mortem debugging

--pdb start the interactive Python debugger on errors or KeyboardInterrupt (useful when tests are stuck).

Some baddies

We still need to remember to declare the test in the BUILD file

Otherwise, pytest ./tensorflow_addons will run it, but not bazel test.

We still need to remember to use the boilerplate at the bottom of the test file

Otherwise, pytest ./tensorflow_addons will run it, but not bazel test.
We should use sys.exit(pytest.main([__file__])) from now on (bazel will run the file with pytest when this is written, instead of unittest).

Pytest prevents us from using certain functions.

On Windows, for some unknown reason, tf.compat.v1.test.get_temp_dir() crash with pytest.
Black magic 😨

If this pull request is too big, I can extract some smaller pull requests from it to reduce the size.

And thanks everyone for the great feedback in the meeting and in github!

[bot-of-gabrieldemarmiesse]

@rahulunair @pkan2 @RaphaelMeudec

You are owners of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

[seanpmorgan]

Thanks very much! Excited to unify the test setup

[seanpmorgan]

I would think this is an issue. addons_cpu will compile the custom-ops without any particular toolchain:
https://github.com/tensorflow/addons/blob/master/tools/ci_testing/addons_cpu.sh#L60-L64

I thought the compile at the subsequent build would be skipped since they're already created, but they won't be compatible with tf from pypi. However the test-release-wheel is passing, so I wanted to verify with you that this isn't reverting to python_op fallback?

[seanpmorgan]

Could we add a bazel clean --expunge after this just for sanity.

[gabrieldemarmiesse]

Sure thing. I'll make sure the right toolchaine is used.

[seanpmorgan]

LGTM thanks for all your work!

[seanpmorgan]

@gabrieldemarmiesse So this is failing only for py37 since the push:
https://github.com/tensorflow/addons/runs/495036979

ModuleNotFoundError: No module named '_sqlite3' Does that make any sense to you?

[gabrieldemarmiesse]

The problem seems to come from the coverage module having problems with the custom_ops docker image.

You can reproduce with

docker run -it --rm tensorflow/tensorflow:2.1.0-custom-op-gpu-ubuntu16 bash
python3.7 -m pip install pytest~=5.3 pytest-xdist~=1.31 pytest-cov~=2.8
python3.7 -c "import coverage"

but in a normal python docker image:

docker run -it --rm python:3.7 bash
pip install pytest~=5.3 pytest-xdist~=1.31 pytest-cov~=2.8
python -c "import coverage"

The error doesn't happen.

So there is a problem with sqllite in the custom_ops docker image. The quickest fix right now is to remove code coverage from the CI. People can still use it locally. That's until we find out how to fix this.

[rainwoodman]

Does the pytest support allow declaring tests with pytest.marks.parameterized?

A lot of our tests use nested loops for parametrization can probably read cleaner converted to parameterized.