Python: default test runner fails often because it imports all modules

[FRidh]

Since we upgraded setuptools it changed behavior as to how python setup.py test functions.
Before, it would often not do anything, silently passing the checkPhase.

After the upgrade, it became more aggressive, and it now appears it tries to find tests by importing all modules. When importing a module exceptions can occur, but these are not catched by setuptools, resulting straight away in a failing checkPhase.

Example:

Traceback (most recent call last):
  File "nix_run_setup.py", line 6, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 173, in <module>
    **kwargs
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 159, in run
  File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 140, in with_project_on_sys_path
  File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 180, in run_tests
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/main.py", line 93, in __init__
    self.parseArgs(argv)
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/main.py", line 123, in parseArgs
    self._do_discovery([])
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/main.py", line 228, in _do_discovery
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 341, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 398, in _find_tests
    full_path, pattern, namespace)
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 475, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 38, in loadTestsFromModule
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 190, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/8qi91pnjxk49s3nzgs2y4xqnrwpraxsl-python3.5m-setuptools-19.4/lib/python3.5/site-packages/setuptools-19.4-py3.5.egg/setuptools/command/test.py", line 38, in loadTestsFromModule
  File "/nix/store/ln2cn6g6rf116vaq1fgngaqzpzfq495x-python3-3.5.1/lib/python3.5/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/tmp/nix-build-python3.5-tornado-4.2.1.drv-0/tornado-4.2.1/tornado/platform/kqueue.py", line 23, in <module>
    assert hasattr(select, 'kqueue'), 'kqueue not supported'
AssertionError: kqueue not supported
builder for ‘/nix/store/s32ylw2jql4fbphb1fyzlgrldmn5bmig-python3.5-tornado-4.2.1.drv’ failed with exit code 1
error: build of ‘/nix/store/s32ylw2jql4fbphb1fyzlgrldmn5bmig-python3.5-tornado-4.2.1.drv’ failed

[FRidh]

See pypa/setuptools#446
The change was introduced in setuptools 18.4

[Profpatsch]

triage: is this still a problem?

[FRidh]

I have not seen this anymore, closing.