Always use unittest for collecting tests in regrtests

[serhiy-storchaka]

BPO	45229
Nosy	@terryjreedy, @ambv, @ambv, @zware, @serhiy-storchaka, @pablogsal, @miss-islington
PRs	bpo-45229: Remove test_main in many tests #28405 bpo-45229: Fix setUpModule in test_ssl #28454 [3.10] bpo-45229: Remove test_main in many tests (GH-28405) #28455 [3.9] bpo-45229: Remove test_main in many tests (GH-28405) #28456 bpo-45229: Make pickle tests discoverable #28467 bpo-45229: Use doctest.DocTestSuite instead of run_doctest #28468 [3.10] bpo-45229: Make pickle tests discoverable (GH-28467) #28478 [3.9] bpo-45229: Make pickle tests discoverable (GH-28467) #28479 bpo-45229: Make datetime tests discoverable #28615 bpo-45229: Make tkinter tests discoverable #28637 [3.10] bpo-45229: Make datetime tests discoverable (GH-28615) #28639 [3.9] bpo-45229: Make datetime tests discoverable (GH-28615). #28645 bpo-45229: Revert "Make datetime tests discoverable" #28650 [3.9] Revert "[3.9] bpo-45229: Make datetime tests discoverable (GH-28615) (GH-28645)" #28660 [3.9] Revert "[3.9] Revert "[3.9] bpo-45229: Make datetime tests discoverable (GH-28615) (GH-28645)"" #28666 Revert "bpo-45229: Revert "Make datetime tests discoverable"" #28667 [3.10] bpo-45229: Make datetime tests discoverable (GH-28615) #28668 [3.10] bpo-45229: Make tkinter tests discoverable (GH-28637) #28927 [3.9] bpo-45229: Make tkinter tests discoverable (GH-28637) #28929 bpo-45229: Make doctest tests discoverable #28986 bpo-45229: Make test_http_cookiejar discoverable #29004 [3.10] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) #29006 [3.9] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) #29007 [3.10] bpo-45229: Make doctest tests discoverable (GH-28986) #29095 [3.9] bpo-45229: Make doctest tests discoverable (GH-28986) #29096
Dependencies	bpo-40058: Running test_datetime twice fails with: module 'datetime' has no attribute '_divide_and_round' bpo-40173: test.support.import_fresh_module fails to correctly block submodules when fresh is specified bpo-45185: test.test_ssl.TestEnumerations is not run bpo-45187: Some tests in test_socket are not run bpo-45500: Rewrite test_dbm bpo-45502: Fix test_shelve and make it discoverable

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2021-09-17.07:31:12.126>
labels = ['type-feature', 'tests', '3.9', '3.10', '3.11']
title = 'Always use unittest for collecting tests in regrtests'
updated_at = <Date 2021-10-20.16:52:45.847>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-10-20.16:52:45.847>
actor = 'lukasz.langa'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2021-09-17.07:31:12.126>
creator = 'serhiy.storchaka'
dependencies = ['40058', '40173', '45185', '45187', '45500', '45502']
files = []
hgrepos = []
issue_num = 45229
keywords = ['patch']
message_count = 33.0
messages = ['402006', '402009', '402053', '402076', '402154', '402156', '402168', '402195', '402214', '402228', '402245', '402254', '402255', '402903', '402945', '402952', '402953', '402956', '402959', '402974', '402982', '402996', '402999', '403000', '403846', '403853', '403855', '404137', '404142', '404143', '404466', '404495', '404496']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'lukasz.langa', 'lukasz.langa', 'zach.ware', 'serhiy.storchaka', 'pablogsal', 'miss-islington', 'jerrylikerice1s']
pr_nums = ['28405', '28454', '28455', '28456', '28467', '28468', '28478', '28479', '28615', '28637', '28639', '28645', '28650', '28660', '28666', '28667', '28668', '28927', '28929', '28986', '29004', '29006', '29007', '29095', '29096']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45229'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-89392: Make test_decimal discoverable #106209
• [3.11] gh-89392: Make test_decimal discoverable (GH-106209) #106229
• [3.12] gh-89392: Make test_decimal discoverable (GH-106209) #106230
• bpo-45229: Make ElementTree tests discoverable #108859
• gh-89392: Remove test_main() in test_netrc #108860
• gh-89392: Make test_pep646_syntax discoverable #108861
• [3.11] gh-89392: Remove test_main() in test_netrc (GH-108860) #108868
• [3.11] gh-89392: Make test_pep646_syntax discoverable (GH-108861) #108869
• [3.12] gh-89392: Make test_pep646_syntax discoverable (GH-108861) #108871
• [3.12] gh-89392: Remove test_main() in test_netrc (GH-108860) #108872
• [3.12] bpo-45229: Make ElementTree tests discoverable (GH-108859) #108873
• [3.11] bpo-45229: Make ElementTree tests discoverable (GH-108859) #108874
• gh-89392: Fix running test_pep646_syntax as script #108875
• gh-89392: Remove support of test_main() in libregrtest #108876
• [3.11] gh-89392: Fix running test_pep646_syntax as script (GH-108875) #108877
• [3.12] gh-89392: Remove support of test_main() in libregrtest (GH-108876) #108897
• [3.11] gh-89392: Remove support of test_main() in libregrtest (GH-108876) #108898
• gh-89392: Use normal unittest runner in test_type_cache #108911
• [3.12] gh-89392: Use normal unittest runner in test_type_cache (GH-108911) #108913
• [3.11] gh-89392: Use normal unittest runner in test_type_cache (GH-108911) #108914
• gh-89392: Use unittest test runner for doctests in test_getopt #108916
• [3.12] gh-89392: Use unittest test runner for doctests in test_getopt (GH-108916) #108919
• [3.11] gh-89392: Use unittest test runner for doctests in test_getopt (GH-108916) #108920

[serhiy-storchaka]

Currently regrtest supports two ways of collecting and running tests in module.

1. If the module contains the "test_main" function, regrtest just calls it. This function usually calls run_unittest() with a list of test classes, composed manually, it can also run doctests with run_doctest(), generate list of test classes dynamically, and run some code before and after running tests.

2. Otherwise regrtest uses unittest for for collecting tests.

The disadvantage of the former way is that new test classes should be added manually to the list. If you forget to do this, new tests will not be run. See for example bpo-45185 and bpo-45187. Not runned tests can hide bugs.

So it would be better to eliminate human factor and detect tests automatically.

[serhiy-storchaka]

PR 28405 contains simple changes for 60 test files. It does not cover files which use complex code for generating list of test classes and running doctests (because there are subtle differences between running doctests with run_doctest() and via DocTestSuite). They are left for the following PRs.

[zware]

See also bpo-16748

[serhiy-storchaka]

Thank you Zachary, I almost forgot about that issue.

The goal of this issue is to make regrtests using test discovery instead of manually composed lists of test classes. Of course it requires that tests should be discoverable. And most of them already are, thanks to bpo-16748 and similar.

[serhiy-storchaka]

New changeset 40348ac by Serhiy Storchaka in branch 'main':
bpo-45229: Remove test_main in many tests (GH-28405)
40348ac

[serhiy-storchaka]

New changeset a624177 by Serhiy Storchaka in branch 'main':
bpo-45229: Fix setUpModule in test_ssl (GH-28454)
a624177

[serhiy-storchaka]

New changeset bedce35 by Serhiy Storchaka in branch '3.10':
[3.10] bpo-45229: Remove test_main in many tests (GH-28405) (GH-28455)
bedce35

[serhiy-storchaka]

New changeset 5822ab6 by Serhiy Storchaka in branch '3.9':
[3.9] bpo-45229: Remove test_main in many tests (GH-28405) (GH-28456)
5822ab6

[serhiy-storchaka]

New changeset a856364 by Serhiy Storchaka in branch 'main':
bpo-45229: Use doctest.DocTestSuite instead of run_doctest (GH-28468)
a856364

[terryjreedy]

Thank you for doing this. When I started setting up the IDLE test suite 8 years ago, I was a bit confused about the mixture of old ways (test_main) and new ways (unittest.main and possibly load_tests) in the test suite. Fortunately, Nick C. told me which was which. Updating the test suite should make it better as an example for new unittest users.

[serhiy-storchaka]

New changeset e6ba992 by Serhiy Storchaka in branch 'main':
bpo-45229: Make pickle tests discoverable (GH-28467)
e6ba992

[ambv]

New changeset 74c6acc by Miss Islington (bot) in branch '3.10':
bpo-45229: Make pickle tests discoverable (GH-28467) (GH-28478)
74c6acc

[ambv]

New changeset 04dea46 by Miss Islington (bot) in branch '3.9':
bpo-45229: Make pickle tests discoverable (GH-28467) (GH-28479)
04dea46

[serhiy-storchaka]

New changeset d441437 by Serhiy Storchaka in branch 'main':
bpo-45229: Make datetime tests discoverable (GH-28615)
d441437

[pablogsal]

Unfortunately, this PR28615 seems to have broken a bunch of buildbots. For instance:

https://buildbot.python.org/all/#/builders/75/builds/172/steps/5/logs/stdio

test_divide_and_round (test.datetimetester.TestModule_Pure) ... ok
test_divide_and_round (test.datetimetester.TestModule_Fast) ... skipped 'Only run for Pure Python implementation'
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=1)
test_divide_and_round (test.datetimetester.TestModule_Pure) ... ERROR
.test test_datetime failed
test_divide_and_round (test.datetimetester.TestModule_Fast) ... skipped 'Only run for Pure Python implementation'
======================================================================
ERROR: test_divide_and_round (test.datetimetester.TestModule_Pure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/datetimetester.py", line 88, in test_divide_and_round
    dar = datetime_module._divide_and_round
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'datetime' has no attribute '_divide_and_round'
----------------------------------------------------------------------
Ran 2 tests in 0.003s
FAILED (errors=1, skipped=1)
1 test failed again:
    test_datetime
== Tests result: FAILURE then FAILURE ==

[serhiy-storchaka]

Seems this is not a new bug:

$ ./python -m test test_datetime test_datetime -m test_divide_and_round
0:00:00 load avg: 2.36 Run tests sequentially
0:00:00 load avg: 2.36 [1/2] test_datetime
0:00:00 load avg: 2.36 [2/2] test_datetime
test test_datetime failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.9/Lib/test/datetimetester.py", line 88, in test_divide_and_round
    dar = datetime_module._divide_and_round
AttributeError: module 'datetime' has no attribute '_divide_and_round'

test_datetime failed (1 error)

== Tests result: FAILURE ==

But for some causes it was not reproduced with refleak tests.

[pablogsal]

Seems this is not a new bug:

I agree, is unfortunate. But sadly is breaking all buildbots and therefore is hiding other issues and making the 'test-with-buildbots' label not useful. I would prefer to revert until we have a fix (and not land the backports until then).

[serhiy-storchaka]

See bpo-40058 and bpo-40173.

[serhiy-storchaka]

Finally it has been finished.