py3: make all tests pass

[fchapoton]

This ticket is used to keep track of the progress towards python3.

As this ticket is too long, final steps are moved to #28298

release	files having failing doctests	more than 100	from 10 to 100	from 1 to 10	failing doctests
8.4.b3	915	42	251	622	17972
8.4.b4	803	23	175	605	10921
8.4.b5	657	17	145	495	8509
8.4.b6	594	17	127	450	7706
8.4.b7	587	17	128	442	7678
8.4.rc0	524	16	119	389	6976
8.5.b0	461	6	94	361	4206
8.5.b1	414	3	76	335	3206
8.5.b2	370	1	66	303	2465
8.5.b3	339	1	62	276	2245
8.5.b4	312	0	58	254	1947
8.5.b5	295	0	51	244	1741
8.5.b6	264	0	43	221	1514
8.5.rc0	252	0	39	213	1393
8.6.b0	234	0	30	204	1163
8.6.b1	231	0	30	201	1146
8.6.rc0	232	0	28	204	1124
8.6.rc1	236	0	30	206	1136
8.7.b0	240	1	38	201	1605
8.7.b1	200	0	23	177	866
8.7.b2	198	0	24	174	863
8.7.b3	166	0	20	146	717
8.7.b4	165	0	19	146	703
8.7.b5	143	0	12	131	562
8.7.b6	137	0	8	129	464
8.7.b7	122	0	8	114	447
8.7.rc0	105	0	7	98	384
8.8.b0	102	0	7	95	370

Now with python 3.7.3:

release	files having failing doctests	more than 100	from 10 to 100	from 1 to 10	failing doctests
8.8.b1	105	0	6	99	382
8.8.b2	94	0	6	88	324
8.8.b3	82	0	6	76	282
8.8.b4	77	0	2	75	246
8.8.b5	56	0	2	54	173
8.8.b6	49	0	1	48	151
8.8.b7	44	0	1	43	136
8.8.rc0	32	0	0	32	92
8.9.b0	22	0	0	22	71
8.9.b2	18	0	0	18	58
8.9.b3	15	0	0	15	47
8.9.b4	12	0	0	12	33
8.9.b5	8	0	0	8	18
8.9.b7	5	0	0	5	15
8.9.b9	4	0	0	4	14
8.9.rc0	3	0	0	3	7

Folders where all tests pass

• algebras py3: doctests for commutative_dga.py #26316 py3: fix doctest in yangian #26399 Improve comparisons of PolyDicts #26279 py3: fix doctests in Weyl algebra #26646 py3: some fixes in quaternion algebras #26688 py3: some fixes for algebras #26766 Fix some py3 doctests in algebras #27414 py3: algebras/lie_algebras #27415 (final)
• algebras/steenrod ✔ Python 3 tests for algebras/steenrod #26219 and Fix CoercionException imports #26220
• arith py3: fix doctests in arith/ #26208 ✔
• calculus py3: some fixes for calculus/ #26751 py3: one tiny fix for isidentifier #26988 (final)
• categories Py3: Many python3 fixes in categories #26545 py3: fix noncrossing partitions lattices #26780 py3 compatible test for the category of finite posets #27349 py3: some care for categories #27541 py3: fix one doctest in semigroups category #27732 (remains 8) py3: remaining fixes in categories #27757 (final?)
• coding Updates to Huffman codings #26026 py3: fix last 2 doctests in coding #26765 (final)
• combinat/cluster_algebra_quiver py3: some care for cluster seeds #27517 Port cluster_algebra_seed to py3 #27854 (2 files and 10 tests)
• combinat/crystals Py3: Fix combinat.crystals.kac_modules.py for python3 #27945 Py3: Fix crystals.kirillov_reshetikhin for python3 #27977 (final)
• combinat/designs py3: fix some doctests in combinat/designs #26430, py3: Fix combinat.designs module for python3 (final) #26837 (regression)
• combinat/finite_state_machine py3: replace tests for attribute iteritems #26147, Py3: Some fixes in combinat/finite_state_machine.py #26468, py3: some work on finite state machines #26737, py3: partial cure for finite state machines #27667 py3: finite state machines #27679
• combinat/posets py3 fix last 4 failing doctests in combinat/posets #26998 (final)
• combinat/rigged_configurations py3: Make doctest pass in combinat/rigged_configurations module #26805
• combinat/sf (Fix CoercionException imports #26220 +) py3: fix doctests in combinat/sf/ #26215
• combinat/species py3: tiny fix in species #26328, py3: fix most doctests in combinat/species #26610, py3: final fix for species #27072 (final)
• combinat/subset Py3: Fix combinat.subset.py for python3 #26510 py3: fix repr and iter for submultisets #27727
• combinat/words py3: some care for islice in words #26362, Py3: Fix DeprecationWarning: generator X raised StopIteration for combinat/words #26458, py3: fix doctests failures of sage/combinat/words/word_*datatype* #26473, py3: all tests pass in combinat/words #26590
• cpython py3: fixes for doctests in cpython #26855, py3: fix one doctest in cpython folder #27964, Misc pip upgrades again #27886
• crypto py3: fix doctest in sboxes #26398 py3: fix last doctests in crypto #27580 (final)
• databases Py3: Fix database/sql_db module for python3 #26702 Py3 : database.oeis implement __getitem__ to replace __getslice__ #26704
• data_structures Fix bytes/str in sage.data_structures.bitset #24852 py3: fix last doctest in mutable_poset #27433 Py3: Fix bounded_integer_sequences.pyx for python3. #27439 (final)
• docs ✔ py3: fix doctests in docs/ #26310
• doc/en py3: a bit of work in src/doc #26346 py3: fix one doctest in doc/en #27698 py3: remaining fixes in doc/en #27758 (final)
• dynamics py3: fix last doctest in soliton.py #27453 remove dynamics/interval_exchanges and dynamics/flat_surfaces #26982
• dynamics/arithmetic_dynamics Py3: Fix dynamics.arithmetic_dynamics for python3 #27779 (remains 1)
• ext some py3 fixes in ext #27776 py3: fix last doctest in ext folder #27975
• features ✔
• functions py3: minor fixes to sage.functions #24758, py3: log(int(7), base=2) returns 1 #26591 remove the deprecated piecewise_old module #26865 Patch pynac with PR: 336 #26770 py3: pynac bug with logb on int #25979 py3: fix comparison of constants #27812
• finance py3: minor cleanup in sage.finance.stock #26213 ✔
• games ✔
• game_theory py3: fix some doctests in game_theory #26429 py3: fix all doctests in game_theory #26738 (final)
• geometry Py3: Some python3 fixes in geometry module. #26726 py3: fix some doctests in ppl lattice polytopes #27772 py3: fix the last doctest in geometry #27965,py3: polyhedron folder with optional packages #28235
• graphs py3: not sorting vertices on some graph functions #26431 py3: do not sort vertices in static graphs #26447 py3 work on automorphism groups of graphs #26527 Meta-ticket: make graphs compatible with Python 3 #26640 py3: some fixes in graph generators #26645 py3: fix sparse graph isomorphism #26846 py3: fix some doctests in graph.py #27125 is_isomorphic broken with keyword edge_labels=True #27232 py3: failing doctest in graph_database.py with interactive_query #27435
• groups py3: some care for abelian groups #26799, py3: fix remaining doctests for groups/perm_gps #27830 (final)
• groups/lie_gps py3: fix for zip in nilpotent Lie groups #26518
• groups/matrix_gps py3 fix all doctests in groups/matrix_gps #26400 (regression) py3: fix again doctests in matrix groups #27459 (final again)
• homology py3: fundamental group of simplicial complexes #27452 (final)
• interacts Doctest Jupyter implementation of interact library #27117, py3 fixes in interacts #27157
• interfaces py3: Fix interfaces module for python3 #26314 py3: fix last doctest in interfaces #27904 (1 file and 1 test)
• knots ✔
• libs/gsl
• lfunctions ✔
• logic py3: fix doctests in logic/ #26214 ✔
• manifolds py3: make all tests in manifolds passed #26475
• matrix/ py3: some care for matrix/ #26236 py3: some small care for matrix #26716, py3: fixes in matrices #27585, py3: use of "round" in matrix_double_dense.pyx #27615 (final)
• matroids py3: some care for matroids #26767, Py3: Fix graphic_matroids.py for python3 #27378, py3: matroid/unpickling.pyx fix #27771, py3: matroids.utilities.lift_cross_ratios #27787, py3 fixes for matroids #27889 (final)
• media ✔
• modular py3: sorting some doctests in AnalyticType #26382, py3: fix hypergeometric motives #26611, py3: some work on modular/modform_hecketriangle/ #26967 sparse_2term_quotient_only_pm1 gives different results depending on set iteration order #25301 py3: some fixes in modular/ #27145 py3: last fix in modular folder #27313 (final)
• modular/modsym py3: make some doctests of modular symbols more robust #26673 py3: some fixes in Heilbronn matrices #27177
• modules py3: misc fixes for sage.modules #26276 py3: some tentative fixes for multi-filtered modules #27115, py3: modules/with_basis/morphism.py #27466 (final)
• monoids ✔ py3: fix doctests in monoids/ #26294
• parallel ✔
• plot py3: fix some doctests in plot/ #26210 py3: use repr for floats in plot3d file format outputs #26299 Python 3 bug in zipfile #26301 py3: hodgepodge of (mostly) trivial doctest fixes #26321, py3: Fix plot module for python3 #27126 (final)
• probability py3: fix doctests in probability/ #26216 ✔
• quadratic_forms/ py3: fix doctests in quadratic/ #26224 ✔
• quivers py3: fix doctests in quivers/ #26211 ✔
• rings Py3 : Fix py3 errors in sage.rings.tate_algebra* files #26697 py3: one fix in invariant_theory #27048 py3: fixing the hash of real and complex lazy fields #27440 py3: fix hash doctests in real and complex fields #27650 py3: fix regression in qqbar.py #27767 py3: Fix hash function of Integer #28112 py3 failure in rings/morphism.pyx #28139
• rings/asymptotic py3: doctests AsymptoticRing #26146
• rings/number_field py3: minor fixes for sage.rings.number_field #26263, py3: one fix in bdd_height.py #26619
• rings/function_field py3: fix function fields #26721 py3: fix doctests in rings/function_field (final) #26764 (final)
• rings/padics Py3 doctests in padic_lattice_element.py #27612 (final)
• rings/polynomial py3: cleanup of toy d basis #26670 py3: one fix in infinite polynomial ring #26760 py3: cleanup the groebner_fan file #26909 py3: Stop to use sort on KeyConvertingDict object #26792 py3: some fixes to rings/polynomial #27674 py3: fix a flaky doctest in multipoly element #27768 Py3: Minor enhancements in rings.polynomial.multi_polynomial.pyx #27791 py3: fix doctests in multi_polynomial_ideal and toy_buchberger #27908 py3: fix a doctest in binary_form_reduce #27909 py3: hash collisions of Laurent polynomials #27914
• rings/valuation py3: last fix in valuation #27868
• rings (power series) py3: fix doctests in power series #26539
• sandpiles py3: miscellaneous fixes and cleanup to sage.sandpiles #26016 py3: some fixes for sandpiles tutorial #26628 py3: work on the sandpile problem #26722
• sat py3: fix doctests in sat/ #26543 ✔
• schemes py3: some fix in Riemann surfaces #27049 py3: fix all doctests in schemes/product_projective #27449 py3:fix last doctest in affine_curve.py #27471 py3: various fixes #27619 (final)
• schemes/toric py3: fix tensor operations for Klyachko bundles #26664
• schemes/curves
• schemes/product_projective
• schemes/elliptic_curves py3: fix doctest in the file kraus.py #26536, py3 fix elliptic curve L-series #26529, py3: fix some doctests in elliptic curves #26570, py3: fix round in sha_tate #26594 py3: fix sorting of isogeny classes of elliptic curves #26913 py3: fixing doctests in elliptic_curves/cm.py #26914 py3: fix last doctest in ell_rational_field #27432 py3: some care for reraise #27520 py3: add __reduce__ to eclib modular symbol #27539
• server ✔
• sets py3: Fix some doctests errors in sets module for python3 #26659 py3: fix one file in sets/ #27456 py3: doctests in sets #27794 (final)
• stats py3: fix doctests in stats #26401 ✔
• structure py3: miscellaneous test fixes in sage.structure #25694 py3: caring for structure/ doctests #26768
• tensor py3: fix doctests in tensor/ #26295, py3: make all tests in src/sage/tensor passed #26601
• typeset ✔
• sage_setup py3: fix all tests in sage_setup #27075, py3: fix artificial exception test in sphinxbuild.py #27073

Fragile folders (still fails on the reference machine)

These doctests pass when run on each file alone.

• repl py3: make doctest pass in repl module #26881

┗ /attach.py  # 4

• numerical py3: numerous string fixes to sage.numerical.backends #24740, py3: numerous additional fixes for sage.numerical #24741, py3 minor fixes in numerical #26631, py3: fix round issue with cplex and gurobi #27773 and py3: fix various issues in cplex backend #27774 py3: some tiny fixes in numerical folder #27342 py3: more fixes in numerical #27614

┗ backends/generic_backend.pyx  # 2

• libs python3: 'filter' object is not subscriptable in libs/gap/util.pyx #26665 py3: fix sage.libs.readline.pyx for python3 #27275 py3: fix output buffering issues in eclib #27383 upgrade eclib to v20190226 #27360 py3: libs/ppl.pyx fixes #27465 Py3: Fix libs.ntl.ntl_ZZ_pX.pyx doctests. #27588 Py3 : Fix failures in src.sage.symbolic.constants #27630

┗ singular/function.pyx  # 1
┗ eclib/interface.py # 1 (line 597)

• FLINT issue in src/sage/rings/polynomial/polynomial_rational_flint.pyx

Folders with work in progress

• combinat/root_system Py3: Fixes in combinat/root_system module #26649 py3 : Fix combinat/root_system/integrable_representations.py for python3. #27654 Py3: Fix RecursivelyEnumeratedSet BFS for python3. #27967 Py3: Fix combinat.root_system.root_lattice_realisations doctests #28167 py3 fixes for weyl_characters.py #28227

┣ non_symmetric_macdonald_polynomials.py  # 1
┣ weyl_characters.py  # 3
┗ weyl_group.py  # 7

• combinat py3: fix doctests in Hillman-Grassl file #26535 py3: fix doctests in latin squares #26671 py3: fix doctests in symmetric group algebra #26719 py3: more fixes for KR crystals #26778 Py3: Fix several combinat files for python3 #26781 py3: some fix in set_partition #26917 py3: care for Schnyder woods #26918 py3: some care for Hadamard matrices #27013 py3: combinat/designs: bytes-like object is required, not 'str' #26592 py3 Fixing combinat/tutorial.py #27710 Py3: Fix combinat.set_partition doctests #27777 Py3: Fix combinat.set_partition_ordered doctests #27930 Py3: Fix doctest in multiset_partition_into_sets_ordered #28079 py3 fix for combinat/finite_state.py #28177 py3 perfect_matching #28210

see above for the remaining issues

• doctest Fix some py3 doctests in src/sage/doctests #27775 py3: fix one doctest in doctest folder #27976 py3: fix doctests in doctest/test #28232

┗ test.py  # 2

• misc py3: misc minor fixes for sage.misc.rest_index_of_methods #26312, py3: some changes in misc #26390 py3: fix doctests in weak dict #26674 py3: some fixes for misc/abstract_method #26720 py3: fix pip_installed_packages #27001 py3: handle the explain_pickle problem #27065 A few small py3 fixes in sage.misc #27301 Py3: test_class_pickling.py fix method resolution order error #27390 py3: fix misc/unittest #27431 py3: care for two files in misc #27691 py3: fix src/sage/misc/nested_class.pyx #27692 py3: fix src/sage/misc/classcall_metaclass.pyx #27693 py3 failures in sage/misc/sageinspect.py and sagedoc.py #27971 py3 fix in misc/functional.py #28035 py3: misc/dev_tools.py #28138

┣ nested_class.pyx  # 9

• symbolic py3: fixes for isidentifier #27457 Py3: symbolic/expression.pyx fix some errors #27885 py3: progress on symbolic/expression.pyx #28140 add a doctest for pynac's rational hash function #28219

┗ expression.pyx  # 1 hash + 2 flaky in lines 11633 and 11635

• tests py3: fixing the hash of real and complex lazy fields #27440 py3: last test in tests/cmdline.py #28002 (see below for the remaining issues)
• tests/french_book Py3 : Fix doctests in module tests/french_book #26495 doctests for the english translation of the book "Calcul mathématique avec Sage" #23572 py3: fix one doctest in French book #27574 py3: fix doctests in one file of the French book : nonlinear_doctest.py #27651 py3: fix some doctests in French book (graph and programming) #27668 py3: fix repr and iter for submultisets #27727 py3: one more fix in French book #28067

┣ graphtheory_doctest.py  # 1
┗ sol/graphtheory_doctest.py  # 1

CC: @vinklein @embray @jm58660 @timokau

Component: python3

Issue created by migration from https://trac.sagemath.org/ticket/26212

[fchapoton]

comment:393

status on 8.9.beta4: remains 12 files with a total of 33 failing doctests:

'src/sage/combinat/root_system/non_symmetric_macdonald_polynomials.py  # 1 doctest failed',
 'src/sage/libs/singular/function.pyx  # 1 doctest failed',
 'src/sage/schemes/hyperelliptic_curves/hyperelliptic_finite_field.py  # 1 doctest failed',
 'src/sage/symbolic/expression.pyx  # 1 doctest failed',
 'src/sage/tests/books/computational-mathematics-with-sagemath/graphtheory_doctest.py  # 1 doctest failed',
 'src/sage/tests/books/computational-mathematics-with-sagemath/sol/graphtheory_doctest.py  # 1 doctest failed',
 'src/sage/doctest/test.py  # 2 doctests failed',
 'src/sage/numerical/backends/generic_backend.pyx  # 2 doctests failed',
 'src/sage/combinat/root_system/weyl_characters.py  # 3 doctests failed',
 'src/sage/repl/attach.py  # 4 doctests failed',
 'src/sage/combinat/root_system/weyl_group.py  # 7 doctests failed',
 'src/sage/misc/nested_class.pyx  # 9 doctests failed'

[dcoudert]

comment:394

Yes, the 8 last failing doctests in generic_graph.py have been fixed with #27232. So, all doctests in the standard graphs module are now fixed !!!

We still have some failing doctests in optional packages like plantri #28108. I have now yet checked all other optional packages. Do we have a meta-ticket to record them ?

[jhpalmieri]

comment:395

Is this a new one? All tests in schemes are supposed to pass.

'src/sage/schemes/hyperelliptic_curves/hyperelliptic_finite_field.py  # 1 doctest failed'

[fchapoton]

comment:396

I think I have seen that already in previous betas, maybe since 8.9.beta3 or before.

EDIT: it was already there in 8.9.b0 (see above)

[jhpalmieri]

comment:397

Is there a ticket for it? What's the error? (I don't see it on my OS X machines.)

[vinklein]

comment:398

I don't have it either on ubuntu 16.04.

[fchapoton]

comment:399

There is no ticket for this error in schemes. The exact error can be seen in the patchbot log:

https://patchbot.sagemath.org/log/0/Ubuntu/16.04/x86_64/4.4.0-148-generic/atlas/2019-07-29%2008:49:23?short

This patchtbot is running on an ubuntu 16.04, using gcc 8.3.0.

Maybe we should start a new ticket for the final polishing ? This one is very long already.

[jhpalmieri]

comment:400

Replying to @fchapoton:

Maybe we should start a new ticket for the final polishing ? This one is very long already.

Good idea. It's great seeing the progress!

[fchapoton]

comment:401

I have created #28298 for the few remaining final steps.

[fchapoton]

comment:402

LARGE SCALE REGRESSION in 8.9.b5, with 241 failing doctests !!!!

EDIT: This could have been caused by a wrong version of gcc being used. Will retry.

[jhpalmieri]

comment:403

Replying to @fchapoton:

LARGE SCALE REGRESSION in 8.9.b5, with 241 failing doctests !!!!

EDIT: This could have been caused by a wrong version of gcc being used. Will retry.

For what it's worth, on OS X I only see one new failure, due to #28225, a proposed fix at #28320.

[fchapoton]

comment:404

Replying to @fchapoton:

LARGE SCALE REGRESSION in 8.9.b5, with 241 failing doctests !!!!

EDIT: This could have been caused by a wrong version of gcc being used. Will retry.

This was a false alert, due to a wrong gcc. There are 18 failing doctests.