3.4.1rc1 test_pydoc fails: pydoc_data.topics.topics values are type "bytes" not "str"

[ned-deily]

BPO	21431
Nosy	@birkenfeld, @larryhastings, @benjaminp, @ned-deily, @zware

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2014-09-22.19:22:30.094>
created_at = <Date 2014-05-04.20:10:51.068>
labels = ['build', 'release-blocker']
title = '3.4.1rc1 test_pydoc fails: pydoc_data.topics.topics values are type "bytes" not "str"'
updated_at = <Date 2014-09-22.22:21:09.263>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2014-09-22.22:21:09.263>
actor = 'larry'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2014-09-22.19:22:30.094>
closer = 'python-dev'
components = ['Build']
creation = <Date 2014-05-04.20:10:51.068>
creator = 'ned.deily'
dependencies = []
files = []
hgrepos = []
issue_num = 21431
keywords = []
message_count = 12.0
messages = ['217887', '217889', '217890', '217898', '217899', '227296', '227297', '227311', '227312', '227313', '227314', '227315']
nosy_count = 6.0
nosy_names = ['georg.brandl', 'larry', 'benjamin.peterson', 'ned.deily', 'python-dev', 'zach.ware']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21431'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[ned-deily]

Something went wrong with the update of pydoc_data topics for 3.4.1rc1. As can be seen in http://hg.python.org/releasing/3.4.1/rev/c67a19e11a71, the values for the topics dict should be strings but were updated as bytes. This causes pydoc topics searches to fail. The process problem needs to be fixed for 3.4.1 final.

An example:

$ /usr/local/bin/pydoc3.4 def
Traceback (most recent call last):
  File "/usr/local/bin/pydoc3.4", line 5, in <module>
    pydoc.cli()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pydoc.py", line 2580, in cli
    help.help(arg)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pydoc.py", line 1860, in help
    elif request in self.keywords: self.showtopic(request)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pydoc.py", line 1941, in showtopic
    pager(doc.strip() + '\n')
TypeError: can't concat bytes to str

[larryhastings]

3.4.1rc1 is the first release I've cut where the makefile didn't auto-download Sphinx. And then the makefile used "python" and "sphinx-build" straight off the path, rather than finding the local ones. To generate pydoc-topics, I had to do the following:

% ./configure --prefix=pwd/release && make
% ./release/bin/pip install sphinx
% cd Doc
% make pydoc-topics PYTHON=../release/bin/python SPHINXBUILD=../release/bin/sphinx-build

And apparently this didn't work.

Maybe there should be a "smoke test" to make sure pydoc-topics is okay?

And... maybe "make pydoc-topics" should copy the data file itself, rather than making me cut and paste paths out of PEP-101? And *then* automatically test it?

[ned-deily]

From a first quick look, it appears that the problem occurs when using a Python 3 version of sphinx-build. With Python 2, the topics appear to be generated correctly.

[ned-deily]

The problem is in PydocTopicsBuilder in Doc/tools/sphinxext/pyspecific.py. It needs to be smarter so that ideally it should continue to work with any Python >= 2.5 and independent of the Python being built.

[larryhastings]

Well, surely working with the current python is sufficient? I'd be happy if it was only guaranteed to run with the python tree it's a part of.

[python-dev]

New changeset d71c351a6a0f by Georg Brandl in branch '3.4':
Closes bpo-21431: make docs depend on Sphinx 1.2 and fix pydoc-topics builder to
https://hg.python.org/cpython/rev/d71c351a6a0f

[birkenfeld]

Should be fixed now. I didn't merge into default since someone (cough) has to null-merge all the release related stuff first.

[python-dev]

New changeset 1248796b7945 by Ned Deily in branch 'default':
Issue bpo-21431: merge from 3.4
https://hg.python.org/cpython/rev/1248796b7945

[ned-deily]

To expedite matters, I did the almost-null post-release cleanup merge and then merged the fix for this issue. So I think this issue is now complete.

[larryhastings]

If this is fixed, then how come I hit it again today?

[ned-deily]

It was just fixed today, after the release.

[larryhastings]

Oh, because it was only fixed today. As Emily Litella used to say... never mind!