Use dirhtml builder for Read the Docs

[hugovk]

Follow on from #933, we should use the dirhtml builder instead of html, so the redirects work as before.

Example redirect:

• https://cpython-devguide--965.org.readthedocs.build/pullrequest/ ➡️ https://cpython-devguide--965.org.readthedocs.build/getting-started/pull-request-lifecycle/index.html

[hugovk]

Build finished. The HTML pages are in _build/dirhtml.
...
No "_readthedocs/html" folder was created during this build.

We call make dirhtml BUILDDIR=_readthedocs

RTD is looking for an html dir in BUILDDIR, i.e. _readthedocs/html.

Will update.

[ezio-melotti]

Thanks @hugovk for the quick fix!
Pinging @humitos too since this might be interesting to know for him too.

[humitos]

Hi @ezio-melotti. Thanks for the ping on a readthedocs-related issue 👍🏼 I read the PR but I'm not sure to understand how I can be useful here. Are you suggesting that Read the Docs should also check for _readthedocs/dirhtml directory?

[ezio-melotti]

I pinged you for a few reasons:

• Even though the .readthedocs.yml already specifies builder: dirhtml
  

  devguide/.readthedocs.yml

  Lines 7 to 9 in 05f6d0c

  	sphinx:
  	builder: dirhtml
  	configuration: conf.py

  
  it is then overridden by the make build command (formerly make html):
  

  devguide/.readthedocs.yml

  Lines 16 to 18 in 05f6d0c

  	commands:
  	- make dirhtml BUILDDIR=_readthedocs
  	- mv _readthedocs/dirhtml _readthedocs/html

  
  Perhaps we should remove the builder: dirhtml (or the make command)?
• If the output dir matches the builder name, then RTD could check for _readthedocs/<buildername> automatically (assuming the builder option is kept, otherwise it could just check for html/dirhtml and possibly others -- I'm not sure if this would work with other builders).
• When we switched (in Override RTD build commands to speed up docs build #933), we went from dirhtml to html, and that broke our redirects (we are using sphinx-rediraffe). It took me a few days to realize that there was something wrong, and @hugovk was able to quickly pinpoint the issue to the switch, even though this wasn't immediately obvious. Knowing this might help you review future PRs from other projects that are switching and/or easily debug similar issues.

[hugovk]

Perhaps we should remove the builder: dirhtml (or the make command)?

Yes, we're overriding the build and running Sphinx our own way via make, I think we can remove that whole sphinx block?

devguide/.readthedocs.yml

Lines 7 to 9 in 05f6d0c

	sphinx:
	builder: dirhtml
	configuration: conf.py

[humitos]

@ezio-melotti Got it! All your bullets make sense and this is good feedback 👍🏼

Even though the .readthedocs.yml already specifies builder: dirhtml
it is then overridden by the make build command (formerly make html):

When using build.commands the whole Read the Docs build process is overwrite and Read the Docs won't execute any command on behalf of the user. So, the sphinx config key, and others (like build.apt_packages, python.install, etc) have no effect (also readthedocs/readthedocs.org#9280)

There are some "inconsistencies" in the config file currently that we plan to achieve in the v3 of it. We didn't want to introduce too many changes together with build.commands because it was released as a beta feature as an experiment to know how much adoption it has and grow from there. That's why your feedback is really valuable 😉

If the output dir matches the builder name, then RTD could check for _readthedocs/ automatically (assuming the builder option is kept, otherwise it could just check for html/dirhtml and possibly others -- I'm not sure if this would work with other builders).

This does not work like you are describing on purpose. Instead of looking for builder's name output we decided to look for format type (html, pdf, epub, etc). The idea behind this is to be able to support more generic doctools that are not Sphinx. Currently, we only support HTML, but we planned to support all the other formats as well.

When we switched (in #933), we went from dirhtml to html, and that broke our redirects (we are using sphinx-rediraffe).

What would expect from Read the Docs to do in this case? (I'm asking a general/open question without mentioning any idea to not interfere in your answer 😄 )

[ezio-melotti]

What would expect from Read the Docs to do in this case?

I'm not to familiar with what RTD does behind the scenes, but the way the .readthedoc.yml was configured before looked better, since you could just specify the builder name and the config file and everything else worked out of the box. Currently we have to invoke make manually, and since RTD doesn't know which builder we used and doesn't look for a dirhtml dir then we also need to manually rename it to html to make it work.

This is not a big deal and in a way it even seems more explicit and flexible (which is good!), but it also feel a bit hackish and not too easy to figure out.

[humitos]

@ezio-melotti

I'm not to familiar with what RTD does behind the scenes, but the way the .readthedoc.yml was configured before looked better, since you could just specify the builder name and the config file and everything else worked out of the box. Currently we have to invoke make manually, and since RTD doesn't know which builder we used and doesn't look for a dirhtml dir then we also need to manually rename it to html to make it work.

Here you are describing exactly one of the benefits of "let Read the Docs to handle the build" over "overwrite the build process completely with a custom one 100% managed by the user". When Read the Docs handles the build it has a lot more data about how that process will be and can make accurate decisions on behalf of the user automatically. On the other hand, if the build is customized, Read the Docs cannot know what will be the out of those command and where it will end ¹

IMHO, if your build process follow the standards and is compatible with the regular Read the Docs build process, build.commands should not be used. On #933 the build process was swapped to use build.commands to earn some seconds of build time and that came with the removal of the benefits that you already noticed.

As you can see, there are pros and cons for each of the approaches considered here and I don't think Read the Docs will be able to have the best of the two until we implemented what I mentioned in the note ¹ --but we are not ready to start working on that yet.

I know that I'm not giving you a concrete answer here or "what to do next?" or "how Read the Docs could help on these cases?", but I hope that at least it clarifies why we can't know exactly what's happening under build.commands due that we don't have control over it.

Footnotes

1. we are working on "defining a contract" for the build process that will help users to customize the build while letting know Read the Docs how the build and artifacts generated looks like: https://github.com/readthedocs/readthedocs.org/issues/9088. Unfortunately, a lot of the discussion was done privately and we haven't exposed yet --but we have that issue to write done these notes when we are ready ↩ ↩²

[ezio-melotti]

Thank you for your comment, now I definitely have a better understanding of how everything works.