Build: use new Docker images from design document #8453

humitos · 2021-08-31T12:34:07Z

Minimal implementation for POC of #8447

It uses the new config file (#8478) to select the new build.os image and install Python versions via asdf (readthedocs/readthedocs-docker-images#166) using build.tools config option.

MinIO requires a new bucket called build-tools with a pre-compiled versions to work (compiled with ./scripts/compile_version_upload_s3.sh). If a version that's not pre-compiled is selected it will be downloaded from official mirrors, installed and used.

Build times on latest version for test-build:

using the new image + cached Python version: 112s
using the new image + non cached Python version: 288s
using old image (current production): 87s

Note that all the parsing of the Config File to support build.os and build.tools is not included in this PR on purpose. That work can be split as a separate work and done in parallel with the rest of work required here.

Testing this PR

build the image from New image structure based on design document readthedocs-docker-images#166
spin up your local development instance
go to your MinIO instance (http://localhost:9000/minio/) and create a new bucket called build-tools
run ./scripts/compile_version_upload_s3.sh python 3.8.12
merge this other PR New config for new docker build images #8478 into this one
build test-builds project with humitos/new-config-file branch/version

TODO

Note that not all these points are going to be done on this particular PR, but it's a good way to track what else is required

implement build.os and build.language on the Config File (see New config for new docker build images #8478)
remove the Feature flag to use the new docker image
consider compiling Python with --enable-shared (see Build python with --enable-shared readthedocs-docker-images#130)
update "Development Installation" documentation https://docs.readthedocs.io/en/latest/development/install.html

Closes #7239

Minimal implementation for POC of #8447 It uses a Feature flag for now as a way to select the new `readthedocs/build:ubuntu20` image and install Python versions via `asdf` (readthedocs/readthedocs-docker-images#166) MinIO requires a new bucket called `languages` with a pre-compiled Python 3.9.6 version to work (*) (this version is hardcoded for now). However, if a different version is selected it will be downloaded from official mirrors, installed and used. Build times on `latest` version for `test-build`: * using the new image + cached Python version: 112s * using the new image + non cached Python version: 288s * using old image (current production): 87s > Note that all the parsing of the Config File to support `build.os` and > `build.languages` is not included in this PR on purpose. That work can be > split as a separate work and done in parallel with the rest of work required > here. (*) to pre-compile a Python version: ```bash docker run -it readthedocs/build:ubuntu20 /bin/bash asdf install python 3.9.6 asdf global python 3.9.6 python -m pip install -U pip setuptools virtualenv cd /home/docs/.asdf/installs/python tar -cfvz ubuntu20-python-3.9.6.tar.gz 3.9.6 docker cp <container id>:/home/docs/.asdf/installs/python/ubuntu20-python-3.9.6.tar.gz . ``` and upload the .tar.gz file to MinIO `languages` bucket using the web interface

See readthedocs/readthedocs-docker-images#130

…cs/readthedocs.org into humitos/build-new-docker-image

agjohnson

Things look good so far! I haven't pulled this down to test however.

I did note it might be worth using a different name for languages so that it doesn't conflict with languages as in "locale languages" or "RTD project language". A more specific term wouldn't be as overloaded already.

readthedocs/doc_builder/python_environments.py

readthedocs/projects/tasks.py

Makes the name more specific and reduces potential confusion with other type of languages.

readthedocs/settings/docker_compose.py

humitos · 2021-09-20T16:06:01Z

Note for the future: if downloading .tar.gz on each build is a problem, we can pre-download them on builder startup instead and make them available for the Docker containers to just mv them.

ericholscher · 2021-09-20T18:14:16Z

Note for the future: if downloading .tar.gz on each build is a problem, we can pre-download them on builder startup instead and make them available for the Docker containers to just mv them.

This is a good idea. We could also do it at image build time, to make sure it's consistent. We have a few options, but I do think having them cached on disk is going to be a win in the future, but let's not block on that for now.

readthedocs/doc_builder/python_environments.py

readthedocs/projects/tasks.py

readthedocs/doc_builder/python_environments.py

…humitos/build-new-docker-image

readthedocs/doc_builder/python_environments.py

readthedocs/doc_builder/environments.py

Docs need to be updated, but I'm not doing that here, since this isn't implemented yet. Required by #8453 Co-authored-by: Manuel Kaufmann <[email protected]>

…humitos/build-new-docker-image

When installing miniconda3, `pyenv-build` calls: conda install --yes pip which automatically updates conda to its latest version and making us lossing our pinned version. This commit adds a patch to keep installing pip after conda install, but pinning conda to the current version to avoid upgrading it automatically. See pyenv/pyenv#2070

ericholscher

This looks close to ready -- I had a lot of nitpicks, but nothing super important that should block merging.

I do worry that we don't have an obvious idea of what versions will be cached at the start. Should we have a script that just contains the commands that we're planning to run on production to create cached versions of all the tools that we're planning to support?

readthedocs/doc_builder/environments.py

readthedocs/doc_builder/python_environments.py

scripts/compile_version_upload_s3.sh

humitos · 2021-09-27T18:03:56Z

I do worry that we don't have an obvious idea of what versions will be cached at the start. Should we have a script that just contains the commands that we're planning to run on production to create cached versions of all the tools that we're planning to support?

My plan is to cache all the current supported versions. That would be at most 1Gb on S3. We do have a command ./scripts/compile_version_upload_s3.sh that has to run with all the tools and versions once from production --I've already added noted this in the release.

The proposed patch was merged upstream and it's not required anymore. See pyenv/pyenv#2074

humitos · 2021-09-27T19:06:15Z

Thanks @ericholscher for the review on this. I'll merge now so we can deploy tomorrow and start doing some QA. I'll catch the suggestions in another PR!

See #8453 (review)

humitos requested a review from a team August 31, 2021 12:34

humitos force-pushed the humitos/build-new-docker-image branch from da1e227 to c3cec8a Compare September 1, 2021 09:12

This was referenced Sep 1, 2021

Docker build images: update design doc #8447

Merged

New image structure based on design document readthedocs/readthedocs-docker-images#166

Merged

humitos added 19 commits September 2, 2021 12:57

Script to compile a version and upload it to the cache (S3)

b553dc8

Compile Python with --enable-shared

3c3e51e

See readthedocs/readthedocs-docker-images#130

Terminate the script on failures

49e489d

Pass --env to docker exec properly

8948587

Typo

677ec6c

Comment the Python compilation options for now

d2da08d

Rename variable to not hide os module

0415235

Make compile&upload to work with Python 2.7 and its dependencies

94ce225

Don't run tar with --verbose

fd322bb

Use bash for the script

1ea6c60

Add awscli dependency to upload .tar.gz to S3 (MinIO)

d86e1b9

Reduce sleep time

6d2f4ed

Pass the variables to the sub-process so aws can access to them

5f0df77

Fix MinIO default password in script

1029806

Pass the variables to aws again :)

3bac624

Merge branch 'master' into humitos/build-new-docker-image

81d957b

Update if to avoid installing Python dependencies on conda/mamba

eb932b8

Merge branch 'humitos/build-new-docker-image' of github.com:readthedo…

128a361

…cs/readthedocs.org into humitos/build-new-docker-image

Remove the container used on timeout

009bff1

agjohnson reviewed Sep 8, 2021

View reviewed changes

Rename install_languages to install_build_languages

213a4bd

Makes the name more specific and reduces potential confusion with other type of languages.

astrojuanlu mentioned this pull request Sep 8, 2021

New Read the Docs tutorial, part II #8473

Merged

agjohnson reviewed Sep 8, 2021

View reviewed changes

readthedocs/settings/docker_compose.py Outdated Show resolved Hide resolved

This was referenced Sep 16, 2021

New config for new docker build images #8478

Merged

[MNT,DOC] use latest code in builds of the docs with RTD jupyter-book/thebe#472

Merged

humitos requested a review from a team September 20, 2021 15:59

humitos added 2 commits September 20, 2021 18:16

Remove feature flag

e6c05a4

Better syntax

0e0c08b

stsewd reviewed Sep 20, 2021

View reviewed changes

readthedocs/doc_builder/python_environments.py Outdated Show resolved Hide resolved

humitos added 2 commits September 21, 2021 10:13

Update with latest config object changes

c5f920e

Merge branch 'master' of github.com:readthedocs/readthedocs.org into …

5f8b4e8

…humitos/build-new-docker-image

stsewd reviewed Sep 21, 2021

View reviewed changes

readthedocs/doc_builder/python_environments.py Outdated Show resolved Hide resolved

readthedocs/doc_builder/python_environments.py Outdated Show resolved Hide resolved

readthedocs/doc_builder/environments.py Outdated Show resolved Hide resolved

humitos added 3 commits September 21, 2021 16:09

Test we are executing the asdf commands required

2131b74

Feedback addressed with new config file options

f2735ae

Lint

063d2b3

stsewd added a commit that referenced this pull request Sep 21, 2021

New config for new docker build images (#8478)

2e1b121

Docs need to be updated, but I'm not doing that here, since this isn't implemented yet. Required by #8453 Co-authored-by: Manuel Kaufmann <[email protected]>

humitos added 3 commits September 21, 2021 18:52

Merge branch 'master' of github.com:readthedocs/readthedocs.org into …

1e42d8d

…humitos/build-new-docker-image

Fix tests

326b231

ericholscher approved these changes Sep 27, 2021

View reviewed changes

humitos added 2 commits September 27, 2021 20:55

Do not patch pyenv-build anymore

9528849

The proposed patch was merged upstream and it's not required anymore. See pyenv/pyenv#2074

Explain $1 and $2 arguments for the script

94231b9

humitos enabled auto-merge September 27, 2021 19:06

humitos merged commit 0f87bc2 into master Sep 27, 2021

humitos deleted the humitos/build-new-docker-image branch September 27, 2021 19:12

humitos added a commit that referenced this pull request Sep 29, 2021

Build: missing updates from review

0c7c483

See #8453 (review)

humitos mentioned this pull request Sep 29, 2021

Build: missing updates from review #8544

Merged

stsewd mentioned this pull request Sep 29, 2021

Add Ability to ask for latest Python 3.X runtime to build docs #7814

Closed

jameslamb mentioned this pull request Jan 16, 2022

[ci] [docs] readthedocs configuration options are deprecated microsoft/LightGBM#4954

Closed

dhalbert mentioned this pull request Jan 28, 2022

correct python version for RTD adafruit/Adafruit_CircuitPython_AzureIoT#40

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build: use new Docker images from design document #8453

Build: use new Docker images from design document #8453

humitos commented Aug 31, 2021 •

edited

Loading

agjohnson left a comment

humitos commented Sep 20, 2021

ericholscher commented Sep 20, 2021

ericholscher left a comment •

edited

Loading

humitos commented Sep 27, 2021

humitos commented Sep 27, 2021

Build: use new Docker images from design document #8453

Build: use new Docker images from design document #8453

Conversation

humitos commented Aug 31, 2021 • edited Loading

Testing this PR

TODO

agjohnson left a comment

Choose a reason for hiding this comment

humitos commented Sep 20, 2021

ericholscher commented Sep 20, 2021

ericholscher left a comment • edited Loading

Choose a reason for hiding this comment

humitos commented Sep 27, 2021

humitos commented Sep 27, 2021

humitos commented Aug 31, 2021 •

edited

Loading

ericholscher left a comment •

edited

Loading