Build: install all the latest Python "core requirements" #10508

humitos · 2023-07-04T11:20:06Z

I created a new feature flag called INSTALL_LATEST_CORE_REQUIREMENTS that does
not depend on any other feature flag. Its goal is to always install the latest
Python packages.

We are trying to move away from pinning the dependencies on our side and telling
users to do that on their side if they want to have reproducible builds over
time.

I think this is the best outcome for new projects since they will immediately
use the latest versions of everything instead of old (and maybe broken)
dependencies.

I propose to set a particular date for this feature flag and make new projects
to start building with all the latest dependencies. This will, at least, stop
making the problem bigger. Later, we can decide how to communicate to old
projects that we are going to install the latest requirements and if they don't
want that, they should pin their dependencies. We can follow our new and shiny
deprecation path we have built and tested in the last month.

Related readthedocs/meta#8
Related #9562
Related #10402
Related #9081

📚 Documentation previews 📚

User's documentation (docs): https://docs--10508.org.readthedocs.build/en/10508/

Developer's documentation (dev): https://dev--10508.org.readthedocs.build/en/10508/

I created a new feature flag called `INSTALL_LATEST_CORE_REQUIREMENTS` that does not depend on any other feature flag. Its goal is to always install the latest Python packages. We are trying to move away from pinning the dependencies on our side and telling users to do that on their side if they want to have reproducible builds over time. I think this is the best outcome for new projects since they will immediately use the latest versions of everything instead of old (and maybe broken) dependencies. I propose to set a particular date for this feature flag and make new projects to start building with all the latest dependencies. This will, at least, stop making the problem bigger. Later, we can decide how to communicate to old projects that we are going to install the latest requirements and if they don't want that, they should pin their dependencies. We can follow our new and shiny deprecation path we have built and tested in the last month. Related readthedocs/meta#8 Related #9562 Related #10402 Related #9081

readthedocs/doc_builder/python_environments.py

ericholscher

This makes a ton of sense to me, to get things on a good state for new projects, and then work backwards for old projects to move to this state 👍

ericholscher · 2023-07-05T22:43:47Z

docs/user/build-default-versions.rst

@@ -56,20 +56,25 @@ setuptools:
  All other projects use the latest version.

 mock:
-  ``1.0.1`` (could be removed in the future).
+  ``1.0.1``.
+  Projects created after August 7, 2023 won't install this dependency by default.


I don't think this will break anything, since it's for new projects, so I'm fine doing it sooner.

I targeted this for next month to give us some time to review the PR, make the adjustments, etc and maybe write a pretty small changelog blog post announcing this so we have something to link people to when they ask.

readthedocs/doc_builder/python_environments.py

Do not install `sphinx-rtd-theme` and clarify the docstring.

humitos · 2023-07-06T10:12:54Z

Are we OK moving one step forward?:

stop upgrading pip and setuptools
and removing Sphinx and MkDocs

Is this too much? 💣 Is this the goal we want to reach eventually? If that's the goal, are we OK with this intermediate step where we are removing most of the core dependencies but not the doctools?

ericholscher · 2023-07-06T15:56:07Z

Is this too much? 💣 Is this the goal we want to reach eventually? If that's the goal, are we OK with this intermediate step where we are removing most of the core dependencies but not the doctools?

I think we probably want to keep installing the actual build tool, since they are configuring it in the config file. If folks don't want anything installed, they can use build.commands or eventually override build.steps.install?

I don't feel super strongly though. The number of projects that just want Sphinx, and no extensions seems pretty low, but like a reasonable thing to support out of the box? Otherwise, we need to start requiring a requirements file, or a supported way to pin dependencies in the RTD YAML, and validating the build tool is installed, and that seems like a larger step.

humitos · 2023-07-06T16:27:10Z

Yeah, good point. I think it's fine to keep installing those for now. We can re-think this in the feature if required and when we are more prepared to do this step.

Announce the changes made at readthedocs/readthedocs.org#10508 Note that we are not exposing this feature flag to our users. We don't want them to contact us to enable this flag. If they need to install the latest versions they can just specify them in their `requirements.txt` file.

* Post: small post about "latest core requirements" Announce the changes made at readthedocs/readthedocs.org#10508 Note that we are not exposing this feature flag to our users. We don't want them to contact us to enable this flag. If they need to install the latest versions they can just specify them in their `requirements.txt` file. * Update python-core-requirements-changed.rst Co-authored-by: Eric Holscher <[email protected]> * Update python-core-requirements-changed.rst --------- Co-authored-by: Eric Holscher <[email protected]>

humitos · 2023-07-11T08:20:24Z

@ericholscher this PR is ready for re-review. This is the new behavior:

ericholscher

Looks good to me 👍

ericholscher · 2023-07-11T18:28:38Z

readthedocs/doc_builder/python_environments.py

+        This is the new behavior and where we are moving towards.
+        """
+        # First, upgrade pip and setuptools to their latest versions
+        cmd = pip_install_cmd + ["pip", "setuptools"]


I do think upgrading pip & setuptools will definitely cause us issues when they ship busted versions, but hopefully that will teach users to pin their deps! 🗡️

humitos added 4 commits July 4, 2023 12:53

Build: remove deprecated dependencies

f028546

Build: install latest core requirements when using Conda

1af8d48

Docs: update default versions documentation to match the changes

b1e44d5

humitos requested review from a team as code owners July 4, 2023 11:20

humitos requested a review from ericholscher July 4, 2023 11:20

auto-assign bot assigned humitos Jul 4, 2023

stsewd reviewed Jul 5, 2023

View reviewed changes

readthedocs/doc_builder/python_environments.py Outdated Show resolved Hide resolved

readthedocs/doc_builder/python_environments.py Outdated Show resolved Hide resolved

ericholscher reviewed Jul 5, 2023

View reviewed changes

Feedback from review

2ce5d41

Do not install `sphinx-rtd-theme` and clarify the docstring.

humitos requested review from ericholscher and stsewd July 6, 2023 10:14

Update documentation

8e3e874

This was referenced Jul 6, 2023

Feature flag: MkDocs use latest has its logic inverted #10402

Closed

Build fails for sphinx=6.2.1, claiming "requires docutils >=0.18.1,<0.20, but none of the providers can be installed" #10477

Closed

humitos mentioned this pull request Jul 6, 2023

Post: small post about "latest core requirements" readthedocs/blog#226

Merged

humitos mentioned this pull request Jul 10, 2023

Deprecate old feature flags #9779

Closed

ericholscher approved these changes Jul 11, 2023

View reviewed changes

ericholscher reviewed Jul 11, 2023

View reviewed changes

humitos merged commit cd498b5 into main Jul 12, 2023

humitos deleted the humitos/install-latest-core-requirements branch July 12, 2023 08:31

humitos mentioned this pull request Aug 1, 2023

Deprecations: list of all past and ongoing deprecations #10587

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build: install all the latest Python "core requirements" #10508

Build: install all the latest Python "core requirements" #10508

humitos commented Jul 4, 2023 •

edited by github-actions bot

Loading

ericholscher left a comment

ericholscher Jul 5, 2023

humitos Jul 6, 2023

humitos commented Jul 6, 2023

ericholscher commented Jul 6, 2023 •

edited

Loading

humitos commented Jul 6, 2023

humitos commented Jul 11, 2023

ericholscher left a comment

ericholscher Jul 11, 2023

Build: install all the latest Python "core requirements" #10508

Build: install all the latest Python "core requirements" #10508

Conversation

humitos commented Jul 4, 2023 • edited by github-actions bot Loading

ericholscher left a comment

Choose a reason for hiding this comment

ericholscher Jul 5, 2023

Choose a reason for hiding this comment

humitos Jul 6, 2023

Choose a reason for hiding this comment

humitos commented Jul 6, 2023

ericholscher commented Jul 6, 2023 • edited Loading

humitos commented Jul 6, 2023

humitos commented Jul 11, 2023

ericholscher left a comment

Choose a reason for hiding this comment

ericholscher Jul 11, 2023

Choose a reason for hiding this comment

humitos commented Jul 4, 2023 •

edited by github-actions bot

Loading

ericholscher commented Jul 6, 2023 •

edited

Loading