Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a dedicated "Build system interface" reference section #10497

Merged
merged 6 commits into from
Sep 30, 2021
Merged

Add a dedicated "Build system interface" reference section #10497

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d37aa21
Add a dedicated "Build system interface" reference section
pradyunsg Sep 11, 2021
61b1e69
:newspaper:
pradyunsg Sep 22, 2021
2298088
Update according to the PEP 660 implementation
pradyunsg Sep 29, 2021
25a31d4
Improve build system interface docs based on feedback
pradyunsg Sep 29, 2021
d6d7217
Rephase historical note about `easy_install`/`setup_requires`
pradyunsg Sep 29, 2021
4d7d38a
Addresss spelling nit
pradyunsg Sep 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 1 addition & 168 deletions docs/html/cli/pip.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,178 +74,11 @@ when decision is needed.
*(a)abort*
Abort pip and return non-zero exit status.

.. _`build-interface`:


Build System Interface
======================

pip builds packages by invoking the build system. By default, builds will use
``setuptools``, but if a project specifies a different build system using a
``pyproject.toml`` file, as per :pep:`517`, pip will use that instead. As well
as package building, the build system is also invoked to install packages
direct from source. This is handled by invoking the build system to build a
wheel, and then installing from that wheel. The built wheel is cached locally
by pip to avoid repeated identical builds.

The current interface to the build system is via the ``setup.py`` command line
script - all build actions are defined in terms of the specific ``setup.py``
command line that will be run to invoke the required action.

Setuptools Injection
~~~~~~~~~~~~~~~~~~~~

When :pep:`517` is not used, the supported build system is ``setuptools``.
However, not all packages use ``setuptools`` in their build scripts. To support
projects that use "pure ``distutils``", pip injects ``setuptools`` into
``sys.modules`` before invoking ``setup.py``. The injection should be
transparent to ``distutils``-based projects, but 3rd party build tools wishing
to provide a ``setup.py`` emulating the commands pip requires may need to be
aware that it takes place.

Projects using :pep:`517` *must* explicitly use setuptools - pip does not do
the above injection process in this case.

Build System Output
~~~~~~~~~~~~~~~~~~~

Any output produced by the build system will be read by pip (for display to the
user if requested). In order to correctly read the build system output, pip
requires that the output is written in a well-defined encoding, specifically
the encoding the user has configured for text output (which can be obtained in
Python using ``locale.getpreferredencoding``). If the configured encoding is
ASCII, pip assumes UTF-8 (to account for the behaviour of some Unix systems).

Build systems should ensure that any tools they invoke (compilers, etc) produce
output in the correct encoding. In practice - and in particular on Windows,
where tools are inconsistent in their use of the "OEM" and "ANSI" codepages -
this may not always be possible. pip will therefore attempt to recover cleanly
if presented with incorrectly encoded build tool output, by translating
unexpected byte sequences to Python-style hexadecimal escape sequences
(``"\x80\xff"``, etc). However, it is still possible for output to be displayed
using an incorrect encoding (mojibake).

Under :pep:`517`, handling of build tool output is the backend's responsibility,
and pip simply displays the output produced by the backend. (Backends, however,
will likely still have to address the issues described above).

PEP 517 and 518 Support
~~~~~~~~~~~~~~~~~~~~~~~

As of version 10.0, pip supports projects declaring dependencies that are
required at install time using a ``pyproject.toml`` file, in the form described
in :pep:`518`. When building a project, pip will install the required
dependencies locally, and make them available to the build process.
Furthermore, from version 19.0 onwards, pip supports projects specifying the
build backend they use in ``pyproject.toml``, in the form described in
:pep:`517`.

When making build requirements available, pip does so in an *isolated
environment*. That is, pip does not install those requirements into the user's
``site-packages``, but rather installs them in a temporary directory which it
adds to the user's ``sys.path`` for the duration of the build. This ensures
that build requirements are handled independently of the user's runtime
environment. For example, a project that needs a recent version of setuptools
to build can still be installed, even if the user has an older version
installed (and without silently replacing that version).

In certain cases, projects (or redistributors) may have workflows that
explicitly manage the build environment. For such workflows, build isolation
can be problematic. If this is the case, pip provides a
``--no-build-isolation`` flag to disable build isolation. Users supplying this
flag are responsible for ensuring the build environment is managed
appropriately (including ensuring that all required build dependencies are
installed).

By default, pip will continue to use the legacy (direct ``setup.py`` execution
based) build processing for projects that do not have a ``pyproject.toml`` file.
Projects with a ``pyproject.toml`` file will use a :pep:`517` backend. Projects
with a ``pyproject.toml`` file, but which don't have a ``build-system`` section,
will be assumed to have the following backend settings::

[build-system]
requires = ["setuptools>=40.8.0", "wheel"]
build-backend = "setuptools.build_meta:__legacy__"

.. note::

``setuptools`` 40.8.0 is the first version of setuptools that offers a
:pep:`517` backend that closely mimics directly executing ``setup.py``.

If a project has ``[build-system]``, but no ``build-backend``, pip will also use
``setuptools.build_meta:__legacy__``, but will expect the project requirements
to include ``setuptools`` and ``wheel`` (and will report an error if the
installed version of ``setuptools`` is not recent enough).

If a user wants to explicitly request :pep:`517` handling even though a project
doesn't have a ``pyproject.toml`` file, this can be done using the
``--use-pep517`` command line option. Similarly, to request legacy processing
even though ``pyproject.toml`` is present, the ``--no-use-pep517`` option is
available (although obviously it is an error to choose ``--no-use-pep517`` if
the project has no ``setup.py``, or explicitly requests a build backend). As
with other command line flags, pip recognises the ``PIP_USE_PEP517``
environment veriable and a ``use-pep517`` config file option (set to true or
false) to set this option globally. Note that overriding pip's choice of
whether to use :pep:`517` processing in this way does *not* affect whether pip
will use an isolated build environment (which is controlled via
``--no-build-isolation`` as noted above).

Except in the case noted above (projects with no :pep:`518` ``[build-system]``
section in ``pyproject.toml``), pip will never implicitly install a build
system. Projects **must** ensure that the correct build system is listed in
their ``requires`` list (this applies even if pip assumes that the
``setuptools`` backend is being used, as noted above).

.. _pep-518-limitations:

**Historical Limitations**:

* ``pip<18.0``: only supports installing build requirements from wheels, and
does not support the use of environment markers and extras (only version
specifiers are respected).

* ``pip<18.1``: build dependencies using .pth files are not properly supported;
as a result namespace packages do not work under Python 3.2 and earlier.

Future Developments
~~~~~~~~~~~~~~~~~~~

:pep:`426` notes that the intention is to add hooks to project metadata in
version 2.1 of the metadata spec, to explicitly define how to build a project
from its source. Once this version of the metadata spec is final, pip will
migrate to using that interface. At that point, the ``setup.py`` interface
documented here will be retained solely for legacy purposes, until projects
have migrated.

Specifically, applications should *not* expect to rely on there being any form
of backward compatibility guarantees around the ``setup.py`` interface.


Build Options
~~~~~~~~~~~~~

The ``--global-option`` and ``--build-option`` arguments to the ``pip install``
and ``pip wheel`` inject additional arguments into the ``setup.py`` command
(``--build-option`` is only available in ``pip wheel``). These arguments are
included in the command as follows:

.. tab:: Unix/macOS

.. code-block:: console

python setup.py <global_options> BUILD COMMAND <build_options>

.. tab:: Windows

.. code-block:: shell

py setup.py <global_options> BUILD COMMAND <build_options>

The options are passed unmodified, and presently offer direct access to the
distutils command line. Use of ``--global-option`` and ``--build-option``
should be considered as build system dependent, and may not be supported in the
current form if support for alternative build systems is added to pip.

This is now covered in :doc:`../reference/build-system/index`.

.. _`General Options`:

Expand Down
80 changes: 5 additions & 75 deletions docs/html/cli/pip_install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -419,7 +419,7 @@ with other repeatability strategies.
(rare) packages that use it will cause those dependencies to be downloaded
by setuptools directly, skipping pip's hash-checking. If you need to use
such a package, see :ref:`Controlling
setup_requires<controlling-setup-requires>`.
setup_requires <controlling-setup_requires>`.

.. warning::

Expand Down Expand Up @@ -537,83 +537,10 @@ the project path. This is one advantage over just using ``setup.py develop``,
which creates the "egg-info" directly relative the current working directory.


.. _`controlling-setup-requires`:

Controlling setup_requires
--------------------------

Setuptools offers the ``setup_requires`` `setup() keyword
<https://setuptools.readthedocs.io/en/latest/userguide/keywords.html>`_
for specifying dependencies that need to be present in order for the
``setup.py`` script to run. Internally, Setuptools uses ``easy_install``
to fulfill these dependencies.

pip has no way to control how these dependencies are located. None of the
package index options have an effect.

The solution is to configure a "system" or "personal" `Distutils configuration
file
<https://docs.python.org/3/install/index.html#distutils-configuration-files>`_ to
manage the fulfillment.

For example, to have the dependency located at an alternate index, add this:

::

[easy_install]
index_url = https://my.index-mirror.com

To have the dependency located from a local directory and not crawl PyPI, add this:

::

[easy_install]
allow_hosts = ''
find_links = file:///path/to/local/archives/


Build System Interface
----------------------

In order for pip to install a package from source, ``setup.py`` must implement
the following commands::

setup.py egg_info [--egg-base XXX]
setup.py install --record XXX [--single-version-externally-managed] [--root XXX] [--compile|--no-compile] [--install-headers XXX]

The ``egg_info`` command should create egg metadata for the package, as
described in the setuptools documentation at
https://setuptools.readthedocs.io/en/latest/userguide/commands.html#egg-info-create-egg-metadata-and-set-build-tags

The ``install`` command should implement the complete process of installing the
package to the target directory XXX.

To install a package in "editable" mode (``pip install -e``), ``setup.py`` must
implement the following command::

setup.py develop --no-deps

This should implement the complete process of installing the package in
"editable" mode.

All packages will be attempted to built into wheels::

setup.py bdist_wheel -d XXX

One further ``setup.py`` command is invoked by ``pip install``::

setup.py clean

This command is invoked to clean up temporary commands from the build. (TODO:
Investigate in more detail when this command is required).

No other build system commands are invoked by the ``pip install`` command.

Installing a package from a wheel does not invoke the build system at all.

.. _PyPI: https://pypi.org/
.. _setuptools extras: https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies

This is now covered in :doc:`../reference/build-system/index`.


.. _`pip install Options`:
Expand Down Expand Up @@ -906,3 +833,6 @@ Examples

.. [1] This is true with the exception that pip v7.0 and v7.0.1 required quotes
around specifiers containing environment markers in requirement files.

.. _setuptools extras: https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies
.. _PyPI: https://pypi.org/
55 changes: 1 addition & 54 deletions docs/html/cli/pip_wheel.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,60 +28,7 @@ Description
Build System Interface
----------------------

In order for pip to build a wheel, ``setup.py`` must implement the
``bdist_wheel`` command with the following syntax:

.. tab:: Unix/macOS

.. code-block:: shell

python setup.py bdist_wheel -d TARGET

.. tab:: Windows

.. code-block:: shell

py setup.py bdist_wheel -d TARGET


This command must create a wheel compatible with the invoking Python
interpreter, and save that wheel in the directory TARGET.

No other build system commands are invoked by the ``pip wheel`` command.

Customising the build
^^^^^^^^^^^^^^^^^^^^^

It is possible using ``--global-option`` to include additional build commands
with their arguments in the ``setup.py`` command. This is currently the only
way to influence the building of C extensions from the command line. For
example:

.. tab:: Unix/macOS

.. code-block:: shell

python -m pip wheel --global-option bdist_ext --global-option -DFOO wheel

.. tab:: Windows

.. code-block:: shell

py -m pip wheel --global-option bdist_ext --global-option -DFOO wheel


will result in a build command of

::

setup.py bdist_ext -DFOO bdist_wheel -d TARGET

which passes a preprocessor symbol to the extension build.

Such usage is considered highly build-system specific and more an accident of
the current implementation than a supported interface.


This is now covered in :doc:`../reference/build-system/index`.

Options
=======
Expand Down
Loading