From b8cd76339d3a95e40c47a985edee2dc8031feaf0 Mon Sep 17 00:00:00 2001 From: Bernard Date: Wed, 21 Oct 2020 17:50:04 +0200 Subject: [PATCH 01/14] WIP updating user guide documentation with explanation and solutions to pip backtracking --- docs/html/user_guide.rst | 132 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 3c9cfec6319..faceeb4fdbf 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1306,6 +1306,138 @@ issue tracker`_ if you believe that your problem has exposed a bug in pip. .. _`Using pip from your program`: +Dependency resolution backtracking +================================== + +Or more commonly known as *"Why does pip download multiple versions of +the same package over and over again?"*. + +The purpose of this section is to provide explanation and practical +suggestions to pip users who encounter dependency resolution +backtracking during a ``pip install`` command. + +Why does backtracking occur? +---------------------------- + +With the release of the new resolver (resolver 2020), pip is now more +strict in the package versions it installs, when a users does a +``pip install`` command. This new behaviour means that pip works harder +to find out which version of a package is a good candidate to install. + +What is backtracking? +--------------------- + +During a pip install (e.g. ``pip install tea``), pip needs to work out +the packages dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc). +For each of these dependent packages, it needs to work out which version +of a package is a good candidate to install. When it does, it has a set +of compatible packages which is installs. + +If pip tries one version, finds out it isn’t compatible it needs to “go +back” (backtrack) and download an older version. It then tries that, if +it is successful it will continue onto the next package - if not it will +continue to backtrack until it finds a compatible version. + +This backtrack behaviour can end in 2 ways - either 1) it will +successfully find a set packages it can install (good news!), or 2) it +will eventually display `resolution impossible`_ error message (not so +good). + +If pip starts backtracking during dependency resolution, it does not +know how long it will backtrack, and how much computation would be +needed. + +.. _resolution impossible: https://pip.pypa.io/en/latest/user_guide/#id35 + +What does this behaviour look like? +----------------------------------- + +Right now backtracking looks like this: +:: + + $ pip install tea==1.9.8 + Collecting tea==1.9.8 + Downloading tea-1.9.8-py2.py3-none-any.whl (346 kB) + |████████████████████████████████| 346 kB 10.4 MB/s + Collecting spoon==2.27.0 + Downloading spoon-2.27.0-py2.py3-none-any.whl (312 kB) + |████████████████████████████████| 312 kB 19.2 MB/s + Collecting hot-water>=0.1.9 + Downloading hot-water-0.1.13-py3-none-any.whl (9.3 kB) + Collecting cup>=1.6.0 + Downloading cup-3.22.0-py2.py3-none-any.whl (397 kB) + |████████████████████████████████| 397 kB 28.2 MB/s + INFO: pip is looking at multiple versions of this package to determine + which version is compatible with other requirements. + This could take a while. + Downloading cup-3.21.0-py2.py3-none-any.whl (395 kB) + |████████████████████████████████| 395 kB 27.0 MB/s + Downloading cup-3.20.0-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 24.4 MB/s + Downloading cup-3.19.1-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 21.3 MB/s + Downloading cup-3.19.0-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 26.2 MB/s + Downloading cup-3.18.0-py2.py3-none-any.whl (393 kB) + |████████████████████████████████| 393 kB 22.1 MB/s + Downloading cup-3.17.0-py2.py3-none-any.whl (382 kB) + |████████████████████████████████| 382 kB 23.8 MB/s + Downloading cup-3.16.0-py2.py3-none-any.whl (376 kB) + |████████████████████████████████| 376 kB 27.5 MB/s + Downloading cup-3.15.1-py2.py3-none-any.whl (385 kB) + |████████████████████████████████| 385 kB 30.4 MB/s + INFO: pip is looking at multiple versions of this package to determine + which version is compatible with other requirements. + This could take a while. + Downloading cup-3.15.0-py2.py3-none-any.whl (378 kB) + |████████████████████████████████| 378 kB 21.4 MB/s + Downloading cup-3.14.0-py2.py3-none-any.whl (372 kB) + |████████████████████████████████| 372 kB 21.1 MB/s + Downloading cup-3.13.1-py2.py3-none-any.whl (381 kB) + |████████████████████████████████| 381 kB 21.8 MB/s + This is taking longer than usual. You might need to provide the + dependency resolver with stricter constraints to reduce runtime. + If you want to abort this run, you can press Ctrl + C to do so. + Downloading cup-3.13.0-py2.py3-none-any.whl (374 kB) + +In the above sample output, pip must downloaded multiple versions of +package cup - cup-3.22.0 to cup-3.13.0 - to find a version that will be +compatible with the other packages - ``spoon``, ``hot-water``, ``cup`` +etc. + +## Possible solutions + +====UPDATE FROM HERE====== +:::info +Unsure what possible solutions are. AFAIK there are no solutions - apart 1) waiting til it's finished, or 2) [dealing with potential resolution impossible](https://pip.pypa.io/en/latest/user_guide/#id37). + +Pradyun mentions "stricter constraints to reduce runtime". + +Can we help the user with these questions: +- What can I do to reduce the chances of backtracking occuring? +- Backtracking keeps happening with my installs, what can I do? +- Can I delete these unnecessary packages? +::: + +Possible solutions +------------------ + +:::info Unsure what possible solutions are. AFAIK there are no solutions +- apart 1) waiting til it's finished, or 2) `dealing with potential +resolution impossible`_. + +Pradyun mentions "stricter constraints to reduce runtime". + +Can we help the user with these questions: + +- What can I do to reduce the chances of backtracking occuring? +- Backtracking keeps happening with my installs, what can I do? +- Can I delete these unnecessary packages? ::: + +.. _dealing with potential resolution impossible: https://pip.pypa.io/en/latest/user_guide/#id37 +=====UPDATE UNTIL HERE====== + + Using pip from your program =========================== From cf89b0b6c2cf2449560d53b868cdd9e0d52f6abd Mon Sep 17 00:00:00 2001 From: Bernard Date: Fri, 23 Oct 2020 15:14:19 +0200 Subject: [PATCH 02/14] updating user guide with backtracking information --- docs/html/user_guide.rst | 188 ++++++++++++++++++++++++++++++--------- 1 file changed, 144 insertions(+), 44 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index faceeb4fdbf..aecb427f3e4 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1306,53 +1306,93 @@ issue tracker`_ if you believe that your problem has exposed a bug in pip. .. _`Using pip from your program`: +Backtracking user guide information +=================================== + +Related to `GH issue 8713 `__. + +Objective +--------- + +The purpose of this content is to provide documentation of 1) why this +occurs, and 2) how the user can possible solve it. + +Location +-------- + +The end location of this content will be on\ `the pip user +guide `__, somewhere +near the Resolution Impossible section. + +-------------- + Dependency resolution backtracking ================================== Or more commonly known as *"Why does pip download multiple versions of -the same package over and over again?"*. +the same package over and over again during an install?"*. -The purpose of this section is to provide explanation and practical -suggestions to pip users who encounter dependency resolution -backtracking during a ``pip install`` command. - -Why does backtracking occur? ----------------------------- - -With the release of the new resolver (resolver 2020), pip is now more -strict in the package versions it installs, when a users does a -``pip install`` command. This new behaviour means that pip works harder -to find out which version of a package is a good candidate to install. +The purpose of this section is to provide explanation of why +backtracking happens, and practical suggestions to pip users who +encounter dependency resolution backtracking during a ``pip install`` +command. What is backtracking? --------------------- +Backtracking is not a bug, or an unexpected behaviour. It is part of the +way pip's dependency resolution process works. + During a pip install (e.g. ``pip install tea``), pip needs to work out -the packages dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc). +the packages dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc), +and the versions of each of these packages it needs to install. + For each of these dependent packages, it needs to work out which version -of a package is a good candidate to install. When it does, it has a set -of compatible packages which is installs. +of a package is a good candidate to install. + +If everythig goes well, this will result in pip computing a set of +compatible versions of all these packages. + +In the case where a package has a lot of versions, arriving at a good +candidate can take a lot of time. (The amount of time depends on the +package size, the number of versions pip must try, among other things) + +How does backtracking work? +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +pip does this by trying one version, if it finds out it isn’t compatible +it needs to “go back” (backtrack) and download an older version. -If pip tries one version, finds out it isn’t compatible it needs to “go -back” (backtrack) and download an older version. It then tries that, if -it is successful it will continue onto the next package - if not it will -continue to backtrack until it finds a compatible version. +It then tries that version, if it is successful it will continue onto +the next package - if not it will continue to backtrack until it finds a +compatible version. + +At the end of this process, pip will have the set of compatible +packages, which it then installs. This backtrack behaviour can end in 2 ways - either 1) it will successfully find a set packages it can install (good news!), or 2) it -will eventually display `resolution impossible`_ error message (not so -good). +will eventually display `resolution +impossible `__ error +message (not so good). If pip starts backtracking during dependency resolution, it does not know how long it will backtrack, and how much computation would be needed. -.. _resolution impossible: https://pip.pypa.io/en/latest/user_guide/#id35 +Why does backtracking occur? +---------------------------- + +With the release of the new resolver (resolver 2020), pip is now more +strict in the package versions it installs, when a users does a +``pip install`` command. This new behaviour means that pip works harder +to find out which version of a package is a good candidate to install. What does this behaviour look like? ----------------------------------- Right now backtracking looks like this: + :: $ pip install tea==1.9.8 @@ -1405,37 +1445,97 @@ package cup - cup-3.22.0 to cup-3.13.0 - to find a version that will be compatible with the other packages - ``spoon``, ``hot-water``, ``cup`` etc. -## Possible solutions +These multiple ``Downloading cup-version`` lines shows pip backtracking. -====UPDATE FROM HERE====== -:::info -Unsure what possible solutions are. AFAIK there are no solutions - apart 1) waiting til it's finished, or 2) [dealing with potential resolution impossible](https://pip.pypa.io/en/latest/user_guide/#id37). +Possible ways to reduce backtracking occuring +--------------------------------------------- -Pradyun mentions "stricter constraints to reduce runtime". +It's important to mention backtracking behaviour is expected during a +``pip install`` process. What pip is trying to do is complicated - it is +working through potentially millions of package versions to identify the +comptible versions. -Can we help the user with these questions: -- What can I do to reduce the chances of backtracking occuring? -- Backtracking keeps happening with my installs, what can I do? -- Can I delete these unnecessary packages? -::: +There is no guaranteed solution to backtracking but you can reduce it - +here are a number of ways. -Possible solutions ------------------- +.. _1-allow-pip-to-complete-its-backtracking: + +1. Allow pip to complete its backtracking +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In most cases, pip will complete the backtracking process successfully. +It is possible this could take a very long time to complete - this may +not be the preferred option. + +However there is a possibility pip will not be able to find a set of +compatible versions. + +If you'd prefer not to wait, you can interrupt pip (ctrl and c) and use +constraints to reduce the number of package versions it tries. + +.. _2-reduce-the-versions-of-the-backtracking-package: + +2. Reduce the versions of the backtracking package +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If pip is backtracking more than you'd like, the next option is to +constrain the number of package versions it tries. + +A first good candidate for this constraining is the package(s) it is +backtracking on (e.g. in the above example - ``cup``). + +You coud try: + +==\ ``pip install tea cup > 3.13`` please check this syntax== -:::info Unsure what possible solutions are. AFAIK there are no solutions -- apart 1) waiting til it's finished, or 2) `dealing with potential -resolution impossible`_. +This will reduce the number of versions of ``cup`` it tries, and +possibly reduce the time pip takes to install. -Pradyun mentions "stricter constraints to reduce runtime". +There is a possibility that if you're wrong (in this case a newer +version would have worked) then you missed the chance to use it. This +can be trial and error. -Can we help the user with these questions: +.. _3-use-constraint-files-or-lockfiles: -- What can I do to reduce the chances of backtracking occuring? -- Backtracking keeps happening with my installs, what can I do? -- Can I delete these unnecessary packages? ::: +3. Use constraint files or lockfiles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. _dealing with potential resolution impossible: https://pip.pypa.io/en/latest/user_guide/#id37 -=====UPDATE UNTIL HERE====== +This option is a progression of 2 above. It requires users to know how +to inspect: + +- the packages they're are trying to install +- the package release frequency and compatibility policies +- their release notes and changelogs from past versions + +During deployment creating a lockfile stating the exact package and +version number for for each dependency of that package. You can do this +with `pip-tools `__. + +This means the "work" is done once during development process, and so +will save users this work during deployment. + +The pip team is not available to provide support in helping you create a +suitable constraints file. + +.. _4-be-more-strict-on-package-dependencies-during-development: + +4. Be more strict on package dependencies during development +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For package maintainers during the development, give pip some help by +creating constraint files for the dependency tree. This will reduce the +number of versions it will try. + +If you (as the user installing a package) see the package has specific +version dependencies, you could contact the maintainer and ask them to +be more strict with the packages requirements. + +Getting help +------------ + +If none of the suggestions above work for you, we recommend that you ask +for help and you've got `a number of +options `__. Using pip from your program From b49a213b56563831e7981a9a95c09e644e483d69 Mon Sep 17 00:00:00 2001 From: Bernard Date: Fri, 23 Oct 2020 15:16:39 +0200 Subject: [PATCH 03/14] fix syntax --- docs/html/user_guide.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index aecb427f3e4..332ba723d11 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1484,9 +1484,9 @@ constrain the number of package versions it tries. A first good candidate for this constraining is the package(s) it is backtracking on (e.g. in the above example - ``cup``). -You coud try: +You could try: -==\ ``pip install tea cup > 3.13`` please check this syntax== +``pip install tea cup > 3.13`` This will reduce the number of versions of ``cup`` it tries, and possibly reduce the time pip takes to install. From e28f30d0704e56bd99b7937bca9012312cad29ab Mon Sep 17 00:00:00 2001 From: Bernard Date: Fri, 23 Oct 2020 15:26:41 +0200 Subject: [PATCH 04/14] Adding news file for #9309 --- news/9039.doc.rst | 1 + 1 file changed, 1 insertion(+) create mode 100644 news/9039.doc.rst diff --git a/news/9039.doc.rst b/news/9039.doc.rst new file mode 100644 index 00000000000..c87972d70d0 --- /dev/null +++ b/news/9039.doc.rst @@ -0,0 +1 @@ +adding a section to pip user guide to cover pip backtracking. From d185aea3e1a57c4bb81e909e44d807adf3d3f219 Mon Sep 17 00:00:00 2001 From: Bernard Date: Fri, 23 Oct 2020 15:27:20 +0200 Subject: [PATCH 05/14] updating --- news/9039.doc.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/news/9039.doc.rst b/news/9039.doc.rst index c87972d70d0..8f6dde8cba2 100644 --- a/news/9039.doc.rst +++ b/news/9039.doc.rst @@ -1 +1 @@ -adding a section to pip user guide to cover pip backtracking. +adding a section to pip user guide to cover pip backtracking, connected to #8975. From ecda23819baceb5f0986a0bd16ad612998b34553 Mon Sep 17 00:00:00 2001 From: Bernard Date: Mon, 26 Oct 2020 10:14:34 +0100 Subject: [PATCH 06/14] Updating file with PR review edits. --- docs/html/user_guide.rst | 95 +++++++++++++++++++--------------------- 1 file changed, 46 insertions(+), 49 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 332ba723d11..0efb64b89e2 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1306,26 +1306,6 @@ issue tracker`_ if you believe that your problem has exposed a bug in pip. .. _`Using pip from your program`: -Backtracking user guide information -=================================== - -Related to `GH issue 8713 `__. - -Objective ---------- - -The purpose of this content is to provide documentation of 1) why this -occurs, and 2) how the user can possible solve it. - -Location --------- - -The end location of this content will be on\ `the pip user -guide `__, somewhere -near the Resolution Impossible section. - --------------- - Dependency resolution backtracking ================================== @@ -1344,49 +1324,70 @@ Backtracking is not a bug, or an unexpected behaviour. It is part of the way pip's dependency resolution process works. During a pip install (e.g. ``pip install tea``), pip needs to work out -the packages dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc), +the package's dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc), and the versions of each of these packages it needs to install. For each of these dependent packages, it needs to work out which version of a package is a good candidate to install. -If everythig goes well, this will result in pip computing a set of +A good candidate is a version of each package that is compatible with all the +other packages versions being installing in the same command, and works with +other requirements or constraints the user has specified. + +If everything goes well, this will result in pip computing a set of compatible versions of all these packages. In the case where a package has a lot of versions, arriving at a good candidate can take a lot of time. (The amount of time depends on the -package size, the number of versions pip must try, among other things) +package size, the number of versions pip must try, and other concerns.) How does backtracking work? ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -pip does this by trying one version, if it finds out it isn’t compatible -it needs to “go back” (backtrack) and download an older version. +When doing a pip install, it needs to start by making assumptions about the +packages it needs to install. During this install process it needs check this +assumptions as it goes along. -It then tries that version, if it is successful it will continue onto -the next package - if not it will continue to backtrack until it finds a -compatible version. +When it finds that an assumption is incorrect, it has to try another approach +(backtrack), which means discarding some of the work that has already been done, +and going back to choose another path. + +For example; The user requests `pip install tea`. +`tea` has dependencies of `cup`, `hot-water`, amongst others. -At the end of this process, pip will have the set of compatible -packages, which it then installs. +pip starts by installing a version of cup. If it finds out it isn’t compatible +(with the other package versions) it needs to “go back” (backtrack) and +download an older version. + +It then tries to install that version. If it is successful, it will continue +onto the next package. If not it will continue to backtrack until it finds a +compatible version. This backtrack behaviour can end in 2 ways - either 1) it will -successfully find a set packages it can install (good news!), or 2) it -will eventually display `resolution -impossible `__ error +successfully find a set packages it can install (good news!), or 2) it will +eventually display `resolution impossible `__ error message (not so good). If pip starts backtracking during dependency resolution, it does not know how long it will backtrack, and how much computation would be -needed. +needed, which for the user, means it can possibly take a long time to complete. Why does backtracking occur? ---------------------------- -With the release of the new resolver (resolver 2020), pip is now more -strict in the package versions it installs, when a users does a -``pip install`` command. This new behaviour means that pip works harder -to find out which version of a package is a good candidate to install. +With the release of the new resolver (:ref:`Resolver changes 2020`), pip is now +more strict in the package versions it installs when a users runs a +``pip install`` command. + +This new behaviour means that pip works harder to find out which version of a +package is a good candidate to install. It reduces the risk that installing a +new package will accidentally break an existing installed package, and so +reducing the risk of your environment gets messed up. + +Pip needs to backtrack because initially, it doesn't have all the information it +needs to work out the correct set of packages. This is because package indexes +don't provide full package dependency information before you have downloaded +the package. What does this behaviour look like? ----------------------------------- @@ -1440,20 +1441,20 @@ Right now backtracking looks like this: If you want to abort this run, you can press Ctrl + C to do so. Downloading cup-3.13.0-py2.py3-none-any.whl (374 kB) -In the above sample output, pip must downloaded multiple versions of +In the above sample output, pip had to download multiple versions of package cup - cup-3.22.0 to cup-3.13.0 - to find a version that will be compatible with the other packages - ``spoon``, ``hot-water``, ``cup`` etc. These multiple ``Downloading cup-version`` lines shows pip backtracking. -Possible ways to reduce backtracking occuring +Possible ways to reduce backtracking occurring --------------------------------------------- It's important to mention backtracking behaviour is expected during a ``pip install`` process. What pip is trying to do is complicated - it is working through potentially millions of package versions to identify the -comptible versions. +compatible versions. There is no guaranteed solution to backtracking but you can reduce it - here are a number of ways. @@ -1486,12 +1487,12 @@ backtracking on (e.g. in the above example - ``cup``). You could try: -``pip install tea cup > 3.13`` +``pip install tea "cup > 3.13"`` This will reduce the number of versions of ``cup`` it tries, and possibly reduce the time pip takes to install. -There is a possibility that if you're wrong (in this case a newer +There is a possibility that if you're wrong (in this case an older version would have worked) then you missed the chance to use it. This can be trial and error. @@ -1507,8 +1508,8 @@ to inspect: - the package release frequency and compatibility policies - their release notes and changelogs from past versions -During deployment creating a lockfile stating the exact package and -version number for for each dependency of that package. You can do this +During deployment, you can create a lockfile stating the exact package and +version number for for each dependency of that package. You can create this with `pip-tools `__. This means the "work" is done once during development process, and so @@ -1526,10 +1527,6 @@ For package maintainers during the development, give pip some help by creating constraint files for the dependency tree. This will reduce the number of versions it will try. -If you (as the user installing a package) see the package has specific -version dependencies, you could contact the maintainer and ask them to -be more strict with the packages requirements. - Getting help ------------ From 95e78ab284f0f102cff6c3e58194b46875128647 Mon Sep 17 00:00:00 2001 From: Bernard Date: Mon, 26 Oct 2020 13:23:07 +0100 Subject: [PATCH 07/14] Editing to make shorten content (reduce/remove duplication), add anchor. --- docs/html/user_guide.rst | 48 ++++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 27 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 0efb64b89e2..bf2e6359b70 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1304,7 +1304,7 @@ issue tracker`_ if you believe that your problem has exposed a bug in pip. .. _"How do I ask a good question?": https://stackoverflow.com/help/how-to-ask .. _pip issue tracker: https://github.com/pypa/pip/issues -.. _`Using pip from your program`: +.. _`Dependency resolution backtracking`: Dependency resolution backtracking ================================== @@ -1314,8 +1314,7 @@ the same package over and over again during an install?"*. The purpose of this section is to provide explanation of why backtracking happens, and practical suggestions to pip users who -encounter dependency resolution backtracking during a ``pip install`` -command. +encounter it during a ``pip install``. What is backtracking? --------------------- @@ -1324,18 +1323,12 @@ Backtracking is not a bug, or an unexpected behaviour. It is part of the way pip's dependency resolution process works. During a pip install (e.g. ``pip install tea``), pip needs to work out -the package's dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc), -and the versions of each of these packages it needs to install. - -For each of these dependent packages, it needs to work out which version -of a package is a good candidate to install. - -A good candidate is a version of each package that is compatible with all the -other packages versions being installing in the same command, and works with -other requirements or constraints the user has specified. +the package's dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc), the +versions of each of these packages it needs to install. For each of these +it needs to decide which version is a good candidate to install. -If everything goes well, this will result in pip computing a set of -compatible versions of all these packages. +A "good candidate" means a version of each package that is compatible with all +the other package versions being installed at the same time. In the case where a package has a lot of versions, arriving at a good candidate can take a lot of time. (The amount of time depends on the @@ -1345,19 +1338,19 @@ How does backtracking work? ~~~~~~~~~~~~~~~~~~~~~~~~~~~ When doing a pip install, it needs to start by making assumptions about the -packages it needs to install. During this install process it needs check this +packages it needs to install. During the install process it needs check this assumptions as it goes along. When it finds that an assumption is incorrect, it has to try another approach (backtrack), which means discarding some of the work that has already been done, and going back to choose another path. -For example; The user requests `pip install tea`. -`tea` has dependencies of `cup`, `hot-water`, amongst others. +For example; The user requests ``pip install tea``. ```tea`` has dependencies of +``cup``, ``hot-water``, ``spoon`` amongst others. -pip starts by installing a version of cup. If it finds out it isn’t compatible -(with the other package versions) it needs to “go back” (backtrack) and -download an older version. +pip starts by installing a version of ``cup``. If it finds out it isn’t +compatible (with the other package versions) it needs to “go back” +(backtrack) and download an older version. It then tries to install that version. If it is successful, it will continue onto the next package. If not it will continue to backtrack until it finds a @@ -1370,7 +1363,7 @@ message (not so good). If pip starts backtracking during dependency resolution, it does not know how long it will backtrack, and how much computation would be -needed, which for the user, means it can possibly take a long time to complete. +needed. For the user this means it can take a long time to complete. Why does backtracking occur? ---------------------------- @@ -1379,20 +1372,20 @@ With the release of the new resolver (:ref:`Resolver changes 2020`), pip is now more strict in the package versions it installs when a users runs a ``pip install`` command. -This new behaviour means that pip works harder to find out which version of a -package is a good candidate to install. It reduces the risk that installing a -new package will accidentally break an existing installed package, and so -reducing the risk of your environment gets messed up. - Pip needs to backtrack because initially, it doesn't have all the information it needs to work out the correct set of packages. This is because package indexes don't provide full package dependency information before you have downloaded the package. +This new resolver behaviour means that pip works harder to find out which +version of a package is a good candidate to install. It reduces the risk that +installing a new package will accidentally break an existing installed package, +and so reducing the risk of your environment gets messed up. + What does this behaviour look like? ----------------------------------- -Right now backtracking looks like this: +Right now backtracking behaviour looks like this: :: @@ -1534,6 +1527,7 @@ If none of the suggestions above work for you, we recommend that you ask for help and you've got `a number of options `__. +.. _`Using pip from your program`: Using pip from your program =========================== From ca63189b606fb137c66a40758e643f1592304924 Mon Sep 17 00:00:00 2001 From: Bernard Tyers Date: Mon, 26 Oct 2020 14:52:58 +0100 Subject: [PATCH 08/14] Update news/9039.doc.rst Co-authored-by: Pradyun Gedam <3275593+pradyunsg@users.noreply.github.com> --- news/9039.doc.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/news/9039.doc.rst b/news/9039.doc.rst index 8f6dde8cba2..37241250dac 100644 --- a/news/9039.doc.rst +++ b/news/9039.doc.rst @@ -1 +1 @@ -adding a section to pip user guide to cover pip backtracking, connected to #8975. +Add a section to the User Guide to cover backtracking during dependency resolution. From e3675c5a936c7dc1eb301a348a874e6fe39b7852 Mon Sep 17 00:00:00 2001 From: Bernard Date: Wed, 28 Oct 2020 12:56:45 +0100 Subject: [PATCH 09/14] Committing edits. --- docs/html/user_guide.rst | 72 +++++++++++++++++++++------------------- 1 file changed, 37 insertions(+), 35 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index bf2e6359b70..387a8bf61ac 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1276,6 +1276,8 @@ In this situation, you could consider: - Refactoring your project to reduce the number of dependencies (for example, by breaking up a monolithic code base into smaller pieces) +.. _`Getting help`: + Getting help ------------ @@ -1338,7 +1340,7 @@ How does backtracking work? ~~~~~~~~~~~~~~~~~~~~~~~~~~~ When doing a pip install, it needs to start by making assumptions about the -packages it needs to install. During the install process it needs check this +packages it needs to install. During the install process it needs to check these assumptions as it goes along. When it finds that an assumption is incorrect, it has to try another approach @@ -1357,7 +1359,7 @@ onto the next package. If not it will continue to backtrack until it finds a compatible version. This backtrack behaviour can end in 2 ways - either 1) it will -successfully find a set packages it can install (good news!), or 2) it will +successfully find a set of packages it can install (good news!), or 2) it will eventually display `resolution impossible `__ error message (not so good). @@ -1369,7 +1371,7 @@ Why does backtracking occur? ---------------------------- With the release of the new resolver (:ref:`Resolver changes 2020`), pip is now -more strict in the package versions it installs when a users runs a +more strict in the package versions it installs when a user runs a ``pip install`` command. Pip needs to backtrack because initially, it doesn't have all the information it @@ -1380,7 +1382,7 @@ the package. This new resolver behaviour means that pip works harder to find out which version of a package is a good candidate to install. It reduces the risk that installing a new package will accidentally break an existing installed package, -and so reducing the risk of your environment gets messed up. +and so reducing the risk that your environment gets messed up. What does this behaviour look like? ----------------------------------- @@ -1391,55 +1393,55 @@ Right now backtracking behaviour looks like this: $ pip install tea==1.9.8 Collecting tea==1.9.8 - Downloading tea-1.9.8-py2.py3-none-any.whl (346 kB) - |████████████████████████████████| 346 kB 10.4 MB/s + Downloading tea-1.9.8-py2.py3-none-any.whl (346 kB) + |████████████████████████████████| 346 kB 10.4 MB/s Collecting spoon==2.27.0 - Downloading spoon-2.27.0-py2.py3-none-any.whl (312 kB) - |████████████████████████████████| 312 kB 19.2 MB/s + Downloading spoon-2.27.0-py2.py3-none-any.whl (312 kB) + |████████████████████████████████| 312 kB 19.2 MB/s Collecting hot-water>=0.1.9 Downloading hot-water-0.1.13-py3-none-any.whl (9.3 kB) Collecting cup>=1.6.0 - Downloading cup-3.22.0-py2.py3-none-any.whl (397 kB) - |████████████████████████████████| 397 kB 28.2 MB/s + Downloading cup-3.22.0-py2.py3-none-any.whl (397 kB) + |████████████████████████████████| 397 kB 28.2 MB/s INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while. - Downloading cup-3.21.0-py2.py3-none-any.whl (395 kB) - |████████████████████████████████| 395 kB 27.0 MB/s - Downloading cup-3.20.0-py2.py3-none-any.whl (394 kB) - |████████████████████████████████| 394 kB 24.4 MB/s - Downloading cup-3.19.1-py2.py3-none-any.whl (394 kB) - |████████████████████████████████| 394 kB 21.3 MB/s - Downloading cup-3.19.0-py2.py3-none-any.whl (394 kB) - |████████████████████████████████| 394 kB 26.2 MB/s - Downloading cup-3.18.0-py2.py3-none-any.whl (393 kB) - |████████████████████████████████| 393 kB 22.1 MB/s - Downloading cup-3.17.0-py2.py3-none-any.whl (382 kB) - |████████████████████████████████| 382 kB 23.8 MB/s - Downloading cup-3.16.0-py2.py3-none-any.whl (376 kB) - |████████████████████████████████| 376 kB 27.5 MB/s - Downloading cup-3.15.1-py2.py3-none-any.whl (385 kB) - |████████████████████████████████| 385 kB 30.4 MB/s + Downloading cup-3.21.0-py2.py3-none-any.whl (395 kB) + |████████████████████████████████| 395 kB 27.0 MB/s + Downloading cup-3.20.0-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 24.4 MB/s + Downloading cup-3.19.1-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 21.3 MB/s + Downloading cup-3.19.0-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 26.2 MB/s + Downloading cup-3.18.0-py2.py3-none-any.whl (393 kB) + |████████████████████████████████| 393 kB 22.1 MB/s + Downloading cup-3.17.0-py2.py3-none-any.whl (382 kB) + |████████████████████████████████| 382 kB 23.8 MB/s + Downloading cup-3.16.0-py2.py3-none-any.whl (376 kB) + |████████████████████████████████| 376 kB 27.5 MB/s + Downloading cup-3.15.1-py2.py3-none-any.whl (385 kB) + |████████████████████████████████| 385 kB 30.4 MB/s INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while. - Downloading cup-3.15.0-py2.py3-none-any.whl (378 kB) - |████████████████████████████████| 378 kB 21.4 MB/s - Downloading cup-3.14.0-py2.py3-none-any.whl (372 kB) - |████████████████████████████████| 372 kB 21.1 MB/s - Downloading cup-3.13.1-py2.py3-none-any.whl (381 kB) - |████████████████████████████████| 381 kB 21.8 MB/s + Downloading cup-3.15.0-py2.py3-none-any.whl (378 kB) + |████████████████████████████████| 378 kB 21.4 MB/s + Downloading cup-3.14.0-py2.py3-none-any.whl (372 kB) + |████████████████████████████████| 372 kB 21.1 MB/s + Downloading cup-3.13.1-py2.py3-none-any.whl (381 kB) + |████████████████████████████████| 381 kB 21.8 MB/s This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. - Downloading cup-3.13.0-py2.py3-none-any.whl (374 kB) + Downloading cup-3.13.0-py2.py3-none-any.whl (374 kB) In the above sample output, pip had to download multiple versions of package cup - cup-3.22.0 to cup-3.13.0 - to find a version that will be compatible with the other packages - ``spoon``, ``hot-water``, ``cup`` etc. -These multiple ``Downloading cup-version`` lines shows pip backtracking. +These multiple ``Downloading cup-version`` lines show pip backtracking. Possible ways to reduce backtracking occurring --------------------------------------------- @@ -1525,7 +1527,7 @@ Getting help If none of the suggestions above work for you, we recommend that you ask for help and you've got `a number of -options `__. +options :ref:`Getting help`. .. _`Using pip from your program`: From 0cf00929816acc6303df368d5f657f584d610cad Mon Sep 17 00:00:00 2001 From: Bernard Tyers Date: Wed, 28 Oct 2020 14:30:04 +0100 Subject: [PATCH 10/14] Apply suggestions from code review Co-authored-by: Pradyun Gedam <3275593+pradyunsg@users.noreply.github.com> Co-authored-by: Sumana Harihareswara --- docs/html/user_guide.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 387a8bf61ac..ca3a58cb67b 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1326,8 +1326,8 @@ way pip's dependency resolution process works. During a pip install (e.g. ``pip install tea``), pip needs to work out the package's dependencies (e.g. ``spoon``, ``hot-water``, ``cup`` etc), the -versions of each of these packages it needs to install. For each of these -it needs to decide which version is a good candidate to install. +versions of each of these packages it needs to install. For each package +pip needs to decide which version is a good candidate to install. A "good candidate" means a version of each package that is compatible with all the other package versions being installed at the same time. @@ -1339,11 +1339,11 @@ package size, the number of versions pip must try, and other concerns.) How does backtracking work? ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When doing a pip install, it needs to start by making assumptions about the +When doing a pip install, pip starts by making assumptions about the packages it needs to install. During the install process it needs to check these assumptions as it goes along. -When it finds that an assumption is incorrect, it has to try another approach +When pip finds that an assumption is incorrect, it has to try another approach (backtrack), which means discarding some of the work that has already been done, and going back to choose another path. @@ -1360,7 +1360,7 @@ compatible version. This backtrack behaviour can end in 2 ways - either 1) it will successfully find a set of packages it can install (good news!), or 2) it will -eventually display `resolution impossible `__ error +eventually display a `resolution impossible `__ error message (not so good). If pip starts backtracking during dependency resolution, it does not @@ -1382,7 +1382,9 @@ the package. This new resolver behaviour means that pip works harder to find out which version of a package is a good candidate to install. It reduces the risk that installing a new package will accidentally break an existing installed package, -and so reducing the risk that your environment gets messed up. +and so reduces the risk that your environment gets messed up. + +Please address this. What does this behaviour look like? ----------------------------------- @@ -1499,7 +1501,7 @@ can be trial and error. This option is a progression of 2 above. It requires users to know how to inspect: -- the packages they're are trying to install +- the packages they're trying to install - the package release frequency and compatibility policies - their release notes and changelogs from past versions @@ -1526,8 +1528,7 @@ Getting help ------------ If none of the suggestions above work for you, we recommend that you ask -for help and you've got `a number of -options :ref:`Getting help`. +for help and you've got `a number of options :ref:`Getting help`. .. _`Using pip from your program`: From 29d1f4506e98814510cc4e67a5a1792d521360f4 Mon Sep 17 00:00:00 2001 From: Bernard Date: Fri, 30 Oct 2020 12:31:01 +0100 Subject: [PATCH 11/14] Adding link to Constraints Files --- docs/html/user_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index ca3a58cb67b..925a0273b50 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1469,7 +1469,7 @@ However there is a possibility pip will not be able to find a set of compatible versions. If you'd prefer not to wait, you can interrupt pip (ctrl and c) and use -constraints to reduce the number of package versions it tries. +:ref:`Constraints Files`: to reduce the number of package versions it tries. .. _2-reduce-the-versions-of-the-backtracking-package: From 00f3d57131b953c01b3c94a6a1e4ae7961f3948c Mon Sep 17 00:00:00 2001 From: Bernard Date: Fri, 30 Oct 2020 12:35:52 +0100 Subject: [PATCH 12/14] Final heading edit. --- docs/html/user_guide.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 925a0273b50..bd6e2baced1 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1473,8 +1473,8 @@ If you'd prefer not to wait, you can interrupt pip (ctrl and c) and use .. _2-reduce-the-versions-of-the-backtracking-package: -2. Reduce the versions of the backtracking package -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +2. Reduce the number of versions pip will try to backtrack through +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If pip is backtracking more than you'd like, the next option is to constrain the number of package versions it tries. From 8383249442660cfc9609b69acbc2db2351cf3a45 Mon Sep 17 00:00:00 2001 From: Bernard Date: Wed, 11 Nov 2020 11:20:29 +0100 Subject: [PATCH 13/14] Incorporating commit: e6acca646abcaaac7515da9b964d5b5291264142 --- docs/html/user_guide.rst | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index bd6e2baced1..7401720ba64 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1337,7 +1337,7 @@ candidate can take a lot of time. (The amount of time depends on the package size, the number of versions pip must try, and other concerns.) How does backtracking work? -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^ When doing a pip install, pip starts by making assumptions about the packages it needs to install. During the install process it needs to check these @@ -1384,8 +1384,6 @@ version of a package is a good candidate to install. It reduces the risk that installing a new package will accidentally break an existing installed package, and so reduces the risk that your environment gets messed up. -Please address this. - What does this behaviour look like? ----------------------------------- @@ -1439,14 +1437,13 @@ Right now backtracking behaviour looks like this: Downloading cup-3.13.0-py2.py3-none-any.whl (374 kB) In the above sample output, pip had to download multiple versions of -package cup - cup-3.22.0 to cup-3.13.0 - to find a version that will be -compatible with the other packages - ``spoon``, ``hot-water``, ``cup`` -etc. +package ``cup`` - cup-3.22.0 to cup-3.13.0 - to find a version that will be +compatible with the other packages - ``spoon``, ``hot-water``, etc. These multiple ``Downloading cup-version`` lines show pip backtracking. Possible ways to reduce backtracking occurring ---------------------------------------------- +---------------------------------------------- It's important to mention backtracking behaviour is expected during a ``pip install`` process. What pip is trying to do is complicated - it is @@ -1459,7 +1456,7 @@ here are a number of ways. .. _1-allow-pip-to-complete-its-backtracking: 1. Allow pip to complete its backtracking -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In most cases, pip will complete the backtracking process successfully. It is possible this could take a very long time to complete - this may @@ -1474,7 +1471,7 @@ If you'd prefer not to wait, you can interrupt pip (ctrl and c) and use .. _2-reduce-the-versions-of-the-backtracking-package: 2. Reduce the number of versions pip will try to backtrack through -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If pip is backtracking more than you'd like, the next option is to constrain the number of package versions it tries. @@ -1496,7 +1493,7 @@ can be trial and error. .. _3-use-constraint-files-or-lockfiles: 3. Use constraint files or lockfiles -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This option is a progression of 2 above. It requires users to know how to inspect: @@ -1518,7 +1515,7 @@ suitable constraints file. .. _4-be-more-strict-on-package-dependencies-during-development: 4. Be more strict on package dependencies during development -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For package maintainers during the development, give pip some help by creating constraint files for the dependency tree. This will reduce the From 90925a71574707749e8743a5001d658fd0237e23 Mon Sep 17 00:00:00 2001 From: Bernard Date: Wed, 11 Nov 2020 12:59:37 +0100 Subject: [PATCH 14/14] =?UTF-8?q?Fixed=20linting=20failures.=20?= =?UTF-8?q?=F0=9F=A4=9E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/html/user_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 7401720ba64..a6ea8f31460 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1525,7 +1525,7 @@ Getting help ------------ If none of the suggestions above work for you, we recommend that you ask -for help and you've got `a number of options :ref:`Getting help`. +for help. :ref:`Getting help`. .. _`Using pip from your program`: