Resolving dependencies in 2.0 takes forever #9956
Comments
ace-han
added
kind/bug
|
poetry 1.8.5 also takes a long time to resolve this one for me. most likely all that has happened is that the cache format has changed in the new release, so poetry 2.0 cannot use your poetry 1.8.5 cache. In that case the second and subsequent runs will be fast again. |
@dimbleby I was setting up a brand new docker env (no poetry installed before), so the situation you mentioned is not my case. |
|
well then I do not reproduce: poetry 1.8.5 also takes a long time to resolve this one for me (with an empty cache). I see no difference in the new release. |
|
either way the usual problem in such cases is awscli / boto3 / botocore - Amazon like to release thousands of versions of the last of these, which can cause resolvers to have to do a lot of backtracking. Probably if you raise the lower bounds on Amazon-ecosystem dependencies that will reduce your exposure to this. |
|
Take it back, this actually is a regression - though one complicated by the problem being slow to solve even without the change. Here is a simplified repro: [tool.poetry]
name = "foo"
version = "1.0.0"
description = ""
authors = []
keywords = []
license = "MIT"
readme = "README.md"
[tool.poetry.dependencies]
python = ">=3.8,<3.11"
opencv-python = {version = "^4.6.0.66", optional = true}
ale-py = { version = "0.8.1", optional = true }
[tool.poetry.extras]
extra01 = ["ale-py"]
extra02 = ["ale-py"]
extra03 = ["ale-py"]
extra04 = ["ale-py"]
extra05 = ["ale-py"]
extra06 = ["ale-py"]
extra07 = ["ale-py"]
extra08 = ["ale-py"]
extra09 = ["ale-py"]
extra10 = ["ale-py"]
extra11 = ["ale-py"]
extra12 = ["ale-py"]
extra13 = ["ale-py"]
extra14 = ["ale-py"]
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"and we are getting stuck in the new code that wants to maintain markers during resolution so that it can write them to the lock file #9427. Because: the markers that we calculate are long and unwieldy, and converting them to and from CNF and DNF is exponentially expensive |
|
Can confirm there seems to be some regression going on. When generating the lock file from the following Poetry Installation Method: pipx
|
|
You may try #10018. As you can see in the PR description it does not fix the regression, it only mitigates it. The performance might be enough for simpler cases. However, it is also possible that it makes no relevant difference for your use case... |
|
After investigating the issue a little further, it seems like the performance regression can be fixed (at least for the simplified repro) if we improve extra handling in _merge_single_markers, i.e. support |
|
@radoering thank you for this update. |
|
With python-poetry/poetry-core#818, at least the simple repro from #9956 (comment) is almost as fast as before. @aliberts You can try the main branch of Poetry if you like. (It should be significantly faster than 2.0.1 but it might still be slower than 1.8.) |
Or better wait for python-poetry/poetry-core#821. This should make it about as fast as with 1.8. (See PR description.) |
|
Hi, I must add another case that running poetry 2.0.1 also takes forever on repo https://github.com/dlt-hub/dlt Hope it helps for another case to test after new changes:
|
|
I checked the examples mentioned in this issue with the main branch of poetry (and poetry-core) and there still seems to be a regression in some use cases:
At first glance, it looks like the resolver takes different routes, but I still do not know why. (#8256 was long before Poetry 1.8) |
|
After having investigated the recursion error mentioned in my previous comment, I got some new insights: The recursion errorThe recursion error is caused by python-poetry/poetry-core#821 and will be fixed by python-poetry/poetry-core#832. The resolver taking another routeThe resolver takes a different route because of poetry/src/poetry/puzzle/provider.py Lines 532 to 541 in 9507528 With the change the duplicates become: and are not merged anymore causing an early split in the dependency graph. While this can lead to more accurate results, it can take quite a lot of time. Avoiding unnecessary overridesWhile searching the recursion error, I noticed that we often raise
Open pointsThe performance regression of the first example is still present. This might also be caused by #9553 but needs to be investigated further. |
|
The original example looks vulnerable to the usual Amazon ecosystem nonsense. If you have reached the point where that is happening then it is likely time just to advise that OP should tighten the allowed range of awscli / boto versions. |
It is - and #9553 disturbs the balance in this case. With Poetry 1.8, With Poetry 2.0, We could fiddle with poetry/src/poetry/mixology/version_solver.py Line 464 in 8e65cbc At first, I thought to take
However, it seems to increase the time to resolve the last example significantly (> 100 s). Another option might be - is_specific_marker = not dependency.marker.is_any()
+ is_specific_marker = not dependency.marker.without_extras().is_any()This should mostly restore the 1.8 heuristics. This also reduces the time to resolve the first example to 20 s. However, it still slows down the last example by about 50 s. This all feels a bit like gambling although it makes a little sense to me to ignore extra markers when choosing the next package. What do you think @dimbleby? Keep it as is or does any change in the heuristics make sense to you? |
|
@axellpadilla Btw, you can reduce the time for locking if you use the same constraint in https://github.com/dlt-hub/dlt/blob/9ea3812d67ed3e94ae9cb59f86c66384a443eb0d/pyproject.toml#L64 and https://github.com/dlt-hub/dlt/blob/9ea3812d67ed3e94ae9cb59f86c66384a443eb0d/pyproject.toml#L282 (in case the difference is not important to you). |
|
My instinct would be towards simplification. The There are always going to be cases where (eg) a boto3 dependency is discovered relatively late in the search, after we have already chosen (eg) a recent urllib3. Making sensible efforts to handle the obvious or common examples is good, but I would not want to try too hard to tune heuristics to specific examples. Sometimes the best answer is just going to be "if you want a faster solve, please make your constraints less relaxed" |
Description
Install cleanrl 's dependencies via
poetry installtakes forever to resolve dependenciesWorkarounds
using
poetry==1.8.5works finePoetry Installation Method
install.python-poetry.org
Operating System
Ubuntu 22.04.2 LTS
Poetry Version
2.0.0
Poetry Configuration
Python Sysconfig
sysconfig.log
Example pyproject.toml
Poetry Runtime Logs
Super large. Can not upload. generated via `poetry install -vvv | tee /tmp/p.log` root@dsw-498793-6c5c78668-ps57p:/tmp# ll -trh 总计 132M drwxr-xr-x 1 root root 4.0K Jan 6 09:21 ../ drwx------ 2 root root 4.0K Jan 6 13:57 tmpfbxagxq5_kernels/ -rw-r--r-- 1 root root 0 Jan 6 13:57 wow_what_a_coincidence -rw------- 1 root root 0 Jan 6 13:57 stdout-stderr---supervisor-oi3so2gc.log drwx------ 33 root root 12K Jan 6 13:57 _MEIZhlMUp/ drwx------ 3 root root 4.0K Jan 6 13:57 tmp9omelb31/ -rw------- 1 root root 30K Jan 6 15:05 stdout-stdout---supervisor-zagcr3vz.log srwxr-xr-x 1 root root 0 Jan 6 15:05 vscode-ipc-8702339b-3baa-4dfe-8dfd-26f2b4d1d897.sock= srwxr-xr-x 1 root root 0 Jan 6 15:05 vscode-git-8511e23733.sock= drwxrwxrwx 5 root root 4.0K Jan 6 15:33 ./ -rw-r--r-- 1 root root 132M Jan 6 15:34 p.logThe text was updated successfully, but these errors were encountered: