Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New Resolver: Do not attempt to use an sdist for the same version after a wheel does not match #8129

Closed
pradyunsg opened this issue Apr 24, 2020 · 7 comments
Closed

New Resolver: Do not attempt to use an sdist for the same version after a wheel does not match #8129

pradyunsg opened this issue Apr 24, 2020 · 7 comments
Labels
type: bug A confirmed bug or unintended behavior

Comments

@pradyunsg
Copy link
Member

Environment

  • pip version: master
  • Python version: 3.8
  • OS: MacOS

Description

pip attempts to download and build an sdist for a package during resolution, when a wheel is not compatible.

Collecting python-keystoneclient==3.22.0
  Downloading python_keystoneclient-3.22.0-py2.py3-none-any.whl (397 kB)
     |████████████████████████████████| 397 kB 9.2 MB/s 
Collecting https://files.pythonhosted.org/packages/f8/f6/c54a3e0ce02dac89f23b35ef73f17f803dda02051030f95b2cfa77a9b134/python-keystoneclient-3.22.0.tar.gz#sha256=6e2b6d2a5ae5d7aa26d4e52d1c682e08417d2c5d73ccc54cb65c54903a868cb4 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.22.0.tar.gz (316 kB)
     |████████████████████████████████| 316 kB 11.6 MB/s 
Collecting python-keystoneclient==3.21.0
  Downloading python_keystoneclient-3.21.0-py2.py3-none-any.whl (395 kB)
     |████████████████████████████████| 395 kB 15.0 MB/s 
Collecting https://files.pythonhosted.org/packages/e8/13/adef3f40d41fb8258f65b81d37adeff48109af7d95ae351457786509922e/python-keystoneclient-3.21.0.tar.gz#sha256=1a34aa80bc65bbe9b3beb24df317e773d98a270cdd53a307fd5c00cfa833ea0e (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.21.0.tar.gz (318 kB)
     |████████████████████████████████| 318 kB 37.8 MB/s

Expected behavior

How to Reproduce

  1. Create a clean virtual environment.
  2. Run pip install pyrax==1.9.8 --unstable-feature=resolver
  3. Watch your CPU and network adapter use power one-after-the-other. :)

Output

$ pip install pyrax==1.9.8 --unstable-feature=resolver                             
Collecting pyrax==1.9.8
  Downloading pyrax-1.9.8-py2.py3-none-any.whl (346 kB)
     |████████████████████████████████| 346 kB 5.6 MB/s 
Requirement already satisfied: six==1.14.0 in /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages (from pyrax==1.9.8) (1.14.0)
Collecting python-novaclient==2.27.0
  Downloading python_novaclient-2.27.0-py2.py3-none-any.whl (312 kB)
     |████████████████████████████████| 312 kB 13.5 MB/s 
Collecting https://files.pythonhosted.org/packages/23/4a/9785a37ed6425918af69909af715ced0fa261e518601a0c70309a708fd08/prettytable-0.7.2.zip#sha256=a53da3b43d7a5c229b5e3ca2892ef982c46b7923b51e98f0db49956531211c4f (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading prettytable-0.7.2.zip (28 kB)
Collecting iso8601==0.1.12
  Downloading iso8601-0.1.12-py3-none-any.whl (12 kB)
Collecting https://files.pythonhosted.org/packages/2a/fc/2c31fea5bc50cd5a849d9fa61343e95af8e2033b35f2650755dcc5365ff1/rackspace-novaclient-2.1.tar.gz#sha256=22fc44f623bae0feb32986ec4630abee904e4c96fba5849386a87e88c450eae7 (from pyrax==1.9.8)
  Downloading rackspace-novaclient-2.1.tar.gz (1.7 kB)
Collecting https://files.pythonhosted.org/packages/01/4e/230d9334ea61efb16dda8bef558fd11f8623f6f3ced8a0cf68559435b125/ip_associations_python_novaclient_ext-0.2.tar.gz#sha256=e4576c3ee149bcca7e034507ad9c698cb07dd9fa10f90056756aea0fa59bae37 (from rackspace-novaclient==2.1->pyrax==1.9.8)
  Downloading ip_associations_python_novaclient_ext-0.2.tar.gz (2.4 kB)
Collecting https://files.pythonhosted.org/packages/3c/14/8932bf613797715bf1fe42b00d413025aac9414cf35bacca091a9191155a/rackspace-auth-openstack-1.3.tar.gz#sha256=c4c069eeb1924ea492c50144d8a4f5f1eb0ece945e0c0d60157cabcadff651cd (from rackspace-novaclient==2.1->pyrax==1.9.8)
  Downloading rackspace-auth-openstack-1.3.tar.gz (6.8 kB)
Collecting https://files.pythonhosted.org/packages/a9/2c/306ef3376bee5fda62c1255da05db2efedc8276be5be98180dbd224d9949/os_diskconfig_python_novaclient_ext-0.1.3.tar.gz#sha256=e7d19233a7b73c70244d2527d162d8176555698e7c621b41f689be496df15e75 (from rackspace-novaclient==2.1->pyrax==1.9.8)
  Downloading os_diskconfig_python_novaclient_ext-0.1.3.tar.gz (2.5 kB)
Collecting https://files.pythonhosted.org/packages/ef/3c/9cd2453e85979f15316953a37a93d5500d8f70046b501b13766c58cf1310/rax_scheduled_images_python_novaclient_ext-0.3.1.tar.gz#sha256=f170cf97b20bdc8a1784cc0b85b70df5eb9b88c3230dab8e68e1863bf3937cdb (from rackspace-novaclient==2.1->pyrax==1.9.8)
  Downloading rax_scheduled_images_python_novaclient_ext-0.3.1.tar.gz (2.9 kB)
Collecting https://files.pythonhosted.org/packages/9e/86/6ec4aa97a5e426034f8cc5657186e18303ffb89f37e71375ee0b342b7b78/os_networksv2_python_novaclient_ext-0.26.tar.gz#sha256=613a75216d98d3ce6bb413f717323e622386c24fc9cc66148507539e7dc5bf19 (from rackspace-novaclient==2.1->pyrax==1.9.8)
  Downloading os_networksv2_python_novaclient_ext-0.26.tar.gz (2.3 kB)
Collecting https://files.pythonhosted.org/packages/36/cf/80aeb67615503898b6b870f17ee42a4e87f1c861798c32665c25d9c0494d/rax_default_network_flags_python_novaclient_ext-0.4.0.tar.gz#sha256=852bf49d90e7a1bc16aa0b25b46a45ba5654069f7321a363c8d94c5496666001 (from rackspace-novaclient==2.1->pyrax==1.9.8)
  Downloading rax_default_network_flags_python_novaclient_ext-0.4.0.tar.gz (2.5 kB)
Collecting https://files.pythonhosted.org/packages/db/33/d5e87b099c9d394a966051cde526c9fcfdd46a51762a8054c98d3ae3b464/os_virtual_interfacesv2_python_novaclient_ext-0.20.tar.gz#sha256=6d39ff4174496a0f795d11f20240805a16bbf452091cf8eb9bd1d5ae2fca449d (from rackspace-novaclient==2.1->pyrax==1.9.8)
  Downloading os_virtual_interfacesv2_python_novaclient_ext-0.20.tar.gz (2.6 kB)
Collecting pbr==1.10.0
  Downloading pbr-1.10.0-py2.py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 10.3 MB/s 
Collecting argparse==1.4.0
  Using cached argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Collecting Babel==2.8.0
  Using cached Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)
Collecting mock==4.0.2
  Using cached mock-4.0.2-py3-none-any.whl (28 kB)
Collecting https://files.pythonhosted.org/packages/98/87/a7b98aa9256c8843f92878966dc3d8d914c14aad97e2c5ce4798d5743e07/simplejson-3.17.0.tar.gz#sha256=2b4b2b738b3b99819a17feaf118265d0753d5536049ea570b3c43b51c4701e81 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Using cached simplejson-3.17.0.tar.gz (83 kB)
Collecting pytz==2019.3
  Using cached pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting python-keystoneclient==4.0.0
  Downloading python_keystoneclient-4.0.0-py3-none-any.whl (397 kB)
     |████████████████████████████████| 397 kB 13.2 MB/s 
Collecting https://files.pythonhosted.org/packages/c9/ec/5cce3af48ac2bd891e1ff7dcaffa2d7322b4438f2324fc3c1d0125c6cd10/python-keystoneclient-4.0.0.tar.gz#sha256=6d93efd494b43d8b4cd8a62281c82d3f02aa531c5523e6bbe7d696e37bc77ba8 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-4.0.0.tar.gz (313 kB)
     |████████████████████████████████| 313 kB 12.9 MB/s 
Collecting python-keystoneclient==3.22.0
  Downloading python_keystoneclient-3.22.0-py2.py3-none-any.whl (397 kB)
     |████████████████████████████████| 397 kB 9.2 MB/s 
Collecting https://files.pythonhosted.org/packages/f8/f6/c54a3e0ce02dac89f23b35ef73f17f803dda02051030f95b2cfa77a9b134/python-keystoneclient-3.22.0.tar.gz#sha256=6e2b6d2a5ae5d7aa26d4e52d1c682e08417d2c5d73ccc54cb65c54903a868cb4 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.22.0.tar.gz (316 kB)
     |████████████████████████████████| 316 kB 11.6 MB/s 
Collecting python-keystoneclient==3.21.0
  Downloading python_keystoneclient-3.21.0-py2.py3-none-any.whl (395 kB)
     |████████████████████████████████| 395 kB 15.0 MB/s 
Collecting https://files.pythonhosted.org/packages/e8/13/adef3f40d41fb8258f65b81d37adeff48109af7d95ae351457786509922e/python-keystoneclient-3.21.0.tar.gz#sha256=1a34aa80bc65bbe9b3beb24df317e773d98a270cdd53a307fd5c00cfa833ea0e (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.21.0.tar.gz (318 kB)
     |████████████████████████████████| 318 kB 37.8 MB/s 
Collecting python-keystoneclient==3.20.0
  Downloading python_keystoneclient-3.20.0-py2.py3-none-any.whl (394 kB)
     |████████████████████████████████| 394 kB 12.2 MB/s 
Collecting https://files.pythonhosted.org/packages/8d/63/b2e9327c4542f75d707f7871e27e0b30dcab6b59832ad1b24a4b523a1577/python-keystoneclient-3.20.0.tar.gz#sha256=477ecf61d425672f04fe7b98f260861d5793ca308493b1a520a8b6a8c57fb203 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.20.0.tar.gz (314 kB)
     |████████████████████████████████| 314 kB 11.6 MB/s 
Collecting python-keystoneclient==3.19.1
  Downloading python_keystoneclient-3.19.1-py2.py3-none-any.whl (394 kB)
     |████████████████████████████████| 394 kB 11.9 MB/s 
Collecting https://files.pythonhosted.org/packages/a9/b3/9d61eb52b3ef61751eb1728f0da48fd0ecc3297c44bed51fd22962782b6c/python-keystoneclient-3.19.1.tar.gz#sha256=c3c91d7f4b8e04c09ad983407c325df100b56e950d099a8ef4babd6df2d387c8 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.19.1.tar.gz (315 kB)
     |████████████████████████████████| 315 kB 10.7 MB/s 
Collecting python-keystoneclient==3.19.0
  Downloading python_keystoneclient-3.19.0-py2.py3-none-any.whl (394 kB)
     |████████████████████████████████| 394 kB 11.7 MB/s 
Collecting https://files.pythonhosted.org/packages/0d/e8/c8dc43c31efeff9df29f9a2d3990ac80742e5918c450341f899aabf35eb0/python-keystoneclient-3.19.0.tar.gz#sha256=097ac07ee4f9956cb637abee42a18094b7f916a975f53b7c8fb0d8429fd529be (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.19.0.tar.gz (314 kB)
     |████████████████████████████████| 314 kB 12.9 MB/s 
Collecting python-keystoneclient==3.18.0
  Downloading python_keystoneclient-3.18.0-py2.py3-none-any.whl (393 kB)
     |████████████████████████████████| 393 kB 33.7 MB/s 
Collecting https://files.pythonhosted.org/packages/4a/d7/7ee2efca36b975da19b5e2ae76a6dec38a38f8fb5cc92145aa9c58903312/python-keystoneclient-3.18.0.tar.gz#sha256=2825f3bf20f8b4f6b43bdf393462de3d4b116135b02bb4059960c7e9caf638d3 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.18.0.tar.gz (312 kB)
     |████████████████████████████████| 312 kB 38.5 MB/s 
Collecting python-keystoneclient==3.17.0
  Downloading python_keystoneclient-3.17.0-py2.py3-none-any.whl (382 kB)
     |████████████████████████████████| 382 kB 9.7 MB/s 
Collecting https://files.pythonhosted.org/packages/f0/b4/f918f4873f1a3d4f7a00e874cddcc1bb67dc5ec1ce58166d96d42738d8fe/python-keystoneclient-3.17.0.tar.gz#sha256=7fb770e194760fa3508e758e6ad316fc55d5b4ff97aa688867ef50f62f687624 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.17.0.tar.gz (308 kB)
     |████████████████████████████████| 308 kB 11.7 MB/s 
Collecting python-keystoneclient==3.16.0
  Downloading python_keystoneclient-3.16.0-py2.py3-none-any.whl (376 kB)
     |████████████████████████████████| 376 kB 12.9 MB/s 
Collecting https://files.pythonhosted.org/packages/17/2f/87f6c87a9c2f935bcc45f3c3c60275e5ce297ce36a89af46d3d6f5c1c00a/python-keystoneclient-3.16.0.tar.gz#sha256=0658240b39cced18784c8c4c0bed24d42f06c048103b397a55992a9e0da01c4a (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.16.0.tar.gz (306 kB)
     |████████████████████████████████| 306 kB 11.9 MB/s 
Collecting python-keystoneclient==3.15.1
  Downloading python_keystoneclient-3.15.1-py2.py3-none-any.whl (385 kB)
     |████████████████████████████████| 385 kB 12.7 MB/s 
Collecting https://files.pythonhosted.org/packages/be/2f/a58fc96d8d156606f283cd096299023bb8db27bb8495d618b21ff84683eb/python-keystoneclient-3.15.1.tar.gz#sha256=1ee5d35994e5b77d256c9f03085bcd1747417bff2c8b73c979bfe85d04e76355 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.15.1.tar.gz (306 kB)
     |████████████████████████████████| 306 kB 39.1 MB/s 
Collecting python-keystoneclient==3.15.0
  Downloading python_keystoneclient-3.15.0-py2.py3-none-any.whl (378 kB)
     |████████████████████████████████| 378 kB 13.2 MB/s 
Collecting https://files.pythonhosted.org/packages/53/a6/b9c2c3a0054cc2c92680e523a5d1050d711f93f91a05a37fc1ce81d894c0/python-keystoneclient-3.15.0.tar.gz#sha256=f82c43c1cde7453ee274090c2e816c27c8a4c1de88583a8a5a509719917396fa (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.15.0.tar.gz (306 kB)
     |████████████████████████████████| 306 kB 13.0 MB/s 
Collecting python-keystoneclient==3.14.0
  Downloading python_keystoneclient-3.14.0-py2.py3-none-any.whl (372 kB)
     |████████████████████████████████| 372 kB 12.4 MB/s 
Collecting https://files.pythonhosted.org/packages/33/d4/49070b0051e9ca47db25f6a8bb40eb13becad86f2de0315eabdcefd80e9e/python-keystoneclient-3.14.0.tar.gz#sha256=88b881d4316dd0a21724c7c123e6e3ed055b59a7aae4c703c7e2e47f64f9d2f2 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.14.0.tar.gz (304 kB)
     |████████████████████████████████| 304 kB 42.4 MB/s 
Collecting python-keystoneclient==3.13.1
  Downloading python_keystoneclient-3.13.1-py2.py3-none-any.whl (381 kB)
     |████████████████████████████████| 381 kB 15.2 MB/s 
Collecting https://files.pythonhosted.org/packages/5c/97/f61c30a56db72b5e5559e5f83034530693a62eb257e58a11092dc24e942d/python-keystoneclient-3.13.1.tar.gz#sha256=dcc24ecf0f0b5abb1955eea31cd10b400be39d01d0a88fc212a5cf64bbdb5da1 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.13.1.tar.gz (303 kB)
     |████████████████████████████████| 303 kB 12.8 MB/s 
Collecting python-keystoneclient==3.13.0
  Downloading python_keystoneclient-3.13.0-py2.py3-none-any.whl (374 kB)
     |████████████████████████████████| 374 kB 12.7 MB/s 
Collecting https://files.pythonhosted.org/packages/a4/a8/b8a83375ee028a0bd28acfcbb9bba20ab27ae2e7e513553e0bc1b901a4e4/python-keystoneclient-3.13.0.tar.gz#sha256=f897eaa6b251a12e5d23130e8435fb5d2ead6f7ea1d1d20faf2ccc1c76c51c90 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.13.0.tar.gz (303 kB)
     |████████████████████████████████| 303 kB 11.6 MB/s 
Collecting python-keystoneclient==3.12.0
  Downloading python_keystoneclient-3.12.0-py2.py3-none-any.whl (374 kB)
     |████████████████████████████████| 374 kB 38.9 MB/s 
Collecting https://files.pythonhosted.org/packages/33/89/d5103fa5d5d0545584a3d85c4e966f2ea4e72956e0f54a327a7e190a368f/python-keystoneclient-3.12.0.tar.gz#sha256=6879e32155d0efd48420cf320cdd826372240ab27f99adb813f0eef9d0db4147 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.12.0.tar.gz (305 kB)
     |████████████████████████████████| 305 kB 36.4 MB/s 
Collecting python-keystoneclient==3.11.0
  Downloading python_keystoneclient-3.11.0-py2.py3-none-any.whl (372 kB)
     |████████████████████████████████| 372 kB 13.1 MB/s 
Collecting https://files.pythonhosted.org/packages/c7/f6/4e156104f8680ac370df6057470ae1b266ffc398029eedd8ee9c6707ab4e/python-keystoneclient-3.11.0.tar.gz#sha256=dfe8626aacf087ee6fee31a37b98aa8f998902461c34651fb32349ef43b0a5b5 (from python-novaclient==2.27.0->pyrax==1.9.8)
  Downloading python-keystoneclient-3.11.0.tar.gz (298 kB)
     |████████████████████████████████| 298 kB 12.4 MB/s 
^C
ERROR: Operation cancelled by user
@pradyunsg pradyunsg added type: bug A confirmed bug or unintended behavior C: new resolver labels Apr 24, 2020
@pradyunsg
Copy link
Member Author

Source: #7406 (comment)

@pfmoore @uranusjr I think this is a really good test case for us to manually check how "done" we are.

@ei8fdb This is a pretty good test case, for using in user testing and research, during beta phase of the new resolver. This is a situation where the resolver would do a lot of things, and it needs to communicate to the user what it is doing. There's a lot of download, choices being made about dependencies, backtracking choices made and so on going on here.

@uranusjr
Copy link
Member

I think I know why this is happening (we’re returning multiple candidates of the same version from find_matches()). Do you plan to include this in 20.1 proper if we can produce a fix in time? I’ll prioritise work on this if you do, but don’t want to rush things otherwise.

@pfmoore
Copy link
Member

pfmoore commented Apr 25, 2020

Interesting case, and I agree with @uranusjr's diagnosis. A fix for this will conflict with the upgrade strategy changes. That's not a big deal, but I'd mildly prefer to leave this until after 20.1 final so that we can co-ordinate the two changes. Unless I'm misreading the context, I don't think this is newly identified by the beta, so there's no immediate reason to suggest it should be pushed into 20.1.

@pradyunsg
Copy link
Member Author

I think I know why this is happening (we’re returning multiple candidates of the same version from find_matches()).

Yea, I thought so too.

Do you plan to include this in 20.1 proper if we can produce a fix in time? I’ll prioritise work on this if you do, but don’t want to rush things otherwise.

No hurries from my end. :)

@pradyunsg
Copy link
Member Author

Oh, and since I haven't mentioned it here -- we're hitting pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 100 during this resolution, before finding a solution.

@uranusjr
Copy link
Member

Had we fixed this yet?

@pradyunsg
Copy link
Member Author

We did! I tried this earlier today. :)

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type: bug A confirmed bug or unintended behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@uranusjr @pfmoore @pradyunsg