Cache parse_links() by --find-links html page content

[cosmicexplorer]

Cherry-pick of pypa#7729! This change is intended to fix pex-tool/pex#887.

[benjyw]

I see, so you're not trying to cache between runs, just within a single run.

Would there be even more benefit in the future to caching between runs?

[cosmicexplorer]

Would there be even more benefit in the future to caching between runs?

EDITED THIS COMMENT A LOT!!

See maybe-related convo over at the pip PR in pypa#7729 (comment).

pypa#5670 is working against us a little here, because we don't want to have caching that avoids reading in updates from publishing. I'm actually not sure if I can see an obvious form of caching between runs that improves on this PR without causing issues like pypa#5670.

[cosmicexplorer]

Hm, I think the above is correct, but also think a really relevant optimization would be to cache the html page fetching as well for --find-links repos (currently the page is still re-fetched on each transitive requirement -- this caching just avoids re-parsing the page constantly). That would probably take the shape of another dictionary somewhere nearby to avoid fetching an HTMLPage more than once, keyed by url. I'll use the same benchmark script as in pypa#7729 to see whether that change affects performance too.

[cosmicexplorer]

...even locally (without going across the network to retrieve the --find-links page), it appears that caching the html page itself by url has an impact on performance. Via the benchmark script at pypa#7729, I'm seeing pip complete at pretty reliably 11.8 seconds with the changes in 1d8f381, and 11.9-12.+ seconds with just the parse_links() caching. I'm going to propose adding this additional level of caching to the upstream pip PR as well.

[cosmicexplorer]

Proposed the change in the upstream pip PR: pypa#7729 (comment).

[cosmicexplorer]

(we will be cherry-picking the resulting changes from pypa#7729, if they differ from this, after it is approved and merged!)