Reduce memory usage for scripts with many Session objects

[jonemo]

Addresses boto/boto3#3614
Alternative to #2889

This retains the cache on botocore.endpoint_provider.EndpointProvider.resolve_endpoint while mitigating the effect on memory usage reported in boto/boto3#3614.

Problem description

The current code results in excessive memory usage when a large number of botocore Session objects is created in a script. Previous to botocore version 1.29.0, Python's garbage collector would have cleaned these objects up in most situations. The introduction of the EndpointProvider and the use of lru_cache within now results in up to 100 Session objects staying memory while they are referenced in the cache.

Solution in this PR:

This PR proposes a solution where Python's weakref module is used to temporarily replace the full reference with a weak reference during the cacheing process. This way, the cache no longer interferes with the garbage collector but otherwise performs the same function. This comes at the expense of a small computational overhead for creating (always) and resolving (for cache misses) the weak reference.

Other solutions considered:

• Remove the cache entirely: This is the best solution for the use case with many sessions. However, creating many sessions is only required in rare circumstances. Session object reuse is the recommended best practice and more common use case. For long-lived sessions where only a few services and operations are called, the cache has positive impact on runtime that we want to retain.
• Use botocore's instance_cache decorator (Replace lru_cache with instance_cache #2889): For long-lived sessions where each operation call results in a cache miss, this can lead to indefinitely growing memory usage. This is because instance_cache has no maxsize parameter.
• Use lru_cache in the constructor (Memory leak after updating to 1.25.0 boto3#3614 (comment)): This solution addresses the problem and performs well on both memory and runtime metrics on all Python versions we tested. However, it relies on an undocumented way of using the lru_cache decorate (namely: not as a decorator) and could therefore result in unexpected behavior changes in future Python versions.
• Keep lru_cache in place but reduce maxsize to 10: This avoids the excessive memory usage but still uses on the order of MBs more than the solution in this PR with a maxsize of 100.

Benchmarking results:

Each line represents a single run.

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (c95feba) 93.41% compared to head (5c44af8) 93.41%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #2934   +/-   ##
========================================
  Coverage    93.41%   93.41%           
========================================
  Files           63       63           
  Lines        13561    13571   +10     
========================================
+ Hits         12668    12678   +10     
  Misses         893      893           

Impacted Files	Coverage Δ
botocore/endpoint_provider.py	99.02% <100.00%> (-0.01%)	⬇️
botocore/utils.py	79.35% <100.00%> (+0.14%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[nateprewitt]

⛵ Thanks @jonemo. Awesome write up!

[nateprewitt]

Can we get a changelog entry too before we release?

[nateprewitt]