Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initialization hook to customize botocore sessions #2682

Merged
merged 1 commit into from
Jun 17, 2022

Conversation

jamesls
Copy link
Member

@jamesls jamesls commented May 20, 2022

This adds an initialization hook to customize any botocore sessions.

Botocore sessions provide context for creating one or more clients.
It allows you to provide configuration, components, hooks, etc. that
can be shared across all clients created within that session.

There is no similar concept for sessions. Specifically, there is no way
to provide a common set of configuration, components, hooks, etc. that
can shared across all sessions created within botocore.

This PR adds this initialization hook. You can now register an initialization
callback that's invoked whenever a session is created:

import botocore

def my_initializer(session: botocore.session.Session) -> None:
    session.register_component('data_loader', MyNewCrazyLoader())

botocore.register_initializer(my_initializer)

Now every session will automatically use this new loader:

some_session = botocore.session.get_session()
assert isinstance(
    some_session.get_component('data_loader'), MyNewCrazyLoader
)

Rationale

Botocore's extensibility points allows customers to customize the behavior of
botocore. However, in order to leverage these extensibility points, customers
must write explicit code to register these hooks that requires a reference to a
specific session (e.g session.register, session.register_component,
session.set_default_client_config). There is no way to provide "plugin" like
behavior where customers can declaratively specify functionality without having
to modify all their existing code.

Here's one concrete use case. Botocore has a data loader component that's
responsible for loading the service model based on various parameters
(service_name, type_name, api_version). The current implementation uses
JSON, which is not efficient in terms of disk space nor load time. There have
been initiatives to improve this (#2628), but in order to make this default
behavior, the existing JSONFileLoader class had to be modified.
There are also crazier use case specific ideas to further optimize both the
load time and disk size of our service models. However, in order to use
a new loader, users must add this line of code to every session they create:

s = botocore.session.get_session()
new_fancy_loader = MyNewCrazyLoader()
s.register_component(new_fancy_loader)

This is fine if you are able to obtain a reference to every single
session you create, but this isn't always the case, especially if
you're not directly using a session (e.g. some high level library
that wraps sessions/clients for you).

I maintain a framework, Chalice, where I would like to allow users
to opt-in to more optimized data loaders through a framework-specific
config file, without having to change any of their existing code.

@jamesls jamesls requested a review from nateprewitt May 20, 2022 20:49
@codecov-commenter
Copy link

codecov-commenter commented May 20, 2022

Codecov Report

Merging #2682 (e017467) into develop (92247f6) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##           develop    #2682   +/-   ##
========================================
  Coverage    95.29%   95.30%           
========================================
  Files           60       60           
  Lines        12252    12261    +9     
========================================
+ Hits         11676    11685    +9     
  Misses         576      576           
Impacted Files Coverage Δ
botocore/__init__.py 100.00% <100.00%> (ø)
botocore/session.py 96.79% <100.00%> (+<0.01%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92247f6...e017467. Read the comment docs.

This adds an initialization hook to customize any botocore sessions.

Botocore sessions provide context for creating one or more clients.
It allows you to provide configuration, components, hooks, etc. that
can be shared across all clients created within that session.

There is no similar concept for sessions.  Specifically, there is no way
to provide a common set of configuration, components, hooks, etc. that
can shared across all sessions created within botocore.

This change adds this initialization hook.  You can now register an
initialization callback that's invoked whenever a session is created:

```python
import botocore

def my_initializer(session: botocore.session.Session) -> None:
    session.register_component('data_loader', MyNewCrazyLoader())

botocore.register_initializer(my_initializer)
```

Now every session will automatically use this new loader:

```python

some_session = botocore.session.get_session()
assert isinstance(
    some_session.get_component('data_loader'), MyNewCrazyLoader
)
```
Copy link
Contributor

@dlm6693 dlm6693 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Only thought is maybe it would be useful to have a method that invokes specific initializers rather than all of them carte blanche. Would require a refactor since invoke_initializers is called at initialization, but just an idea! 🚢

Copy link
Contributor

@nateprewitt nateprewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@nateprewitt nateprewitt merged commit f1a73ff into boto:develop Jun 17, 2022
aws-sdk-python-automation added a commit that referenced this pull request Jun 17, 2022
* release-1.27.12:
  Bumping version to 1.27.12
  Update to latest endpoints
  Update to latest models
  Add initialization hook for session creation (#2682)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants