feat: Enable access to the current state in settings sources

[VictorColomb]

Why this PR?

This PR aims at enabling custom settings sources to access the output of the previous sources, through the .previous_state property.

The ultimate use-case is the same as #224, however this PR does not introduce a breaking change.

Fixes #223.

Use-case

My real-life use-case: retrieving settings from AWS Secrets Manager but setting the secret name in environment variables.

Discussion

There are a number of things to be discussed about this PR, and I would very much appreciate the input / guidance of the pydantic-settings maintainers.

1. I have reversed the order in which the settings sources are discovered (here) in order for the output of the sources with the most priority to be available to the ones of lesser importance. This seems like the most intuitive behavior to me, but this is rather subjective. Any preference?
2. Currently, settings sources are being handed the raw output from the previous ones. In other words, nothing has yet been validated by Pydantic. Would it be desired behavior to instead pass a validated instance of self.settings_cls? I have not gone this way because imho it adds to much complexity for not that much benefit.
3. I have defined the __previous_state setter method as "private", in that Python will perform name mangling on the method. This way, we are almost certain to introduce no breaking changes at all (someone could have defined their own set_previous_state method). However, that means calling _PydanticBaseSettingsSource__set_previous_state in the sources discovery code, which is neither elegant nor pythonic. My question therefore is should I remove the double-underscore for cleaner code at the marginal risk of introducing a breaking change for some?
4. Should I add a bit of documentation, for example as a subsection of the Adding sources section?

[hramezani]

Thanks @VictorColomb for this PR and sorry for the late response.

I have reversed the order in which the settings sources are discovered (here) in order for the output of the sources with the most priority to be available to the ones of lesser importance. This seems like the most intuitive behavior to me, but this is rather subjective. Any preference?

This is good

Currently, settings sources are being handed the raw output from the previous ones. In other words, nothing has yet been validated by Pydantic. Would it be desired behavior to instead pass a validated instance of self.settings_cls? I have not gone this way because imho it adds to much complexity for not that much benefit.

Your approach is fine. don't need to validate beforehand.

I have defined the __previous_state setter method as "private", in that Python will perform name mangling on the method. This way, we are almost certain to introduce no breaking changes at all (someone could have defined their own set_previous_state method). However, that means calling _PydanticBaseSettingsSource__set_previous_state in the sources discovery code, which is neither elegant nor pythonic. My question therefore is should I remove the double-underscore for cleaner code at the marginal risk of introducing a breaking change for some?

Yes, let's have it without double underscore. let's have the previous_states as a property and _previous_states and the setter as a public method.

Should I add a bit of documentation, for example as a subsection of the Adding sources section?

sure.

I have a request:

• The current implementation passes only the previous source state to the next one, I would suggest to pass all the previous states to the next one. so, the states should be something like

{'source_key': {source_state}}

Please let me know when you are ready for the next round of review.

[VictorColomb]

Hi @hramezani. Thanks for your feedback!

I have implemented your suggestions.

However, I do see a potential pitfall in recording all previous states: if two settings sources have the same name (e.g. two instances of the same class, but with different init parameters), only the latest one would be recorded in previous_states.

[hramezani]

Thanks @VictorColomb for updating the PR.

• Please add some tests related to previous_states

[VictorColomb]

@hramezani test added! I also updated my branch from main.

[hramezani]

I think previous_states and previous_state may make user confused. what about replacing previous_states with settings_sources_data and previous_state with current_state?

If you have any suggestions please let me know.

[VictorColomb]

I have current_settings_state in mind, but it's uselessly verbose.

I don't find settings_sources_data particularly explicit, but I cannot find a better name. Maybe previous_settings_sources_output, or previous_sources_output (a little shorter)?

In any case, I've implemented your suggestions. Let me know what you think!

[hramezani]

Thanks @VictorColomb