2.6.0 breaks validation with nested settings when using empty default

[palle-k]

The following code works with 2.5.0, but breaks with 2.6.0:

from pydantic_settings import BaseSettings, PydanticBaseSettingsSource


class DummySource(PydanticBaseSettingsSource):
    def __call__(self):
        return {"bar": 42, "baz": "hello"}

    def get_field_value(self, field, field_name):
        return self()[field_name], field_name, False
    

class Foo(BaseSettings):
    bar: int
    baz: str

    @classmethod
    def settings_customise_sources(cls, settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings):
        return init_settings, DummySource(settings_cls), env_settings, dotenv_settings, file_secret_settings


class Settings(BaseSettings):
    foo: Foo = {}


if __name__ == '__main__':
    print(Settings())

With 2.5.0, Pydantic-Settings would populate all missing fields from the source. With 2.6.0, the source is not used anymore, breaking the code above.

[hramezani]

Thanks @palle-k for reporting this.

Actually, it is not a bug. I think you are misusing the sub-models. as mentioned in the doc
Sub model has to inherit from pydantic.BaseModel, Otherwise pydantic-settings will initialize sub model, collects values for sub model fields separately, and you may get unexpected results.

So, in your example, pydantic-settings initializes the Foo and Settings models. that's why you don't get any errors.

You face the problem after this change

BTW, you can get your desired behavior by changing your code to:

from pydantic import BaseModel

from pydantic_settings import BaseSettings, PydanticBaseSettingsSource


class DummySource(PydanticBaseSettingsSource):
    def __call__(self):
        return {'foo': {"bar": 42, "baz": "hello"}}

    def get_field_value(self, field, field_name):
        return self()[field_name], field_name, False
    

class Foo(BaseModel):
    bar: int
    baz: str


class Settings(BaseSettings):
    foo: Foo = {}

    @classmethod
    def settings_customise_sources(cls, settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings):
        return init_settings, DummySource(settings_cls), env_settings, dotenv_settings, file_secret_settings

if __name__ == '__main__':
    print(Settings())

[palle-k]

Unfortunately, 2.6.0 breaks a bunch of our system and making the change outlined in your response is not trivial in our situation because we load settings from different places and use the structure I outlined in my original report to modularize settings and the loading of settings.

[hramezani]

I understand the situation that you are in.

You are using the sub-models incorrectly, so you need to fix your models at some point. otherwise, you will be faced with this kind of problem in future as well.

my suggestion is to pin your code base on pydantic-settings 2.5.2 and start fixing the settings model.

we really try not to break user codes, there was another issue related to this change and that one also was a misusage.

Do you see anything that I can help here?

[palle-k]

Furthermore, using BaseModels for nested settings prohibits us from accessing env variables that are not prefixed:

class DatabaseSettings(BaseModel):
   mongo_uri: str # I want to fill this using the MONGO_URI env var
   database: str

class MyAppSettings(BaseSettings):
   db: DatabaseSettings
   model_config = SettingsConfigDict(env_nested_delimiter="__")

[hramezani]

even by defining the DatabaseSettings like class DatabaseSettings(BaseSettings):(wrong usage), you only can collect mongo_uri if it doesn't follow with db_mongo_uri. I mean if you have env like db__database=db1 db__mongo_uri=uri1 mongo_uri=uri2, then uri1 will be collected.

[palle-k]

I understand the situation that you are in.

You are using the sub-models incorrectly, so you need to fix your models at some point. otherwise, you will be faced with this kind of problem in future as well.

my suggestion is to pin your code base on pydantic-settings 2.5.2 and start fixing the settings model.

we really try not to break user codes, there was another issue related to this change and that one also was a misusage.

Do you see anything that I can help here?

Generally, what would be the Pydantic approved way of modularizing settings with dedicated loaders for each module?

E.g.:

# module_a_settings.py
class ConfigurationForModuleA(BaseModel):
    ...
    # this should be loaded using a custom source (ModuleASettingsSource)

# module_b_settings.py
class ConfigurationForModuleB(BaseModel):
    ...
    # this should be loaded using another custom source (ModuleBSettingsSource)

# app_settings.py
class MyAppSettings(BaseSettings):
    module_a: ConfigurationForModuleA
    module_b: ConfigurationForModuleB

    model_config = {"env_nested_delimiter": "__"}

Now, I want my module configurations to be loaded from each respective source, but also allow overriding from environment variables (i.e. if I specify MODULE_A__SOME_VALUE, it should have precedence over the value loaded from the ModuleASettingsSource)

[hramezani]

There are a couple of ways that can fix your problem.

changing the default value of foo to Foo() instead of {}

class Settings(BaseSettings):
    foo: Foo = Foo()

Or adding an __init__ which is doing nothing to Foo:

class Foo(BaseSettings):
    bar: int
    baz: str

    def __init__(self, **data):
        super().__init__(**data)

    @classmethod
    def settings_customise_sources(cls, settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings):
        return init_settings, DummySource(settings_cls), env_settings, dotenv_settings, file_secret_settings

But still my suggestion is to change your model definitions

[hramezani]

Please see my comment on the other issue.

[hramezani]

I reverted the fix that caused the problem and released pydantic-settings 2.6.1.