3.7. Pydantic Settings

One of pydantic's most useful applications is settings management.

If you create a model that inherits from BaseSettings, the model initialiser will attempt to determine the values of any fields not passed as keyword arguments by reading from the environment. (Default values will still be used if the matching environment variable is not set.)

This makes it easy to:

  • Create a clearly-defined, type-hinted application configuration class

  • Automatically read modifications to the configuration from environment variables

  • Manually override specific settings in the initialiser where desired (e.g. in unit tests)

3.7.1. Example

from typing import Set

from pydantic import (
    BaseModel,
    BaseSettings,
    PyObject,
    RedisDsn,
    PostgresDsn,
    AmqpDsn,
    Field,
)


class SubModel(BaseModel):
    foo = 'bar'
    apple = 1


class Settings(BaseSettings):
    auth_key: str
    api_key: str = Field(..., env='my_api_key')

    redis_dsn: RedisDsn = 'redis://user:pass@localhost:6379/1'
    pg_dsn: PostgresDsn = 'postgres://user:pass@localhost:5432/foobar'
    amqp_dsn: AmqpDsn = 'amqp://user:pass@localhost:5672/'

    special_function: PyObject = 'math.cos'

    # to override domains:
    # export my_prefix_domains='["foo.com", "bar.com"]'
    domains: Set[str] = set()

    # to override more_settings:
    # export my_prefix_more_settings='{"foo": "x", "apple": 1}'
    more_settings: SubModel = SubModel()

    class Config:
        env_prefix = 'my_prefix_'  # defaults to no prefix, i.e. ""
        fields = {
            'auth_key': {
                'env': 'my_auth_key',
            },
            'redis_dsn': {
                'env': ['service_redis_dsn', 'redis_url']
            }
        }


print(Settings().dict())
"""
{
    'auth_key': 'xxx',
    'api_key': 'xxx',
    'redis_dsn': RedisDsn('redis://user:pass@localhost:6379/1',
scheme='redis', user='user', password='pass', host='localhost',
host_type='int_domain', port='6379', path='/1'),
    'pg_dsn': PostgresDsn('postgres://user:pass@localhost:5432/foobar',
scheme='postgres', user='user', password='pass', host='localhost',
host_type='int_domain', port='5432', path='/foobar'),
    'amqp_dsn': AmqpDsn('amqp://user:pass@localhost:5672/', scheme='amqp',
user='user', password='pass', host='localhost', host_type='int_domain',
port='5672', path='/'),
    'special_function': <built-in function cos>,
    'domains': set(),
    'more_settings': {'foo': 'bar', 'apple': 1},
}
"""

3.7.2. Env

Parsing environment variable values🔗 For most simple field types (such as int, float, str, etc.), the environment variable value is parsed the same way it would be if passed directly to the initialiser (as a string).

Complex types like list, set, dict, and sub-models are populated from the environment by treating the environment variable's value as a JSON-encoded string.

Another way to populate nested complex variables is to configure your model with the env_nested_delimiter config setting, then use an env variable with a name pointing to the nested module fields. What it does is simply explodes yor variable into nested models or dicts. So if you define a variable FOO__BAR__BAZ=123 it will convert it into FOO={'BAR': {'BAZ': 123}} If you have multiple variables with the same structure they will be merged.

With the following environment variables:

$ export V0=0
$ export SUB_MODEL='{"v1": "json-1", "v2": "json-2"}'
$ export SUB_MODEL__V2=nested-2
$ export SUB_MODEL__V3=3
$ export SUB_MODEL__DEEP__V4=v4

from pydantic import BaseModel, BaseSettings


class DeepSubModel(BaseModel):
    v4: str


class SubModel(BaseModel):
    v1: str
    v2: bytes
    v3: int
    deep: DeepSubModel


class Settings(BaseSettings):
    v0: str
    sub_model: SubModel

    class Config:
        env_nested_delimiter = '__'


print(Settings().dict())
"""
{
    'v0': '0',
    'sub_model': {
        'v1': 'json-1',
        'v2': b'nested-2',
        'v3': 3,
        'deep': {'v4': 'v4'},
    },
}
"""

3.7.3. Dotenv (.env) support

  • dotenv file parsing requires python-dotenv to be installed.

  • pip install python-dotenv

  • pip install pydantic[dotenv]

# ignore comment
ENVIRONMENT="production"
REDIS_ADDRESS=localhost:6379
MEANING_OF_LIFE=42
MY_VAR='Hello world'

class Settings(BaseSettings):
    ...

    class Config:
        env_file = '.env'
        env_file_encoding = 'utf-8'

settings = Settings(_env_file='prod.env', _env_file_encoding='utf-8')

3.7.4. Secret Support

Placing secret values in files is a common pattern to provide sensitive configuration to an application.

A secret file follows the same principal as a dotenv file except it only contains a single value and the file name is used as the key. A secret file will look like the following:

$ cat /var/run/database_password
myVoiceIsMyPasswordVerifyMe

class Settings(BaseSettings):
    database_password: str

    class Config:
        secrets_dir = '/var/run'

settings = Settings(_secrets_dir='/var/run')

Even when using a secrets directory, pydantic will still read environment variables from a dotenv file or the environment, a dotenv file and environment variables will always take priority over values loaded from the secrets directory.

Passing a file path via the _secrets_dir keyword argument on instantiation (method 2) will override the value (if any) set on the Config class.

Docker Secrets can be used to provide sensitive configuration to an application running in a Docker container. To use these secrets in a pydantic application the process is simple. More information regarding creating, managing and using secrets in Docker see the official Docker documentation.

First, define your Settings

class Settings(BaseSettings):

my_secret_data: str

class Config:

secrets_dir = '/run/secrets'

By default Docker uses /run/secrets as the target mount point. If you want to use a different location, change Config.secrets_dir accordingly.

Then, create your secret via the Docker CLI

printf "This is a secret" | docker secret create my_secret_data -

Last, run your application inside a Docker container and supply your newly created secret

$ docker service create \
    --name pydantic-with-secrets \
    --secret my_secret_data
    pydantic-app:latest