about things
docs: expand pydantic notes into directory structure
Split single pydantic.md into pydantic/ with:
- README.md — core idea, when to use what
- settings.md — BaseSettings, nested config, splitting by concern
- validation.md — Annotated types as reusable type library (from prefect/types/)
- serialization.md — model_dump, JSON round-trips, TypeAdapter
Preps for plyr.fm config.py decomposition (phase 3).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
+459
-135
+1
-1
languages/python/README.md
+1
-1
languages/python/README.md
-134
languages/python/ecosystem/pydantic.md
-134
languages/python/ecosystem/pydantic.md
···
1
-
# pydantic
2
-
3
-
pydantic is a library, not the language. but it's become foundational enough that it's worth understanding.
4
-
5
-
## the core idea
6
-
7
-
python's type hints don't do anything at runtime by default. `def foo(x: int)` accepts strings, floats, whatever - the annotation is just documentation.
8
-
9
-
pydantic makes them real. define a model with type hints, and pydantic validates and coerces data to match:
10
-
11
-
```python
12
-
from pydantic import BaseModel
13
-
14
-
class User(BaseModel):
15
-
name: str
16
-
age: int
17
-
18
-
user = User(name="alice", age="25") # age coerced to int
19
-
user = User(name="alice", age="not a number") # raises ValidationError
20
-
```
21
-
22
-
this is why pydantic shows up everywhere in python - it bridges the gap between python's dynamic runtime and the desire for validated, typed data.
23
-
24
-
## settings from environment
25
-
26
-
the most common use: replacing `os.getenv()` calls with validated configuration.
27
-
28
-
```python
29
-
from pydantic import Field
30
-
from pydantic_settings import BaseSettings, SettingsConfigDict
31
-
32
-
class Settings(BaseSettings):
33
-
"""settings for atproto cli."""
34
-
35
-
model_config = SettingsConfigDict(
36
-
env_file=str(Path.cwd() / ".env"),
37
-
env_file_encoding="utf-8",
38
-
extra="ignore",
39
-
case_sensitive=False,
40
-
)
41
-
42
-
atproto_pds_url: str = Field(
43
-
default="https://bsky.social",
44
-
description="PDS URL",
45
-
)
46
-
atproto_handle: str = Field(default="", description="Your atproto handle")
47
-
atproto_password: str = Field(default="", description="Your atproto app password")
48
-
49
-
settings = Settings()
50
-
```
51
-
52
-
`model_config` controls where settings come from (environment, .env files) and how to handle unknowns. required fields without defaults fail at import time - not later when you try to use them.
53
-
54
-
from [pdsx/_internal/config.py](https://github.com/zzstoatzz/pdsx/blob/main/src/pdsx/_internal/config.py)
55
-
56
-
## annotated types for reusable validation
57
-
58
-
when multiple schemas share the same validation logic, bind it to the type itself instead of repeating `@field_validator` on each schema:
59
-
60
-
```python
61
-
from datetime import timedelta
62
-
from typing import Annotated
63
-
from pydantic import AfterValidator, BaseModel
64
-
65
-
def _validate_non_negative_timedelta(v: timedelta) -> timedelta:
66
-
if v < timedelta(seconds=0):
67
-
raise ValueError("timedelta must be non-negative")
68
-
return v
69
-
70
-
NonNegativeTimedelta = Annotated[
71
-
timedelta,
72
-
AfterValidator(_validate_non_negative_timedelta)
73
-
]
74
-
75
-
class RunDeployment(BaseModel):
76
-
schedule_after: NonNegativeTimedelta
77
-
```
78
-
79
-
benefits:
80
-
- write validation once
81
-
- field types become swappable interfaces
82
-
- types are self-documenting
83
-
84
-
from [coping with python's type system](https://blog.zzstoatzz.io/coping-with-python-type-system/)
85
-
86
-
## model_validator for side effects
87
-
88
-
run setup code when settings load:
89
-
90
-
```python
91
-
from typing import Self
92
-
from pydantic import model_validator
93
-
from pydantic_settings import BaseSettings
94
-
95
-
class Settings(BaseSettings):
96
-
debug: bool = False
97
-
98
-
@model_validator(mode="after")
99
-
def configure_logging(self) -> Self:
100
-
setup_logging(debug=self.debug)
101
-
return self
102
-
103
-
settings = Settings() # logging configured on import
104
-
```
105
-
106
-
the validator runs after all fields are set. use for side effects that depend on configuration values.
107
-
108
-
from [bot/config.py](https://github.com/zzstoatzz/bot)
109
-
110
-
## when to use what
111
-
112
-
pydantic models are heavier than they look - they do a lot of work on instantiation. for internal data you control, python's `dataclasses` are simpler:
113
-
114
-
```python
115
-
from dataclasses import dataclass
116
-
117
-
@dataclass
118
-
class BatchResult:
119
-
"""result of a batch operation."""
120
-
successful: list[str]
121
-
failed: list[tuple[str, Exception]]
122
-
123
-
@property
124
-
def total(self) -> int:
125
-
return len(self.successful) + len(self.failed)
126
-
```
127
-
128
-
no validation, no coercion, just a class with fields. use pydantic at boundaries (API input, config files, external data) where you need validation. use dataclasses for internal data structures.
129
-
130
-
from [pdsx/_internal/batch.py](https://github.com/zzstoatzz/pdsx/blob/main/src/pdsx/_internal/batch.py)
131
-
132
-
sources:
133
-
- [how to use pydantic-settings](https://blog.zzstoatzz.io/how-to-use-pydantic-settings/)
134
-
- [coping with python's type system](https://blog.zzstoatzz.io/coping-with-python-type-system/)
+56
languages/python/ecosystem/pydantic/README.md
+56
languages/python/ecosystem/pydantic/README.md
···
1
+
# pydantic
2
+
3
+
pydantic makes python's type hints real at runtime. define a model with annotations, and pydantic validates and coerces data to match.
4
+
5
+
```python
6
+
from pydantic import BaseModel
7
+
8
+
class User(BaseModel):
9
+
name: str
10
+
age: int
11
+
12
+
user = User(name="alice", age="25") # age coerced to int
13
+
user = User(name="alice", age="not a number") # raises ValidationError
14
+
```
15
+
16
+
this is why pydantic shows up everywhere — it bridges python's dynamic runtime and the desire for validated, typed data.
17
+
18
+
## when to use what
19
+
20
+
| tool | use when |
21
+
|------|----------|
22
+
| `BaseModel` | API boundaries, external data, anything you serialize |
23
+
| `BaseSettings` | configuration from env vars / .env files |
24
+
| `Annotated[T, ...]` types | reusable validation bound to the type, not the model |
25
+
| `dataclasses` | internal data you control — no validation overhead |
26
+
| `TypedDict` | typed dict unpacking (`**kwargs`) with per-key types |
27
+
28
+
pydantic models are heavier than they look — they do real work on instantiation. for internal data:
29
+
30
+
```python
31
+
from dataclasses import dataclass
32
+
33
+
@dataclass
34
+
class BatchResult:
35
+
successful: list[str]
36
+
failed: list[tuple[str, Exception]]
37
+
38
+
@property
39
+
def total(self) -> int:
40
+
return len(self.successful) + len(self.failed)
41
+
```
42
+
43
+
use pydantic at boundaries. use dataclasses for internal structures.
44
+
45
+
## contents
46
+
47
+
- [settings](./settings.md) — `BaseSettings`, env loading, splitting config by concern
48
+
- [validation](./validation.md) — `Annotated` types, validators, custom types
49
+
- [serialization](./serialization.md) — `model_dump`, computed fields, JSON round-trips
50
+
51
+
## sources
52
+
53
+
- [how to use pydantic-settings](https://blog.zzstoatzz.io/how-to-use-pydantic-settings/)
54
+
- [coping with python's type system](https://blog.zzstoatzz.io/coping-with-python-type-system/)
55
+
- [prefect/src/prefect/types/](https://github.com/prefecthq/prefect/tree/main/src/prefect/types) — `Annotated` type library pattern
56
+
- [pdsx/_internal/config.py](https://github.com/zzstoatzz/pdsx/blob/main/src/pdsx/_internal/config.py)
+109
languages/python/ecosystem/pydantic/serialization.md
+109
languages/python/ecosystem/pydantic/serialization.md
···
1
+
# serialization
2
+
3
+
## model_dump
4
+
5
+
convert a model to a dict. control what's included:
6
+
7
+
```python
8
+
class Track(BaseModel):
9
+
id: int
10
+
title: str
11
+
artist_did: str
12
+
internal_score: float
13
+
14
+
track.model_dump() # all fields
15
+
track.model_dump(exclude={"internal_score"}) # omit internal fields
16
+
track.model_dump(include={"id", "title"}) # only these
17
+
track.model_dump(mode="json") # json-compatible types (datetimes → strings)
18
+
```
19
+
20
+
`mode="json"` is important — without it, you get python objects (datetime, Decimal). with it, everything is JSON-serializable.
21
+
22
+
## model_dump_json / model_validate_json
23
+
24
+
skip the dict intermediary for performance:
25
+
26
+
```python
27
+
# serialize directly to JSON string
28
+
json_str = track.model_dump_json()
29
+
30
+
# deserialize directly from JSON string
31
+
track = Track.model_validate_json(json_str)
32
+
```
33
+
34
+
faster than `json.dumps(track.model_dump())` because pydantic uses rust-based serialization internally.
35
+
36
+
useful for Redis caching:
37
+
38
+
```python
39
+
await redis.set(cache_key, response.model_dump_json(), ex=300)
40
+
41
+
cached = await redis.get(cache_key)
42
+
if cached:
43
+
return AlbumResponse.model_validate_json(cached)
44
+
```
45
+
46
+
## computed fields
47
+
48
+
fields derived from other fields, included in serialization but not accepted as input:
49
+
50
+
```python
51
+
from pydantic import BaseModel, computed_field
52
+
53
+
class Album(BaseModel):
54
+
tracks: list[Track]
55
+
56
+
@computed_field
57
+
@property
58
+
def track_count(self) -> int:
59
+
return len(self.tracks)
60
+
61
+
@computed_field
62
+
@property
63
+
def total_duration(self) -> float:
64
+
return sum(t.duration for t in self.tracks)
65
+
```
66
+
67
+
shows up in `model_dump()` and JSON output. not stored, always recomputed.
68
+
69
+
## TypeAdapter for ad-hoc validation
70
+
71
+
validate data without defining a full model:
72
+
73
+
```python
74
+
from pydantic import TypeAdapter
75
+
76
+
int_adapter = TypeAdapter(int)
77
+
int_adapter.validate_python("42") # → 42
78
+
int_adapter.validate_python("abc") # → ValidationError
79
+
80
+
# useful for parsing env var fragments
81
+
StatusCode = Annotated[int, Field(ge=100, le=599)]
82
+
code_adapter = TypeAdapter(StatusCode)
83
+
code_adapter.validate_python(429) # → 429
84
+
code_adapter.validate_python(999) # → ValidationError
85
+
```
86
+
87
+
also handles complex types: `TypeAdapter(list[int])`, `TypeAdapter(dict[str, float])`.
88
+
89
+
## response models in FastAPI
90
+
91
+
FastAPI uses pydantic models for automatic request validation and response serialization:
92
+
93
+
```python
94
+
class PlaylistResponse(BaseModel):
95
+
id: str
96
+
name: str
97
+
track_count: int
98
+
created_at: str
99
+
100
+
@router.get("/playlists/{id}", response_model=PlaylistResponse)
101
+
async def get_playlist(id: str) -> PlaylistResponse:
102
+
...
103
+
```
104
+
105
+
the `response_model` strips any extra fields from the return value — useful when your internal object has more data than the API should expose.
106
+
107
+
sources:
108
+
- [pydantic docs: serialization](https://docs.pydantic.dev/latest/concepts/serialization/)
109
+
- [pydantic docs: type adapter](https://docs.pydantic.dev/latest/concepts/type_adapter/)
+137
languages/python/ecosystem/pydantic/settings.md
+137
languages/python/ecosystem/pydantic/settings.md
···
1
+
# settings
2
+
3
+
`pydantic-settings` replaces `os.getenv()` with validated, typed configuration.
4
+
5
+
## basic usage
6
+
7
+
```python
8
+
from pydantic import Field
9
+
from pydantic_settings import BaseSettings, SettingsConfigDict
10
+
11
+
class Settings(BaseSettings):
12
+
model_config = SettingsConfigDict(
13
+
env_file=".env",
14
+
env_file_encoding="utf-8",
15
+
extra="ignore",
16
+
case_sensitive=False,
17
+
)
18
+
19
+
database_url: str
20
+
debug: bool = False
21
+
port: int = Field(default=8000, ge=1, le=65535)
22
+
23
+
settings = Settings()
24
+
```
25
+
26
+
required fields without defaults fail at import time — not later when you use them. `extra="ignore"` silently drops unknown env vars instead of erroring.
27
+
28
+
from [pdsx/_internal/config.py](https://github.com/zzstoatzz/pdsx/blob/main/src/pdsx/_internal/config.py)
29
+
30
+
## nested settings
31
+
32
+
for large configs, group related settings into sub-models. use `env_prefix` to namespace env vars:
33
+
34
+
```python
35
+
class DatabaseSettings(BaseSettings):
36
+
model_config = SettingsConfigDict(env_prefix="DB_")
37
+
38
+
url: str
39
+
pool_size: int = 10
40
+
max_overflow: int = 5
41
+
42
+
class StorageSettings(BaseSettings):
43
+
model_config = SettingsConfigDict(env_prefix="R2_")
44
+
45
+
bucket: str
46
+
endpoint: str
47
+
access_key: str
48
+
secret_key: str
49
+
50
+
class Settings(BaseSettings):
51
+
db: DatabaseSettings = DatabaseSettings()
52
+
storage: StorageSettings = StorageSettings()
53
+
debug: bool = False
54
+
```
55
+
56
+
`DB_URL`, `DB_POOL_SIZE`, `R2_BUCKET` etc. map to the right nested field. each group is independently testable and documentable.
57
+
58
+
## splitting config by concern
59
+
60
+
a 1000-line config file is a code smell. split by domain:
61
+
62
+
```
63
+
config/
64
+
├── __init__.py # re-exports Settings
65
+
├── auth.py # OAuth, session, JWT settings
66
+
├── storage.py # R2, filesystem settings
67
+
├── services.py # external service URLs, keys
68
+
└── features.py # feature flags
69
+
```
70
+
71
+
the root `Settings` composes them:
72
+
73
+
```python
74
+
# config/__init__.py
75
+
class Settings(BaseSettings):
76
+
auth: AuthSettings = AuthSettings()
77
+
storage: StorageSettings = StorageSettings()
78
+
services: ServiceSettings = ServiceSettings()
79
+
features: FeatureFlags = FeatureFlags()
80
+
```
81
+
82
+
this maps naturally to infrastructure — each sub-model corresponds to a k8s secret or configmap.
83
+
84
+
## model_validator for initialization side effects
85
+
86
+
```python
87
+
from typing import Self
88
+
from pydantic import model_validator
89
+
90
+
class Settings(BaseSettings):
91
+
debug: bool = False
92
+
log_level: str = "INFO"
93
+
94
+
@model_validator(mode="after")
95
+
def configure_logging(self) -> Self:
96
+
import logging
97
+
logging.basicConfig(level=getattr(logging, self.log_level))
98
+
return self
99
+
```
100
+
101
+
runs once when the singleton is created. use for side effects that depend on resolved config values (logging setup, SDK initialization, connection pool creation).
102
+
103
+
## singleton pattern
104
+
105
+
instantiate once, import everywhere:
106
+
107
+
```python
108
+
from functools import lru_cache
109
+
110
+
@lru_cache
111
+
def get_settings() -> Settings:
112
+
return Settings()
113
+
114
+
settings = get_settings()
115
+
```
116
+
117
+
`lru_cache` ensures the settings object (and its `model_validator` side effects) only run once. useful when settings trigger expensive initialization.
118
+
119
+
## secrets
120
+
121
+
for sensitive values, use `SecretStr`:
122
+
123
+
```python
124
+
from pydantic import SecretStr
125
+
126
+
class Settings(BaseSettings):
127
+
api_key: SecretStr
128
+
129
+
def get_api_key(self) -> str:
130
+
return self.api_key.get_secret_value()
131
+
```
132
+
133
+
`SecretStr` redacts in repr/logging. `.get_secret_value()` for the actual string.
134
+
135
+
sources:
136
+
- [how to use pydantic-settings](https://blog.zzstoatzz.io/how-to-use-pydantic-settings/)
137
+
- [pdsx/_internal/config.py](https://github.com/zzstoatzz/pdsx/blob/main/src/pdsx/_internal/config.py)
+156
languages/python/ecosystem/pydantic/validation.md
+156
languages/python/ecosystem/pydantic/validation.md
···
1
+
# validation
2
+
3
+
## annotated types as a type library
4
+
5
+
the most important pattern in pydantic. instead of repeating `@field_validator` on every model, bind validation to the type itself:
6
+
7
+
```python
8
+
from typing import Annotated
9
+
from pydantic import AfterValidator, BeforeValidator, Field
10
+
11
+
NonNegativeInteger = Annotated[int, Field(ge=0)]
12
+
PositiveInteger = Annotated[int, Field(gt=0)]
13
+
StatusCode = Annotated[int, Field(ge=100, le=599)]
14
+
LogLevel = Annotated[
15
+
Literal["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"],
16
+
BeforeValidator(lambda x: x.upper()),
17
+
]
18
+
```
19
+
20
+
now any model field typed as `StatusCode` gets validation for free. the validator travels with the type.
21
+
22
+
this scales into a full type library:
23
+
24
+
```python
25
+
# types/__init__.py
26
+
from .names import Name, VariableName, URILike
27
+
from ._datetime import DateTime, PositiveInterval
28
+
29
+
NonNegativeFloat = Annotated[float, Field(ge=0.0)]
30
+
LaxUrl = Annotated[str, BeforeValidator(lambda x: str(x).strip())]
31
+
```
32
+
33
+
from [prefect/types/\_\_init\_\_.py](https://github.com/prefecthq/prefect/blob/main/src/prefect/types/__init__.py)
34
+
35
+
## BeforeValidator vs AfterValidator
36
+
37
+
- `BeforeValidator` — runs before pydantic's own parsing. use for coercion (strings to timedeltas, env var parsing):
38
+
39
+
```python
40
+
def _convert_seconds_to_timedelta(value: Any) -> Any:
41
+
if isinstance(value, timedelta):
42
+
return value
43
+
if isinstance(value, (int, float)):
44
+
return timedelta(seconds=value)
45
+
return value
46
+
47
+
SecondsTimeDelta = Annotated[timedelta, BeforeValidator(_convert_seconds_to_timedelta)]
48
+
```
49
+
50
+
- `AfterValidator` — runs after pydantic parses. use for constraints on already-typed values:
51
+
52
+
```python
53
+
def _validate_non_negative_timedelta(v: timedelta) -> timedelta:
54
+
if v.total_seconds() < 0:
55
+
raise ValueError("must be non-negative")
56
+
return v
57
+
58
+
NonNegativeTimedelta = Annotated[timedelta, AfterValidator(_validate_non_negative_timedelta)]
59
+
```
60
+
61
+
## composing validators
62
+
63
+
`Annotated` stacks. you can layer multiple validators and field constraints:
64
+
65
+
```python
66
+
ValidAssetKey = Annotated[
67
+
str,
68
+
AfterValidator(validate_valid_asset_key),
69
+
Field(
70
+
max_length=512,
71
+
description="URI-like string with restricted characters",
72
+
),
73
+
]
74
+
```
75
+
76
+
## domain-specific types in their own module
77
+
78
+
when a domain has enough types, extract them. prefect's `names.py` defines `Name`, `VariableName`, `BlockDocumentName`, `URILike` — all `Annotated` types with validators using `functools.partial` for parameterization:
79
+
80
+
```python
81
+
from functools import partial
82
+
83
+
def raise_on_name_alphanumeric_dashes_only(
84
+
value: str | None, field_name: str = "value"
85
+
) -> str | None:
86
+
if value is not None and not re.match(r"^[a-z0-9-]*$", value):
87
+
raise ValueError(f"{field_name} must only contain lowercase letters, numbers, and dashes.")
88
+
return value
89
+
90
+
BlockDocumentName = Annotated[
91
+
Name,
92
+
AfterValidator(partial(raise_on_name_alphanumeric_dashes_only, field_name="Block document name")),
93
+
]
94
+
```
95
+
96
+
`partial` lets one validator function serve multiple domain types with customized error messages.
97
+
98
+
from [prefect/types/names.py](https://github.com/prefecthq/prefect/blob/main/src/prefect/types/names.py)
99
+
100
+
## model_validator for side effects
101
+
102
+
run setup code when a model is instantiated:
103
+
104
+
```python
105
+
from typing import Self
106
+
from pydantic import model_validator
107
+
from pydantic_settings import BaseSettings
108
+
109
+
class Settings(BaseSettings):
110
+
debug: bool = False
111
+
112
+
@model_validator(mode="after")
113
+
def configure_logging(self) -> Self:
114
+
setup_logging(debug=self.debug)
115
+
return self
116
+
117
+
settings = Settings() # logging configured on import
118
+
```
119
+
120
+
the validator runs after all fields are set. use for side effects that depend on configuration values.
121
+
122
+
from [bot/config.py](https://github.com/zzstoatzz/bot)
123
+
124
+
## env var parsing with BeforeValidator
125
+
126
+
for settings that come from environment variables as strings but need richer types:
127
+
128
+
```python
129
+
def validate_set_T_from_delim_string(
130
+
value: str | T | set[T] | None, type_: Any, delim: str | None = None
131
+
) -> set[T]:
132
+
"""parse comma-delimited env vars into typed sets.
133
+
134
+
PREFECT_CLIENT_RETRY_EXTRA_CODES=429,502,503 → {429, 502, 503}
135
+
"""
136
+
if not value:
137
+
return set()
138
+
T_adapter = TypeAdapter(type_)
139
+
delim = delim or ","
140
+
if isinstance(value, str):
141
+
return {T_adapter.validate_strings(s.strip()) for s in value.split(delim)}
142
+
...
143
+
144
+
ClientRetryExtraCodes = Annotated[
145
+
str | StatusCode | set[StatusCode] | None,
146
+
BeforeValidator(partial(validate_set_T_from_delim_string, type_=StatusCode)),
147
+
]
148
+
```
149
+
150
+
`TypeAdapter` is useful here — it validates individual values without needing a full model.
151
+
152
+
from [prefect/types/\_\_init\_\_.py](https://github.com/prefecthq/prefect/blob/main/src/prefect/types/__init__.py)
153
+
154
+
sources:
155
+
- [coping with python's type system](https://blog.zzstoatzz.io/coping-with-python-type-system/)
156
+
- [prefect/types/](https://github.com/prefecthq/prefect/tree/main/src/prefect/types)