Smart configuration loader
Add documentation for config sourcing and dynamic configs
- doc/discovery/config-sourcing.md: Complete analysis of c12's configuration pipeline, showing how defu merges configs from multiple sources with mermaid diagrams
- doc/discovery/dynamic-configs.md: Exploration of dynamic configuration sources including drop-in directories, provider pattern design, and layering provenance
+932
+394
doc/discovery/config-sourcing.md
+394
doc/discovery/config-sourcing.md
···
1
+
# Configuration Sourcing in c12
2
+
3
+
This document explains how c12 layers together configuration from various sources, using `defu` for deep merging.
4
+
5
+
## Overview
6
+
7
+
c12's configuration loading follows a multi-stage pipeline that sources configuration from multiple locations, merges them using `unjs/defu`, and applies transformations.
8
+
9
+
## Execution Pipeline
10
+
11
+
```mermaid
12
+
flowchart TD
13
+
Start[Start: loadConfig] --> Normalize[Normalize options]
14
+
Normalize --> SetupMerger[Setup merger<br/>(defu or custom)]
15
+
SetupMerger --> LoadEnv{dotenv enabled?}
16
+
LoadEnv -->|yes| LoadDotenv[Load .env files]
17
+
LoadDotenv --> MainConfig
18
+
LoadEnv -->|no| MainConfig[Load main config file<br/>via resolveConfig]
19
+
MainConfig --> LoadRC{rcFile enabled?}
20
+
LoadRC -->|yes| LoadRcFiles[Load RC files<br/>cwd, workspace, home]
21
+
LoadRcFiles --> MergeRC[Merge RC sources<br/>with defu]
22
+
LoadRC -->|no| PackageJson
23
+
MergeRC --> PackageJson{packageJson enabled?}
24
+
PackageJson -->|yes| LoadPkg[Read package.json]
25
+
LoadPkg --> MergePkg[Merge package.json values<br/>with defu]
26
+
PackageJson -->|no| ResolveFuncs
27
+
MergePkg --> ResolveFuncs[Resolve functions<br/>in rawConfigs]
28
+
ResolveFuncs --> Combine{Main config is array?}
29
+
Combine -->|yes| UseArray[Use array directly<br/>no merging]
30
+
Combine -->|no| MergeSources[Merge sources with defu<br/>overrides → main → rc →<br/>packageJson → defaultConfig]
31
+
MergeSources --> Extend{extends enabled?}
32
+
Extend -->|yes| ProcessExtends[Process extends<br/>recursively]
33
+
ProcessExtends --> MergeLayers[Merge extended<br/>layers with defu]
34
+
Extend -->|no| ApplyDefaults
35
+
MergeLayers --> ApplyDefaults
36
+
ApplyDefaults --> ApplyDefaults{defaults provided?}
37
+
ApplyDefaults -->|yes| MergeDefaults[Merge defaults<br/>with defu]
38
+
ApplyDefaults -->|no| Cleanup
39
+
MergeDefaults --> Cleanup{omit$Keys enabled?}
40
+
Cleanup -->|yes| RemoveDollar[Remove $ prefixed keys]
41
+
Cleanup -->|no| Verify
42
+
RemoveDollar --> Verify{configFileRequired?}
43
+
Verify -->|yes| CheckExists[Check file exists<br/>or throw error]
44
+
Verify -->|no| Return
45
+
CheckExists --> Return[Return resolved config]
46
+
```
47
+
48
+
## Main Config Loading Flow
49
+
50
+
```mermaid
51
+
flowchart TD
52
+
subgraph LoadConfig [loadConfig function]
53
+
direction TB
54
+
A[Normalized options] --> B[Load dotenv]
55
+
B --> C[Load main config]
56
+
C --> D[Load RC files]
57
+
D --> E[Load package.json]
58
+
E --> F[Resolve config functions]
59
+
F --> G[Merge all sources]
60
+
G --> H[Process extends]
61
+
H --> I[Apply defaults]
62
+
I --> J[Final cleanup]
63
+
J --> K[Return resolved config]
64
+
end
65
+
```
66
+
67
+
## Config Sources Priority
68
+
69
+
When all sources are present, c12 merges them in this order (highest to lowest priority):
70
+
71
+
```mermaid
72
+
flowchart LR
73
+
overrides[overrides<br/>Highest Priority] --> main[main config file]
74
+
main --> rc[RC files]
75
+
rc --> packageJson[package.json]
76
+
packageJson --> defaultConfig[defaultConfig<br/>Lowest Priority]
77
+
```
78
+
79
+
The merge happens at `src/loader.ts:158-163`:
80
+
81
+
```typescript
82
+
r.config = _merger(
83
+
configs.overrides,
84
+
configs.main,
85
+
configs.rc,
86
+
configs.packageJson,
87
+
configs.defaultConfig,
88
+
) as T;
89
+
```
90
+
91
+
## Where `defu` is Used
92
+
93
+
`defu` (or a custom merger) is used at several points in the pipeline:
94
+
95
+
### 1. Main Merger Setup
96
+
**Location**: `src/loader.ts:71`
97
+
```typescript
98
+
const _merger = options.merger || defu;
99
+
```
100
+
101
+
### 2. RC File Merging
102
+
**Location**: `src/loader.ts:128`
103
+
RC files from cwd, workspace, and home are merged:
104
+
```typescript
105
+
rawConfigs.rc = _merger({} as T, ...rcSources);
106
+
```
107
+
108
+
### 3. package.json Value Merging
109
+
**Location**: `src/loader.ts:140`
110
+
Multiple keys from package.json are merged:
111
+
```typescript
112
+
rawConfigs.packageJson = _merger({} as T, ...values);
113
+
```
114
+
115
+
### 4. Main Source Merging
116
+
**Location**: `src/loader.ts:158-163`
117
+
All primary config sources are merged:
118
+
```typescript
119
+
r.config = _merger(
120
+
configs.overrides,
121
+
configs.main,
122
+
configs.rc,
123
+
configs.packageJson,
124
+
configs.defaultConfig,
125
+
) as T;
126
+
```
127
+
128
+
### 5. Extended Layers Merging
129
+
**Location**: `src/loader.ts:171`
130
+
After processing `extends`, all layers are merged into the main config:
131
+
```typescript
132
+
r.config = _merger(r.config, ...r.layers!.map((e) => e.config)) as T;
133
+
```
134
+
135
+
### 6. Defaults Application
136
+
**Location**: `src/loader.ts:194`
137
+
Default config has the lowest priority:
138
+
```typescript
139
+
r.config = _merger(r.config, options.defaults) as T;
140
+
```
141
+
142
+
### 7. Environment-Specific Config Merging
143
+
**Location**: `src/loader.ts:418` (in `resolveConfig`)
144
+
Env-specific config overrides the base config:
145
+
```typescript
146
+
res.config = _merger(envConfig, res.config);
147
+
```
148
+
149
+
### 8. Meta Merging
150
+
**Location**: `src/loader.ts:423` (in `resolveConfig`)
151
+
Meta from source options and config are merged:
152
+
```typescript
153
+
res.meta = defu(res.sourceOptions!.meta, res.config!.$meta) as MT;
154
+
```
155
+
156
+
### 9. Source Overrides Merging
157
+
**Location**: `src/loader.ts:428` (in `resolveConfig`)
158
+
Per-source overrides are applied:
159
+
```typescript
160
+
res.config = _merger(res.sourceOptions!.overrides, res.config) as T;
161
+
```
162
+
163
+
## resolveConfig: Loading Individual Config Layers
164
+
165
+
The `resolveConfig` function handles loading individual configuration files (including extended configs):
166
+
167
+
```mermaid
168
+
flowchart TD
169
+
Start[resolveConfig start] --> CustomResolver{custom resolver?}
170
+
CustomResolver -->|yes| TryCustom[Try custom resolver]
171
+
TryCustom --> HasResult{result?}
172
+
HasResult -->|yes| ReturnCustom[Return custom result]
173
+
HasResult -->|no| GigetCheck
174
+
CustomResolver -->|no| GigetCheck{giget URI?}
175
+
GigetCheck -->|yes| Download[Download with giget<br/>to local path]
176
+
Download --> NpmCheck
177
+
GigetCheck -->|no| NpmCheck{npm package?}
178
+
NpmCheck -->|yes| ResolvePkg[Resolve npm package]
179
+
ResolvePkg --> LocalImport
180
+
NpmCheck -->|no| LocalImport[Import from local FS]
181
+
LocalImport --> GetExt{has extension?}
182
+
GetExt -->|no| UseDir[Treat as directory<br/>use configFile name]
183
+
GetExt -->|yes| TryResolve
184
+
UseDir --> TryResolve[Try resolve with<br/>multiple paths]
185
+
TryResolve --> FileExists{file exists?}
186
+
FileExists -->|no| ReturnEmpty[Return empty config]
187
+
FileExists -->|yes| CheckFormat
188
+
CheckFormat{Async loader?}
189
+
CheckFormat -->|yes| ParseAsync[Parse with<br/>confbox parsers]
190
+
CheckFormat -->|no| ImportModule[Import module<br/>with jiti fallback]
191
+
ImportModule --> IsFunction{is function?}
192
+
ParseAsync --> IsFunction
193
+
IsFunction -->|yes| CallFunction[Call with context]
194
+
IsFunction -->|no| EnvCheck
195
+
CallFunction --> EnvCheck
196
+
EnvCheck{envName set?}
197
+
EnvCheck -->|yes| MergeEnv[Merge env-specific<br/>config with defu]
198
+
EnvCheck -->|no| MergeMeta
199
+
MergeEnv --> MergeMeta[Merge meta with defu]
200
+
MergeMeta --> SourceOverrides{source overrides?}
201
+
SourceOverrides -->|yes| ApplyOverrides[Apply with defu]
202
+
SourceOverrides -->|no| NormalizePaths
203
+
ApplyOverrides --> NormalizePaths[Normalize paths]
204
+
NormalizePaths --> ReturnResult[Return resolved config]
205
+
```
206
+
207
+
## Environment-Specific Configuration
208
+
209
+
Each config layer can define environment-specific overrides:
210
+
211
+
```mermaid
212
+
flowchart LR
213
+
Config[Config object] --> HasEnv{Has envName?}
214
+
HasEnv -->|no| Skip[Skip env merging]
215
+
HasEnv -->|yes| CheckKeys{Has $<envName><br/>or $env.<envName>?}
216
+
CheckKeys -->|yes| ExtractEnv[Extract env config]
217
+
ExtractEnv --> MergeEnv[Merge env config<br/>over base with defu]
218
+
CheckKeys -->|no| Skip
219
+
MergeEnv --> Skip
220
+
Skip --> Next[Continue pipeline]
221
+
```
222
+
223
+
The lookup order for env-specific config is (per `src/loader.ts:413-415`):
224
+
1. `config.$<envName>` (e.g., `$production`)
225
+
2. `config.$env.<envName>` (e.g., `$env.staging`)
226
+
227
+
## RC File Loading
228
+
229
+
RC files are loaded from multiple locations (if `globalRc` is enabled):
230
+
231
+
```mermaid
232
+
flowchart TD
233
+
Start[Load RC files] --> Cwd[Load from cwd]
234
+
Cwd --> Workspace{globalRc enabled?}
235
+
Workspace -->|yes| FindWorkspace[Find workspace dir]
236
+
FindWorkspace --> LoadWorkspace[Load from workspace]
237
+
Workspace -->|no| Home
238
+
LoadWorkspace --> Home{globalRc enabled?}
239
+
Home -->|yes| LoadHome[Load from user home<br/>via rc9.readUser]
240
+
Home -->|no| Merge
241
+
LoadHome --> Merge[Merge all RC sources<br/>with defu]
242
+
Merge --> End[Return merged RC config]
243
+
```
244
+
245
+
RC file loading uses the `rc9` package, which reads from:
246
+
1. `cwd/.<name>rc`
247
+
2. Workspace root `.<name>rc` (if `globalRc`)
248
+
3. User home directory `.<name>rc` (if `globalRc`)
249
+
250
+
## Extended Configuration Processing
251
+
252
+
The `extends` feature allows configs to inherit from other configs:
253
+
254
+
```mermaid
255
+
flowchart TD
256
+
Start[extendConfig] --> FindExtends{Has extends key?}
257
+
FindExtends -->|no| End[Return]
258
+
FindExtends -->|yes| ExtractSources[Extract extend sources]
259
+
ExtractSources --> LoopSources[For each source]
260
+
LoopSources --> CheckFormat{Format?}
261
+
CheckFormats -->|{source, options}| Extract2[Extract source/options]
262
+
CheckFormats -->|[source, options]| Extract2
263
+
CheckFormats -->|string| ResolveSource
264
+
Extract2 --> ResolveSource
265
+
ResolveSource --> RemoteCheck{Remote URI?}
266
+
RemoteCheck -->|yes| Download[Download with giget]
267
+
RemoteCheck -->|no| NpmCheck
268
+
Download --> NpmCheck{npm package?}
269
+
NpmCheck -->|yes| ResolvePkg[Resolve package]
270
+
NpmCheck -->|no| LocalPath
271
+
ResolvePkg --> LocalPath[Use local path]
272
+
LocalPath --> ResolveConfig2[Call resolveConfig]
273
+
ResolveConfig2 --> RecursiveExtend[Recursive extendConfig<br/>on base]
274
+
RecursiveExtend --> PushLayer[Push to _layers array]
275
+
PushLayer --> NextSource{More sources?}
276
+
NextSource -->|yes| LoopSources
277
+
NextSource -->|no| MergeLayers[Merge layers<br/>with defu]
278
+
MergeLayers --> End
279
+
```
280
+
281
+
## dotenv Integration
282
+
283
+
Environment variables are loaded before any config files (per `src/loader.ts:94-100`):
284
+
285
+
```mermaid
286
+
flowchart TD
287
+
Start[setupDotenv] --> LoadFiles[Load .env files]
288
+
LoadFiles --> ParseFiles[Parse with<br/>node:util.parseEnv]
289
+
ParseFiles --> FileRefs{expandFileReferences?}
290
+
FileRefs -->|yes| ExpandFiles[Read _FILE vars<br/>from disk]
291
+
FileRefs -->|no| Interpolate
292
+
ExpandFiles --> Interpolate{interpolate?}
293
+
Interpolate -->|yes| ExpandVars[Expand ${VAR}<br/>references]
294
+
Interpolate -->|no| ApplyToEnv
295
+
ExpandVars --> ApplyToEnv[Apply to process.env]
296
+
ApplyToEnv --> End[Return]
297
+
```
298
+
299
+
The dotenv loading happens **before** any config files, allowing config files to reference environment variables.
300
+
301
+
## Complete Data Flow
302
+
303
+
```mermaid
304
+
sequenceDiagram
305
+
participant User
306
+
participant LoadConfig as loadConfig()
307
+
participant Dotenv as setupDotenv()
308
+
participant Resolve as resolveConfig()
309
+
participant Defu as defu (merger)
310
+
participant RC9 as rc9
311
+
participant PkgTypes as pkg-types
312
+
313
+
User->>LoadConfig: Call with options
314
+
LoadConfig->>LoadConfig: Normalize options
315
+
LoadConfig->>LoadConfig: Setup merger (defu or custom)
316
+
317
+
opt dotenv enabled
318
+
LoadConfig->>Dotenv: setupDotenv(options)
319
+
Dotenv-->>LoadConfig: process.env populated
320
+
end
321
+
322
+
LoadConfig->>Resolve: resolveConfig(".", options)
323
+
Resolve-->>LoadConfig: Main config object
324
+
325
+
opt rcFile enabled
326
+
LoadConfig->>RC9: rc9.read({ cwd })
327
+
opt globalRc enabled
328
+
LoadConfig->>PkgTypes: findWorkspaceDir()
329
+
PkgTypes-->>LoadConfig: workspace path
330
+
LoadConfig->>RC9: rc9.read({ workspace })
331
+
LoadConfig->>RC9: rc9.readUser()
332
+
end
333
+
LoadConfig->>Defu: Merge all RC sources
334
+
Defu-->>LoadConfig: Merged RC config
335
+
end
336
+
337
+
opt packageJson enabled
338
+
LoadConfig->>PkgTypes: readPackageJSON()
339
+
PkgTypes-->>LoadConfig: package.json object
340
+
LoadConfig->>Defu: Merge package.json values
341
+
Defu-->>LoadConfig: Merged pkg config
342
+
end
343
+
344
+
LoadConfig->>LoadConfig: Resolve config functions
345
+
346
+
LoadConfig->>Defu: Merge all sources
347
+
Note over Defu: overrides → main → rc →<br/>packageJson → defaultConfig
348
+
Defu-->>LoadConfig: Merged config
349
+
350
+
opt extends enabled
351
+
loop Each extend source
352
+
LoadConfig->>Resolve: resolveConfig(source)
353
+
Resolve-->>LoadConfig: Extended layer
354
+
end
355
+
LoadConfig->>Defu: Merge layers into config
356
+
Defu-->>LoadConfig: Extended config
357
+
end
358
+
359
+
opt defaults provided
360
+
LoadConfig->>Defu: Merge defaults
361
+
Defu-->>LoadConfig: Final config
362
+
end
363
+
364
+
opt omit$Keys enabled
365
+
LoadConfig->>LoadConfig: Remove $ prefixed keys
366
+
end
367
+
368
+
LoadConfig-->>User: Resolved config + layers
369
+
```
370
+
371
+
## Key Files
372
+
373
+
| File | Purpose |
374
+
|------|---------|
375
+
| `src/loader.ts` | Main `loadConfig()` function and `resolveConfig()` |
376
+
| `src/dotenv.ts` | Environment variable loading (`setupDotenv`, `loadDotenv`) |
377
+
| `src/types.ts` | TypeScript type definitions |
378
+
| `src/watch.ts` | Config watching with file system events |
379
+
380
+
## Summary
381
+
382
+
The configuration pipeline in c12 is:
383
+
384
+
1. **Normalize options** - Set defaults and normalize paths
385
+
2. **Load environment variables** - Parse `.env` files and populate `process.env`
386
+
3. **Load main config** - Find and import the primary config file
387
+
4. **Load RC files** - Read from cwd, workspace, and home directories
388
+
5. **Load package.json** - Extract config values from package.json
389
+
6. **Merge all sources** - Use `defu` to merge in priority order
390
+
7. **Process extends** - Recursively load and merge extended configs
391
+
8. **Apply defaults** - Merge lowest-priority defaults
392
+
9. **Cleanup** - Remove internal `$` keys if requested
393
+
394
+
`defu` is the core merging function used throughout the pipeline to ensure deep, predictable merging of configuration objects from all sources.
+538
doc/discovery/dynamic-configs.md
+538
doc/discovery/dynamic-configs.md
···
1
+
# Dynamic Configuration Sources in c12
2
+
3
+
This document explores how c12 could support more dynamic configuration sources, such as drop-in config directories (systemd-style `.d` directories), and how configuration layering could be made more first-class.
4
+
5
+
## Background
6
+
7
+
Currently, c12 has a **fixed set of configuration sources** defined in `ConfigSource`:
8
+
9
+
```typescript
10
+
export type ConfigSource = "overrides" | "main" | "rc" | "packageJson" | "defaultConfig";
11
+
```
12
+
13
+
These sources are loaded in a **hardcoded order** within `loadConfig()` at `src/loader.ts:83-92`:
14
+
15
+
```typescript
16
+
const rawConfigs: Record<
17
+
ConfigSource,
18
+
ResolvableConfig<T> | null | undefined
19
+
> = {
20
+
overrides: options.overrides,
21
+
main: undefined,
22
+
rc: undefined,
23
+
packageJson: undefined,
24
+
defaultConfig: options.defaultConfig,
25
+
};
26
+
```
27
+
28
+
This works well for the common case, but lacks flexibility for dynamic configuration discovery.
29
+
30
+
## The Use Case: Drop-in Config Directories
31
+
32
+
Inspired by systemd's drop-in configuration pattern, this feature would allow:
33
+
34
+
1. A main config file: `myapp.config.ts`
35
+
2. A drop-in directory: `myapp.config.d/`
36
+
3. Individual override files in the directory:
37
+
- `myapp.config.d/10-admin-overrides.ts`
38
+
- `myapp.config.d/20-production.ts`
39
+
- `myapp.config.d/99-local.ts`
40
+
41
+
Files in the `.d` directory are merged in **lexicographic order**, with later files overriding earlier ones. This allows:
42
+
- System administrators to layer configurations without modifying the base config
43
+
- Easy enable/disable by adding/removing files
44
+
- Clear provenance of where config values came from
45
+
46
+
## Current Limitations
47
+
48
+
### Fixed Source Types
49
+
50
+
The `ConfigSource` type is a union literal, which means:
51
+
- New sources require type changes
52
+
- Cannot dynamically add sources at runtime
53
+
- Source ordering is fixed
54
+
55
+
### Limited Source Metadata
56
+
57
+
While `ConfigLayer` exists and has `meta` field, it's used differently than a comprehensive provenance system:
58
+
59
+
```typescript
60
+
export interface ConfigLayer<
61
+
T extends UserInputConfig = UserInputConfig,
62
+
MT extends ConfigLayerMeta = ConfigLayerMeta,
63
+
> {
64
+
config: T | null;
65
+
source?: string;
66
+
sourceOptions?: SourceOptions<T, MT>;
67
+
meta?: MT;
68
+
cwd?: string;
69
+
configFile?: string;
70
+
}
71
+
```
72
+
73
+
The `meta` field is primarily for user-defined metadata, not automatic tracking of:
74
+
- Which provider provided the value
75
+
- Where in the merge order the value came from
76
+
- Priority/ranking of the source
77
+
78
+
## Inspiration: Rust's Figment Crate
79
+
80
+
Rust's [figment](https://docs.rs/figment/latest/figment/) takes a more flexible approach:
81
+
82
+
### Provider Trait
83
+
84
+
Any type can implement the `Provider` trait to become a configuration source:
85
+
86
+
```rust
87
+
trait Provider {
88
+
fn metadata(&self) -> Metadata;
89
+
fn data(&self) -> Result<Map<Profile, Dict>, Error>;
90
+
fn profile(&self) -> Option<Profile>;
91
+
}
92
+
```
93
+
94
+
### Metadata Tracking
95
+
96
+
Every value is tagged with `Metadata`:
97
+
98
+
```rust
99
+
pub struct Metadata {
100
+
pub name: Cow<'static, str>, // "TOML File"
101
+
pub source: Option<Source>, // Path, URL, etc.
102
+
pub provide_location: Option<&'static Location<'static>>,
103
+
}
104
+
```
105
+
106
+
This allows:
107
+
- Rich error messages showing exactly where a value came from
108
+
- "Magic" values like `RelativePathBuf` that know their config file location
109
+
- Debugging complex configurations
110
+
111
+
### Third-Party Providers
112
+
113
+
The ecosystem can provide custom providers:
114
+
- `figment-directory` - Config from directories
115
+
- `figment-file-provider-adapter` - Reads `_FILE` suffix variables
116
+
- Custom providers for any data source
117
+
118
+
## Potential Design for c12
119
+
120
+
### Option 1: Provider Pattern
121
+
122
+
Introduce a `ConfigProvider` interface:
123
+
124
+
```typescript
125
+
export interface ConfigProvider<T = UserInputConfig> {
126
+
/** Unique identifier for this provider */
127
+
name: string;
128
+
129
+
/** Metadata about this provider */
130
+
metadata: ConfigProviderMetadata;
131
+
132
+
/** Load configuration from this provider */
133
+
load(context: ConfigProviderContext): Promise<T | null | undefined>;
134
+
135
+
/** Priority (lower = higher priority) */
136
+
priority?: number;
137
+
138
+
/** Should this provider be enabled? */
139
+
enabled?(options: LoadConfigOptions): boolean;
140
+
}
141
+
142
+
export interface ConfigProviderMetadata {
143
+
name: string;
144
+
source?: string;
145
+
description?: string;
146
+
}
147
+
148
+
export interface ConfigProviderContext {
149
+
cwd: string;
150
+
envName: string | false;
151
+
[key: string]: any;
152
+
}
153
+
```
154
+
155
+
#### Drop-in Directory Provider Example
156
+
157
+
```typescript
158
+
class DropInDirProvider<T extends UserInputConfig> implements ConfigProvider<T> {
159
+
name = "drop-in-directory";
160
+
161
+
constructor(
162
+
private basePath: string,
163
+
private configName: string,
164
+
private pattern: string = "*.config.{ts,js,json,yaml,yml}",
165
+
) {}
166
+
167
+
metadata = {
168
+
name: "Drop-in Directory",
169
+
source: this.basePath,
170
+
};
171
+
172
+
priority = 50; // Between RC and package.json
173
+
174
+
async load(context: ConfigProviderContext): Promise<T | null> {
175
+
const dropInDir = path.join(context.cwd, `${this.configName}.d`);
176
+
177
+
if (!fs.existsSync(dropInDir)) {
178
+
return null;
179
+
}
180
+
181
+
const files = await glob(this.pattern, { cwd: dropInDir });
182
+
const sorted = files.sort(); // Lexicographic order
183
+
184
+
let merged: Partial<T> = {};
185
+
for (const file of sorted) {
186
+
const config = await loadConfigFile(path.join(dropInDir, file));
187
+
merged = defu(merged, config);
188
+
}
189
+
190
+
return merged as T;
191
+
}
192
+
}
193
+
```
194
+
195
+
### Option 2: Source Registry
196
+
197
+
Add a source registration system to `LoadConfigOptions`:
198
+
199
+
```typescript
200
+
export interface LoadConfigOptions<T, MT> {
201
+
// Existing options...
202
+
203
+
/**
204
+
* Register additional config sources
205
+
* Sources are loaded in order of priority (lowest first)
206
+
*/
207
+
sources?: ConfigSourceEntry<T, MT>[];
208
+
}
209
+
210
+
export interface ConfigSourceEntry<T, MT> {
211
+
/** Unique identifier */
212
+
id: string;
213
+
214
+
/** Provider function returning config */
215
+
provider: ResolvableConfig<T>;
216
+
217
+
/** Priority (lower = higher priority) */
218
+
priority: number;
219
+
220
+
/** Whether this source should be loaded */
221
+
condition?: (options: LoadConfigOptions<T, MT>) => boolean;
222
+
223
+
/** Metadata about this source */
224
+
metadata?: Partial<ConfigLayerMeta>;
225
+
}
226
+
```
227
+
228
+
#### Usage Example
229
+
230
+
```typescript
231
+
const config = await loadConfig({
232
+
name: "myapp",
233
+
234
+
sources: [
235
+
{
236
+
id: "overrides",
237
+
priority: 10,
238
+
provider: { custom: "overrides" },
239
+
},
240
+
{
241
+
id: "drop-in-dir",
242
+
priority: 20,
243
+
condition: (opts) => opts.envName === "production",
244
+
provider: async (ctx) => {
245
+
const dropInDir = path.join(opts.cwd, "myapp.config.d");
246
+
return await loadDropInConfigs(dropInDir);
247
+
},
248
+
metadata: { name: "Drop-in Directory" },
249
+
},
250
+
{
251
+
id: "main",
252
+
priority: 30,
253
+
provider: { custom: "main" }, // Built-in loader
254
+
},
255
+
],
256
+
});
257
+
```
258
+
259
+
### Option 3: Enhanced Built-in Sources
260
+
261
+
Add a new `dropIn` option to the existing `LoadConfigOptions`:
262
+
263
+
```typescript
264
+
export interface LoadConfigOptions<T, MT> {
265
+
// Existing options...
266
+
267
+
/**
268
+
* Load config from a .d directory
269
+
* Files are merged in lexicographic order
270
+
*/
271
+
dropIn?: boolean | string | DropInOptions;
272
+
}
273
+
274
+
export interface DropInOptions {
275
+
/** Directory name (default: <configFile>.d) */
276
+
dir?: string;
277
+
278
+
/** File pattern to match */
279
+
pattern?: string;
280
+
281
+
/** Where in priority order to insert (default: after main, before defaults) */
282
+
insertAfter?: "main" | "rc" | "packageJson";
283
+
284
+
/** Enable only for specific environments */
285
+
env?: string[];
286
+
}
287
+
```
288
+
289
+
#### Usage
290
+
291
+
```typescript
292
+
// Simple: auto-detect myapp.config.d/
293
+
await loadConfig({ name: "myapp", dropIn: true });
294
+
295
+
// Custom directory
296
+
await loadConfig({
297
+
name: "myapp",
298
+
dropIn: { dir: "config.overrides.d" },
299
+
});
300
+
301
+
// Environment-specific
302
+
await loadConfig({
303
+
name: "myapp",
304
+
dropIn: { env: ["production", "staging"] },
305
+
});
306
+
```
307
+
308
+
## Layering Provenance
309
+
310
+
To make layering "first class" as mentioned in issue #298, we need:
311
+
312
+
### Enhanced Layer Metadata
313
+
314
+
```typescript
315
+
export interface ConfigLayer<T, MT> {
316
+
config: T | null;
317
+
source?: string;
318
+
sourceOptions?: SourceOptions<T, MT>;
319
+
meta?: ConfigLayerMeta; // User metadata
320
+
cwd?: string;
321
+
configFile?: string;
322
+
323
+
// New fields for provenance:
324
+
provider: string; // "main", "drop-in", "rc", etc.
325
+
priority: number; // Merge order
326
+
loadTime: Date; // When it was loaded
327
+
fingerprint?: string; // Content hash for change detection
328
+
}
329
+
```
330
+
331
+
### Layer Tree Visualization
332
+
333
+
A resolved config could show its full provenance:
334
+
335
+
```typescript
336
+
{
337
+
config: { /* merged config */ },
338
+
layers: [
339
+
{ provider: "overrides", priority: 10, configFile: undefined, ... },
340
+
{ provider: "drop-in", priority: 20, configFile: "./myapp.config.d/10-admin.ts", ... },
341
+
{ provider: "drop-in", priority: 20, configFile: "./myapp.config.d/20-production.ts", ... },
342
+
{ provider: "drop-in", priority: 20, configFile: "./myapp.config.d/99-local.ts", ... },
343
+
{ provider: "main", priority: 30, configFile: "./myapp.config.ts", ... },
344
+
{ provider: "rc", priority: 40, configFile: "/home/user/.myapprc", ... },
345
+
]
346
+
}
347
+
```
348
+
349
+
### Value Attribution
350
+
351
+
For debugging, we could trace where each config value came from:
352
+
353
+
```typescript
354
+
function traceValue(
355
+
config: ResolvedConfig,
356
+
keyPath: string[]
357
+
): ConfigLayer | undefined {
358
+
// Search layers in reverse priority order
359
+
for (const layer of [...config.layers].reverse()) {
360
+
let value = layer.config;
361
+
for (const key of keyPath) {
362
+
value = value?.[key];
363
+
}
364
+
if (value !== undefined) {
365
+
return layer;
366
+
}
367
+
}
368
+
return undefined;
369
+
}
370
+
371
+
// Usage:
372
+
const source = traceValue(config, ["server", "port"]);
373
+
console.log(`server.port came from: ${source.configFile}`);
374
+
```
375
+
376
+
## Implementation Considerations
377
+
378
+
### Backward Compatibility
379
+
380
+
Any changes should maintain backward compatibility:
381
+
382
+
1. Default behavior unchanged - drop-ins disabled by default
383
+
2. Existing `ConfigSource` type could remain for internal use
384
+
3. New options as additions only
385
+
386
+
### Performance
387
+
388
+
Loading many small files has overhead:
389
+
390
+
1. Consider caching resolved configs
391
+
2. Watch mode should efficiently track file additions/removals
392
+
3. Content fingerprinting for change detection
393
+
394
+
### Error Handling
395
+
396
+
With more sources, errors are more likely:
397
+
398
+
```typescript
399
+
// Per-layer error tracking
400
+
export interface ConfigLayer<T, MT> {
401
+
// ...
402
+
error?: {
403
+
message: string;
404
+
source: string;
405
+
recoverable: boolean;
406
+
};
407
+
}
408
+
409
+
// Strict vs relaxed modes
410
+
await loadConfig({
411
+
strict: true, // Fail on any source error
412
+
// or
413
+
strict: false, // Log warnings, continue
414
+
});
415
+
```
416
+
417
+
### File Naming Conventions
418
+
419
+
For `.d` directories, establish conventions:
420
+
421
+
```typescript
422
+
// Numeric prefix for ordering:
423
+
// 00-base.conf
424
+
// 10-admin.conf
425
+
// 20-deployment.conf
426
+
// 99-local.conf
427
+
428
+
// Or use alphanumeric sorting:
429
+
// admin.conf
430
+
// base.conf
431
+
// local.conf
432
+
// production.conf
433
+
```
434
+
435
+
## Related Enhancements
436
+
437
+
### Watch Mode Integration
438
+
439
+
For drop-in directories, watch mode should:
440
+
- Detect new files added
441
+
- Detect files removed
442
+
- Detect files renamed
443
+
- Re-merge in correct order when changes occur
444
+
445
+
```typescript
446
+
watchConfig({
447
+
name: "myapp",
448
+
dropIn: true,
449
+
onWatch: (event) => {
450
+
console.log(`Drop-in file ${event.type}: ${event.path}`);
451
+
},
452
+
});
453
+
```
454
+
455
+
### Configuration Validation
456
+
457
+
With layered configs, validation should:
458
+
459
+
1. Validate each layer independently
460
+
2. Validate the final merged result
461
+
3. Show which layer introduced validation errors
462
+
463
+
```typescript
464
+
const config = await loadConfig({
465
+
name: "myapp",
466
+
validate: (layer, merged) => {
467
+
// Check layer-specific constraints
468
+
// Check final merged constraints
469
+
},
470
+
});
471
+
```
472
+
473
+
## Existing Workarounds
474
+
475
+
Before dynamic sources are implemented, you can:
476
+
477
+
### Use `extends` for Multiple Files
478
+
479
+
```typescript
480
+
// main.config.ts
481
+
export default {
482
+
extends: [
483
+
"./configs/base.config",
484
+
"./configs/admin.config",
485
+
"./configs/production.config",
486
+
],
487
+
// ...
488
+
};
489
+
```
490
+
491
+
### Use Custom Resolver
492
+
493
+
```typescript
494
+
const config = await loadConfig({
495
+
name: "myapp",
496
+
async resolve(source, options) {
497
+
if (source.startsWith("drop-in:")) {
498
+
const dir = source.replace("drop-in:", "");
499
+
return await loadDropInConfigs(dir);
500
+
}
501
+
return null; // Use default resolution
502
+
},
503
+
504
+
// Then use it via extends
505
+
overrides: (ctx) => ({
506
+
extends: ["drop-in:./myapp.config.d"],
507
+
}),
508
+
});
509
+
```
510
+
511
+
### Merge Multiple Loads
512
+
513
+
```typescript
514
+
const [main, admin, local] = await Promise.all([
515
+
loadConfig({ name: "myapp" }),
516
+
loadConfig({ name: "admin-overrides" }),
517
+
loadConfig({ name: "local-overrides" }),
518
+
]);
519
+
520
+
const merged = defu(local.config, admin.config, main.config);
521
+
```
522
+
523
+
## Summary
524
+
525
+
Dynamic configuration sources in c12 would enable:
526
+
527
+
1. **Drop-in config directories** - Systemd-style `.d` directories
528
+
2. **Custom providers** - Third-party data sources
529
+
3. **Enhanced provenance** - Clear tracking of where values came from
530
+
4. **Flexible ordering** - Configurable source priorities
531
+
5. **Better debugging** - Layer-by-layer inspection
532
+
533
+
The key design decision is between:
534
+
- **Provider pattern** - Maximum flexibility, ecosystem growth
535
+
- **Enhanced built-ins** - Simpler API, controlled feature set
536
+
- **Source registry** - Middle ground with explicit registration
537
+
538
+
All approaches should maintain backward compatibility while enabling the dynamic configuration discovery that makes layering a first-class concept.