Smart configuration loader
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

Add documentation for config sourcing and dynamic configs

- doc/discovery/config-sourcing.md: Complete analysis of c12's configuration pipeline, showing how defu merges configs from multiple sources with mermaid diagrams
- doc/discovery/dynamic-configs.md: Exploration of dynamic configuration sources including drop-in directories, provider pattern design, and layering provenance

rektide c44c0e55 246eb2b0

+932
+394
doc/discovery/config-sourcing.md
··· 1 + # Configuration Sourcing in c12 2 + 3 + This document explains how c12 layers together configuration from various sources, using `defu` for deep merging. 4 + 5 + ## Overview 6 + 7 + c12's configuration loading follows a multi-stage pipeline that sources configuration from multiple locations, merges them using `unjs/defu`, and applies transformations. 8 + 9 + ## Execution Pipeline 10 + 11 + ```mermaid 12 + flowchart TD 13 + Start[Start: loadConfig] --> Normalize[Normalize options] 14 + Normalize --> SetupMerger[Setup merger<br/>(defu or custom)] 15 + SetupMerger --> LoadEnv{dotenv enabled?} 16 + LoadEnv -->|yes| LoadDotenv[Load .env files] 17 + LoadDotenv --> MainConfig 18 + LoadEnv -->|no| MainConfig[Load main config file<br/>via resolveConfig] 19 + MainConfig --> LoadRC{rcFile enabled?} 20 + LoadRC -->|yes| LoadRcFiles[Load RC files<br/>cwd, workspace, home] 21 + LoadRcFiles --> MergeRC[Merge RC sources<br/>with defu] 22 + LoadRC -->|no| PackageJson 23 + MergeRC --> PackageJson{packageJson enabled?} 24 + PackageJson -->|yes| LoadPkg[Read package.json] 25 + LoadPkg --> MergePkg[Merge package.json values<br/>with defu] 26 + PackageJson -->|no| ResolveFuncs 27 + MergePkg --> ResolveFuncs[Resolve functions<br/>in rawConfigs] 28 + ResolveFuncs --> Combine{Main config is array?} 29 + Combine -->|yes| UseArray[Use array directly<br/>no merging] 30 + Combine -->|no| MergeSources[Merge sources with defu<br/>overrides → main → rc →<br/>packageJson → defaultConfig] 31 + MergeSources --> Extend{extends enabled?} 32 + Extend -->|yes| ProcessExtends[Process extends<br/>recursively] 33 + ProcessExtends --> MergeLayers[Merge extended<br/>layers with defu] 34 + Extend -->|no| ApplyDefaults 35 + MergeLayers --> ApplyDefaults 36 + ApplyDefaults --> ApplyDefaults{defaults provided?} 37 + ApplyDefaults -->|yes| MergeDefaults[Merge defaults<br/>with defu] 38 + ApplyDefaults -->|no| Cleanup 39 + MergeDefaults --> Cleanup{omit$Keys enabled?} 40 + Cleanup -->|yes| RemoveDollar[Remove $ prefixed keys] 41 + Cleanup -->|no| Verify 42 + RemoveDollar --> Verify{configFileRequired?} 43 + Verify -->|yes| CheckExists[Check file exists<br/>or throw error] 44 + Verify -->|no| Return 45 + CheckExists --> Return[Return resolved config] 46 + ``` 47 + 48 + ## Main Config Loading Flow 49 + 50 + ```mermaid 51 + flowchart TD 52 + subgraph LoadConfig [loadConfig function] 53 + direction TB 54 + A[Normalized options] --> B[Load dotenv] 55 + B --> C[Load main config] 56 + C --> D[Load RC files] 57 + D --> E[Load package.json] 58 + E --> F[Resolve config functions] 59 + F --> G[Merge all sources] 60 + G --> H[Process extends] 61 + H --> I[Apply defaults] 62 + I --> J[Final cleanup] 63 + J --> K[Return resolved config] 64 + end 65 + ``` 66 + 67 + ## Config Sources Priority 68 + 69 + When all sources are present, c12 merges them in this order (highest to lowest priority): 70 + 71 + ```mermaid 72 + flowchart LR 73 + overrides[overrides<br/>Highest Priority] --> main[main config file] 74 + main --> rc[RC files] 75 + rc --> packageJson[package.json] 76 + packageJson --> defaultConfig[defaultConfig<br/>Lowest Priority] 77 + ``` 78 + 79 + The merge happens at `src/loader.ts:158-163`: 80 + 81 + ```typescript 82 + r.config = _merger( 83 + configs.overrides, 84 + configs.main, 85 + configs.rc, 86 + configs.packageJson, 87 + configs.defaultConfig, 88 + ) as T; 89 + ``` 90 + 91 + ## Where `defu` is Used 92 + 93 + `defu` (or a custom merger) is used at several points in the pipeline: 94 + 95 + ### 1. Main Merger Setup 96 + **Location**: `src/loader.ts:71` 97 + ```typescript 98 + const _merger = options.merger || defu; 99 + ``` 100 + 101 + ### 2. RC File Merging 102 + **Location**: `src/loader.ts:128` 103 + RC files from cwd, workspace, and home are merged: 104 + ```typescript 105 + rawConfigs.rc = _merger({} as T, ...rcSources); 106 + ``` 107 + 108 + ### 3. package.json Value Merging 109 + **Location**: `src/loader.ts:140` 110 + Multiple keys from package.json are merged: 111 + ```typescript 112 + rawConfigs.packageJson = _merger({} as T, ...values); 113 + ``` 114 + 115 + ### 4. Main Source Merging 116 + **Location**: `src/loader.ts:158-163` 117 + All primary config sources are merged: 118 + ```typescript 119 + r.config = _merger( 120 + configs.overrides, 121 + configs.main, 122 + configs.rc, 123 + configs.packageJson, 124 + configs.defaultConfig, 125 + ) as T; 126 + ``` 127 + 128 + ### 5. Extended Layers Merging 129 + **Location**: `src/loader.ts:171` 130 + After processing `extends`, all layers are merged into the main config: 131 + ```typescript 132 + r.config = _merger(r.config, ...r.layers!.map((e) => e.config)) as T; 133 + ``` 134 + 135 + ### 6. Defaults Application 136 + **Location**: `src/loader.ts:194` 137 + Default config has the lowest priority: 138 + ```typescript 139 + r.config = _merger(r.config, options.defaults) as T; 140 + ``` 141 + 142 + ### 7. Environment-Specific Config Merging 143 + **Location**: `src/loader.ts:418` (in `resolveConfig`) 144 + Env-specific config overrides the base config: 145 + ```typescript 146 + res.config = _merger(envConfig, res.config); 147 + ``` 148 + 149 + ### 8. Meta Merging 150 + **Location**: `src/loader.ts:423` (in `resolveConfig`) 151 + Meta from source options and config are merged: 152 + ```typescript 153 + res.meta = defu(res.sourceOptions!.meta, res.config!.$meta) as MT; 154 + ``` 155 + 156 + ### 9. Source Overrides Merging 157 + **Location**: `src/loader.ts:428` (in `resolveConfig`) 158 + Per-source overrides are applied: 159 + ```typescript 160 + res.config = _merger(res.sourceOptions!.overrides, res.config) as T; 161 + ``` 162 + 163 + ## resolveConfig: Loading Individual Config Layers 164 + 165 + The `resolveConfig` function handles loading individual configuration files (including extended configs): 166 + 167 + ```mermaid 168 + flowchart TD 169 + Start[resolveConfig start] --> CustomResolver{custom resolver?} 170 + CustomResolver -->|yes| TryCustom[Try custom resolver] 171 + TryCustom --> HasResult{result?} 172 + HasResult -->|yes| ReturnCustom[Return custom result] 173 + HasResult -->|no| GigetCheck 174 + CustomResolver -->|no| GigetCheck{giget URI?} 175 + GigetCheck -->|yes| Download[Download with giget<br/>to local path] 176 + Download --> NpmCheck 177 + GigetCheck -->|no| NpmCheck{npm package?} 178 + NpmCheck -->|yes| ResolvePkg[Resolve npm package] 179 + ResolvePkg --> LocalImport 180 + NpmCheck -->|no| LocalImport[Import from local FS] 181 + LocalImport --> GetExt{has extension?} 182 + GetExt -->|no| UseDir[Treat as directory<br/>use configFile name] 183 + GetExt -->|yes| TryResolve 184 + UseDir --> TryResolve[Try resolve with<br/>multiple paths] 185 + TryResolve --> FileExists{file exists?} 186 + FileExists -->|no| ReturnEmpty[Return empty config] 187 + FileExists -->|yes| CheckFormat 188 + CheckFormat{Async loader?} 189 + CheckFormat -->|yes| ParseAsync[Parse with<br/>confbox parsers] 190 + CheckFormat -->|no| ImportModule[Import module<br/>with jiti fallback] 191 + ImportModule --> IsFunction{is function?} 192 + ParseAsync --> IsFunction 193 + IsFunction -->|yes| CallFunction[Call with context] 194 + IsFunction -->|no| EnvCheck 195 + CallFunction --> EnvCheck 196 + EnvCheck{envName set?} 197 + EnvCheck -->|yes| MergeEnv[Merge env-specific<br/>config with defu] 198 + EnvCheck -->|no| MergeMeta 199 + MergeEnv --> MergeMeta[Merge meta with defu] 200 + MergeMeta --> SourceOverrides{source overrides?} 201 + SourceOverrides -->|yes| ApplyOverrides[Apply with defu] 202 + SourceOverrides -->|no| NormalizePaths 203 + ApplyOverrides --> NormalizePaths[Normalize paths] 204 + NormalizePaths --> ReturnResult[Return resolved config] 205 + ``` 206 + 207 + ## Environment-Specific Configuration 208 + 209 + Each config layer can define environment-specific overrides: 210 + 211 + ```mermaid 212 + flowchart LR 213 + Config[Config object] --> HasEnv{Has envName?} 214 + HasEnv -->|no| Skip[Skip env merging] 215 + HasEnv -->|yes| CheckKeys{Has $<envName><br/>or $env.<envName>?} 216 + CheckKeys -->|yes| ExtractEnv[Extract env config] 217 + ExtractEnv --> MergeEnv[Merge env config<br/>over base with defu] 218 + CheckKeys -->|no| Skip 219 + MergeEnv --> Skip 220 + Skip --> Next[Continue pipeline] 221 + ``` 222 + 223 + The lookup order for env-specific config is (per `src/loader.ts:413-415`): 224 + 1. `config.$<envName>` (e.g., `$production`) 225 + 2. `config.$env.<envName>` (e.g., `$env.staging`) 226 + 227 + ## RC File Loading 228 + 229 + RC files are loaded from multiple locations (if `globalRc` is enabled): 230 + 231 + ```mermaid 232 + flowchart TD 233 + Start[Load RC files] --> Cwd[Load from cwd] 234 + Cwd --> Workspace{globalRc enabled?} 235 + Workspace -->|yes| FindWorkspace[Find workspace dir] 236 + FindWorkspace --> LoadWorkspace[Load from workspace] 237 + Workspace -->|no| Home 238 + LoadWorkspace --> Home{globalRc enabled?} 239 + Home -->|yes| LoadHome[Load from user home<br/>via rc9.readUser] 240 + Home -->|no| Merge 241 + LoadHome --> Merge[Merge all RC sources<br/>with defu] 242 + Merge --> End[Return merged RC config] 243 + ``` 244 + 245 + RC file loading uses the `rc9` package, which reads from: 246 + 1. `cwd/.<name>rc` 247 + 2. Workspace root `.<name>rc` (if `globalRc`) 248 + 3. User home directory `.<name>rc` (if `globalRc`) 249 + 250 + ## Extended Configuration Processing 251 + 252 + The `extends` feature allows configs to inherit from other configs: 253 + 254 + ```mermaid 255 + flowchart TD 256 + Start[extendConfig] --> FindExtends{Has extends key?} 257 + FindExtends -->|no| End[Return] 258 + FindExtends -->|yes| ExtractSources[Extract extend sources] 259 + ExtractSources --> LoopSources[For each source] 260 + LoopSources --> CheckFormat{Format?} 261 + CheckFormats -->|{source, options}| Extract2[Extract source/options] 262 + CheckFormats -->|[source, options]| Extract2 263 + CheckFormats -->|string| ResolveSource 264 + Extract2 --> ResolveSource 265 + ResolveSource --> RemoteCheck{Remote URI?} 266 + RemoteCheck -->|yes| Download[Download with giget] 267 + RemoteCheck -->|no| NpmCheck 268 + Download --> NpmCheck{npm package?} 269 + NpmCheck -->|yes| ResolvePkg[Resolve package] 270 + NpmCheck -->|no| LocalPath 271 + ResolvePkg --> LocalPath[Use local path] 272 + LocalPath --> ResolveConfig2[Call resolveConfig] 273 + ResolveConfig2 --> RecursiveExtend[Recursive extendConfig<br/>on base] 274 + RecursiveExtend --> PushLayer[Push to _layers array] 275 + PushLayer --> NextSource{More sources?} 276 + NextSource -->|yes| LoopSources 277 + NextSource -->|no| MergeLayers[Merge layers<br/>with defu] 278 + MergeLayers --> End 279 + ``` 280 + 281 + ## dotenv Integration 282 + 283 + Environment variables are loaded before any config files (per `src/loader.ts:94-100`): 284 + 285 + ```mermaid 286 + flowchart TD 287 + Start[setupDotenv] --> LoadFiles[Load .env files] 288 + LoadFiles --> ParseFiles[Parse with<br/>node:util.parseEnv] 289 + ParseFiles --> FileRefs{expandFileReferences?} 290 + FileRefs -->|yes| ExpandFiles[Read _FILE vars<br/>from disk] 291 + FileRefs -->|no| Interpolate 292 + ExpandFiles --> Interpolate{interpolate?} 293 + Interpolate -->|yes| ExpandVars[Expand ${VAR}<br/>references] 294 + Interpolate -->|no| ApplyToEnv 295 + ExpandVars --> ApplyToEnv[Apply to process.env] 296 + ApplyToEnv --> End[Return] 297 + ``` 298 + 299 + The dotenv loading happens **before** any config files, allowing config files to reference environment variables. 300 + 301 + ## Complete Data Flow 302 + 303 + ```mermaid 304 + sequenceDiagram 305 + participant User 306 + participant LoadConfig as loadConfig() 307 + participant Dotenv as setupDotenv() 308 + participant Resolve as resolveConfig() 309 + participant Defu as defu (merger) 310 + participant RC9 as rc9 311 + participant PkgTypes as pkg-types 312 + 313 + User->>LoadConfig: Call with options 314 + LoadConfig->>LoadConfig: Normalize options 315 + LoadConfig->>LoadConfig: Setup merger (defu or custom) 316 + 317 + opt dotenv enabled 318 + LoadConfig->>Dotenv: setupDotenv(options) 319 + Dotenv-->>LoadConfig: process.env populated 320 + end 321 + 322 + LoadConfig->>Resolve: resolveConfig(".", options) 323 + Resolve-->>LoadConfig: Main config object 324 + 325 + opt rcFile enabled 326 + LoadConfig->>RC9: rc9.read({ cwd }) 327 + opt globalRc enabled 328 + LoadConfig->>PkgTypes: findWorkspaceDir() 329 + PkgTypes-->>LoadConfig: workspace path 330 + LoadConfig->>RC9: rc9.read({ workspace }) 331 + LoadConfig->>RC9: rc9.readUser() 332 + end 333 + LoadConfig->>Defu: Merge all RC sources 334 + Defu-->>LoadConfig: Merged RC config 335 + end 336 + 337 + opt packageJson enabled 338 + LoadConfig->>PkgTypes: readPackageJSON() 339 + PkgTypes-->>LoadConfig: package.json object 340 + LoadConfig->>Defu: Merge package.json values 341 + Defu-->>LoadConfig: Merged pkg config 342 + end 343 + 344 + LoadConfig->>LoadConfig: Resolve config functions 345 + 346 + LoadConfig->>Defu: Merge all sources 347 + Note over Defu: overrides → main → rc →<br/>packageJson → defaultConfig 348 + Defu-->>LoadConfig: Merged config 349 + 350 + opt extends enabled 351 + loop Each extend source 352 + LoadConfig->>Resolve: resolveConfig(source) 353 + Resolve-->>LoadConfig: Extended layer 354 + end 355 + LoadConfig->>Defu: Merge layers into config 356 + Defu-->>LoadConfig: Extended config 357 + end 358 + 359 + opt defaults provided 360 + LoadConfig->>Defu: Merge defaults 361 + Defu-->>LoadConfig: Final config 362 + end 363 + 364 + opt omit$Keys enabled 365 + LoadConfig->>LoadConfig: Remove $ prefixed keys 366 + end 367 + 368 + LoadConfig-->>User: Resolved config + layers 369 + ``` 370 + 371 + ## Key Files 372 + 373 + | File | Purpose | 374 + |------|---------| 375 + | `src/loader.ts` | Main `loadConfig()` function and `resolveConfig()` | 376 + | `src/dotenv.ts` | Environment variable loading (`setupDotenv`, `loadDotenv`) | 377 + | `src/types.ts` | TypeScript type definitions | 378 + | `src/watch.ts` | Config watching with file system events | 379 + 380 + ## Summary 381 + 382 + The configuration pipeline in c12 is: 383 + 384 + 1. **Normalize options** - Set defaults and normalize paths 385 + 2. **Load environment variables** - Parse `.env` files and populate `process.env` 386 + 3. **Load main config** - Find and import the primary config file 387 + 4. **Load RC files** - Read from cwd, workspace, and home directories 388 + 5. **Load package.json** - Extract config values from package.json 389 + 6. **Merge all sources** - Use `defu` to merge in priority order 390 + 7. **Process extends** - Recursively load and merge extended configs 391 + 8. **Apply defaults** - Merge lowest-priority defaults 392 + 9. **Cleanup** - Remove internal `$` keys if requested 393 + 394 + `defu` is the core merging function used throughout the pipeline to ensure deep, predictable merging of configuration objects from all sources.
+538
doc/discovery/dynamic-configs.md
··· 1 + # Dynamic Configuration Sources in c12 2 + 3 + This document explores how c12 could support more dynamic configuration sources, such as drop-in config directories (systemd-style `.d` directories), and how configuration layering could be made more first-class. 4 + 5 + ## Background 6 + 7 + Currently, c12 has a **fixed set of configuration sources** defined in `ConfigSource`: 8 + 9 + ```typescript 10 + export type ConfigSource = "overrides" | "main" | "rc" | "packageJson" | "defaultConfig"; 11 + ``` 12 + 13 + These sources are loaded in a **hardcoded order** within `loadConfig()` at `src/loader.ts:83-92`: 14 + 15 + ```typescript 16 + const rawConfigs: Record< 17 + ConfigSource, 18 + ResolvableConfig<T> | null | undefined 19 + > = { 20 + overrides: options.overrides, 21 + main: undefined, 22 + rc: undefined, 23 + packageJson: undefined, 24 + defaultConfig: options.defaultConfig, 25 + }; 26 + ``` 27 + 28 + This works well for the common case, but lacks flexibility for dynamic configuration discovery. 29 + 30 + ## The Use Case: Drop-in Config Directories 31 + 32 + Inspired by systemd's drop-in configuration pattern, this feature would allow: 33 + 34 + 1. A main config file: `myapp.config.ts` 35 + 2. A drop-in directory: `myapp.config.d/` 36 + 3. Individual override files in the directory: 37 + - `myapp.config.d/10-admin-overrides.ts` 38 + - `myapp.config.d/20-production.ts` 39 + - `myapp.config.d/99-local.ts` 40 + 41 + Files in the `.d` directory are merged in **lexicographic order**, with later files overriding earlier ones. This allows: 42 + - System administrators to layer configurations without modifying the base config 43 + - Easy enable/disable by adding/removing files 44 + - Clear provenance of where config values came from 45 + 46 + ## Current Limitations 47 + 48 + ### Fixed Source Types 49 + 50 + The `ConfigSource` type is a union literal, which means: 51 + - New sources require type changes 52 + - Cannot dynamically add sources at runtime 53 + - Source ordering is fixed 54 + 55 + ### Limited Source Metadata 56 + 57 + While `ConfigLayer` exists and has `meta` field, it's used differently than a comprehensive provenance system: 58 + 59 + ```typescript 60 + export interface ConfigLayer< 61 + T extends UserInputConfig = UserInputConfig, 62 + MT extends ConfigLayerMeta = ConfigLayerMeta, 63 + > { 64 + config: T | null; 65 + source?: string; 66 + sourceOptions?: SourceOptions<T, MT>; 67 + meta?: MT; 68 + cwd?: string; 69 + configFile?: string; 70 + } 71 + ``` 72 + 73 + The `meta` field is primarily for user-defined metadata, not automatic tracking of: 74 + - Which provider provided the value 75 + - Where in the merge order the value came from 76 + - Priority/ranking of the source 77 + 78 + ## Inspiration: Rust's Figment Crate 79 + 80 + Rust's [figment](https://docs.rs/figment/latest/figment/) takes a more flexible approach: 81 + 82 + ### Provider Trait 83 + 84 + Any type can implement the `Provider` trait to become a configuration source: 85 + 86 + ```rust 87 + trait Provider { 88 + fn metadata(&self) -> Metadata; 89 + fn data(&self) -> Result<Map<Profile, Dict>, Error>; 90 + fn profile(&self) -> Option<Profile>; 91 + } 92 + ``` 93 + 94 + ### Metadata Tracking 95 + 96 + Every value is tagged with `Metadata`: 97 + 98 + ```rust 99 + pub struct Metadata { 100 + pub name: Cow<'static, str>, // "TOML File" 101 + pub source: Option<Source>, // Path, URL, etc. 102 + pub provide_location: Option<&'static Location<'static>>, 103 + } 104 + ``` 105 + 106 + This allows: 107 + - Rich error messages showing exactly where a value came from 108 + - "Magic" values like `RelativePathBuf` that know their config file location 109 + - Debugging complex configurations 110 + 111 + ### Third-Party Providers 112 + 113 + The ecosystem can provide custom providers: 114 + - `figment-directory` - Config from directories 115 + - `figment-file-provider-adapter` - Reads `_FILE` suffix variables 116 + - Custom providers for any data source 117 + 118 + ## Potential Design for c12 119 + 120 + ### Option 1: Provider Pattern 121 + 122 + Introduce a `ConfigProvider` interface: 123 + 124 + ```typescript 125 + export interface ConfigProvider<T = UserInputConfig> { 126 + /** Unique identifier for this provider */ 127 + name: string; 128 + 129 + /** Metadata about this provider */ 130 + metadata: ConfigProviderMetadata; 131 + 132 + /** Load configuration from this provider */ 133 + load(context: ConfigProviderContext): Promise<T | null | undefined>; 134 + 135 + /** Priority (lower = higher priority) */ 136 + priority?: number; 137 + 138 + /** Should this provider be enabled? */ 139 + enabled?(options: LoadConfigOptions): boolean; 140 + } 141 + 142 + export interface ConfigProviderMetadata { 143 + name: string; 144 + source?: string; 145 + description?: string; 146 + } 147 + 148 + export interface ConfigProviderContext { 149 + cwd: string; 150 + envName: string | false; 151 + [key: string]: any; 152 + } 153 + ``` 154 + 155 + #### Drop-in Directory Provider Example 156 + 157 + ```typescript 158 + class DropInDirProvider<T extends UserInputConfig> implements ConfigProvider<T> { 159 + name = "drop-in-directory"; 160 + 161 + constructor( 162 + private basePath: string, 163 + private configName: string, 164 + private pattern: string = "*.config.{ts,js,json,yaml,yml}", 165 + ) {} 166 + 167 + metadata = { 168 + name: "Drop-in Directory", 169 + source: this.basePath, 170 + }; 171 + 172 + priority = 50; // Between RC and package.json 173 + 174 + async load(context: ConfigProviderContext): Promise<T | null> { 175 + const dropInDir = path.join(context.cwd, `${this.configName}.d`); 176 + 177 + if (!fs.existsSync(dropInDir)) { 178 + return null; 179 + } 180 + 181 + const files = await glob(this.pattern, { cwd: dropInDir }); 182 + const sorted = files.sort(); // Lexicographic order 183 + 184 + let merged: Partial<T> = {}; 185 + for (const file of sorted) { 186 + const config = await loadConfigFile(path.join(dropInDir, file)); 187 + merged = defu(merged, config); 188 + } 189 + 190 + return merged as T; 191 + } 192 + } 193 + ``` 194 + 195 + ### Option 2: Source Registry 196 + 197 + Add a source registration system to `LoadConfigOptions`: 198 + 199 + ```typescript 200 + export interface LoadConfigOptions<T, MT> { 201 + // Existing options... 202 + 203 + /** 204 + * Register additional config sources 205 + * Sources are loaded in order of priority (lowest first) 206 + */ 207 + sources?: ConfigSourceEntry<T, MT>[]; 208 + } 209 + 210 + export interface ConfigSourceEntry<T, MT> { 211 + /** Unique identifier */ 212 + id: string; 213 + 214 + /** Provider function returning config */ 215 + provider: ResolvableConfig<T>; 216 + 217 + /** Priority (lower = higher priority) */ 218 + priority: number; 219 + 220 + /** Whether this source should be loaded */ 221 + condition?: (options: LoadConfigOptions<T, MT>) => boolean; 222 + 223 + /** Metadata about this source */ 224 + metadata?: Partial<ConfigLayerMeta>; 225 + } 226 + ``` 227 + 228 + #### Usage Example 229 + 230 + ```typescript 231 + const config = await loadConfig({ 232 + name: "myapp", 233 + 234 + sources: [ 235 + { 236 + id: "overrides", 237 + priority: 10, 238 + provider: { custom: "overrides" }, 239 + }, 240 + { 241 + id: "drop-in-dir", 242 + priority: 20, 243 + condition: (opts) => opts.envName === "production", 244 + provider: async (ctx) => { 245 + const dropInDir = path.join(opts.cwd, "myapp.config.d"); 246 + return await loadDropInConfigs(dropInDir); 247 + }, 248 + metadata: { name: "Drop-in Directory" }, 249 + }, 250 + { 251 + id: "main", 252 + priority: 30, 253 + provider: { custom: "main" }, // Built-in loader 254 + }, 255 + ], 256 + }); 257 + ``` 258 + 259 + ### Option 3: Enhanced Built-in Sources 260 + 261 + Add a new `dropIn` option to the existing `LoadConfigOptions`: 262 + 263 + ```typescript 264 + export interface LoadConfigOptions<T, MT> { 265 + // Existing options... 266 + 267 + /** 268 + * Load config from a .d directory 269 + * Files are merged in lexicographic order 270 + */ 271 + dropIn?: boolean | string | DropInOptions; 272 + } 273 + 274 + export interface DropInOptions { 275 + /** Directory name (default: <configFile>.d) */ 276 + dir?: string; 277 + 278 + /** File pattern to match */ 279 + pattern?: string; 280 + 281 + /** Where in priority order to insert (default: after main, before defaults) */ 282 + insertAfter?: "main" | "rc" | "packageJson"; 283 + 284 + /** Enable only for specific environments */ 285 + env?: string[]; 286 + } 287 + ``` 288 + 289 + #### Usage 290 + 291 + ```typescript 292 + // Simple: auto-detect myapp.config.d/ 293 + await loadConfig({ name: "myapp", dropIn: true }); 294 + 295 + // Custom directory 296 + await loadConfig({ 297 + name: "myapp", 298 + dropIn: { dir: "config.overrides.d" }, 299 + }); 300 + 301 + // Environment-specific 302 + await loadConfig({ 303 + name: "myapp", 304 + dropIn: { env: ["production", "staging"] }, 305 + }); 306 + ``` 307 + 308 + ## Layering Provenance 309 + 310 + To make layering "first class" as mentioned in issue #298, we need: 311 + 312 + ### Enhanced Layer Metadata 313 + 314 + ```typescript 315 + export interface ConfigLayer<T, MT> { 316 + config: T | null; 317 + source?: string; 318 + sourceOptions?: SourceOptions<T, MT>; 319 + meta?: ConfigLayerMeta; // User metadata 320 + cwd?: string; 321 + configFile?: string; 322 + 323 + // New fields for provenance: 324 + provider: string; // "main", "drop-in", "rc", etc. 325 + priority: number; // Merge order 326 + loadTime: Date; // When it was loaded 327 + fingerprint?: string; // Content hash for change detection 328 + } 329 + ``` 330 + 331 + ### Layer Tree Visualization 332 + 333 + A resolved config could show its full provenance: 334 + 335 + ```typescript 336 + { 337 + config: { /* merged config */ }, 338 + layers: [ 339 + { provider: "overrides", priority: 10, configFile: undefined, ... }, 340 + { provider: "drop-in", priority: 20, configFile: "./myapp.config.d/10-admin.ts", ... }, 341 + { provider: "drop-in", priority: 20, configFile: "./myapp.config.d/20-production.ts", ... }, 342 + { provider: "drop-in", priority: 20, configFile: "./myapp.config.d/99-local.ts", ... }, 343 + { provider: "main", priority: 30, configFile: "./myapp.config.ts", ... }, 344 + { provider: "rc", priority: 40, configFile: "/home/user/.myapprc", ... }, 345 + ] 346 + } 347 + ``` 348 + 349 + ### Value Attribution 350 + 351 + For debugging, we could trace where each config value came from: 352 + 353 + ```typescript 354 + function traceValue( 355 + config: ResolvedConfig, 356 + keyPath: string[] 357 + ): ConfigLayer | undefined { 358 + // Search layers in reverse priority order 359 + for (const layer of [...config.layers].reverse()) { 360 + let value = layer.config; 361 + for (const key of keyPath) { 362 + value = value?.[key]; 363 + } 364 + if (value !== undefined) { 365 + return layer; 366 + } 367 + } 368 + return undefined; 369 + } 370 + 371 + // Usage: 372 + const source = traceValue(config, ["server", "port"]); 373 + console.log(`server.port came from: ${source.configFile}`); 374 + ``` 375 + 376 + ## Implementation Considerations 377 + 378 + ### Backward Compatibility 379 + 380 + Any changes should maintain backward compatibility: 381 + 382 + 1. Default behavior unchanged - drop-ins disabled by default 383 + 2. Existing `ConfigSource` type could remain for internal use 384 + 3. New options as additions only 385 + 386 + ### Performance 387 + 388 + Loading many small files has overhead: 389 + 390 + 1. Consider caching resolved configs 391 + 2. Watch mode should efficiently track file additions/removals 392 + 3. Content fingerprinting for change detection 393 + 394 + ### Error Handling 395 + 396 + With more sources, errors are more likely: 397 + 398 + ```typescript 399 + // Per-layer error tracking 400 + export interface ConfigLayer<T, MT> { 401 + // ... 402 + error?: { 403 + message: string; 404 + source: string; 405 + recoverable: boolean; 406 + }; 407 + } 408 + 409 + // Strict vs relaxed modes 410 + await loadConfig({ 411 + strict: true, // Fail on any source error 412 + // or 413 + strict: false, // Log warnings, continue 414 + }); 415 + ``` 416 + 417 + ### File Naming Conventions 418 + 419 + For `.d` directories, establish conventions: 420 + 421 + ```typescript 422 + // Numeric prefix for ordering: 423 + // 00-base.conf 424 + // 10-admin.conf 425 + // 20-deployment.conf 426 + // 99-local.conf 427 + 428 + // Or use alphanumeric sorting: 429 + // admin.conf 430 + // base.conf 431 + // local.conf 432 + // production.conf 433 + ``` 434 + 435 + ## Related Enhancements 436 + 437 + ### Watch Mode Integration 438 + 439 + For drop-in directories, watch mode should: 440 + - Detect new files added 441 + - Detect files removed 442 + - Detect files renamed 443 + - Re-merge in correct order when changes occur 444 + 445 + ```typescript 446 + watchConfig({ 447 + name: "myapp", 448 + dropIn: true, 449 + onWatch: (event) => { 450 + console.log(`Drop-in file ${event.type}: ${event.path}`); 451 + }, 452 + }); 453 + ``` 454 + 455 + ### Configuration Validation 456 + 457 + With layered configs, validation should: 458 + 459 + 1. Validate each layer independently 460 + 2. Validate the final merged result 461 + 3. Show which layer introduced validation errors 462 + 463 + ```typescript 464 + const config = await loadConfig({ 465 + name: "myapp", 466 + validate: (layer, merged) => { 467 + // Check layer-specific constraints 468 + // Check final merged constraints 469 + }, 470 + }); 471 + ``` 472 + 473 + ## Existing Workarounds 474 + 475 + Before dynamic sources are implemented, you can: 476 + 477 + ### Use `extends` for Multiple Files 478 + 479 + ```typescript 480 + // main.config.ts 481 + export default { 482 + extends: [ 483 + "./configs/base.config", 484 + "./configs/admin.config", 485 + "./configs/production.config", 486 + ], 487 + // ... 488 + }; 489 + ``` 490 + 491 + ### Use Custom Resolver 492 + 493 + ```typescript 494 + const config = await loadConfig({ 495 + name: "myapp", 496 + async resolve(source, options) { 497 + if (source.startsWith("drop-in:")) { 498 + const dir = source.replace("drop-in:", ""); 499 + return await loadDropInConfigs(dir); 500 + } 501 + return null; // Use default resolution 502 + }, 503 + 504 + // Then use it via extends 505 + overrides: (ctx) => ({ 506 + extends: ["drop-in:./myapp.config.d"], 507 + }), 508 + }); 509 + ``` 510 + 511 + ### Merge Multiple Loads 512 + 513 + ```typescript 514 + const [main, admin, local] = await Promise.all([ 515 + loadConfig({ name: "myapp" }), 516 + loadConfig({ name: "admin-overrides" }), 517 + loadConfig({ name: "local-overrides" }), 518 + ]); 519 + 520 + const merged = defu(local.config, admin.config, main.config); 521 + ``` 522 + 523 + ## Summary 524 + 525 + Dynamic configuration sources in c12 would enable: 526 + 527 + 1. **Drop-in config directories** - Systemd-style `.d` directories 528 + 2. **Custom providers** - Third-party data sources 529 + 3. **Enhanced provenance** - Clear tracking of where values came from 530 + 4. **Flexible ordering** - Configurable source priorities 531 + 5. **Better debugging** - Layer-by-layer inspection 532 + 533 + The key design decision is between: 534 + - **Provider pattern** - Maximum flexibility, ecosystem growth 535 + - **Enhanced built-ins** - Simpler API, controlled feature set 536 + - **Source registry** - Middle ground with explicit registration 537 + 538 + All approaches should maintain backward compatibility while enabling the dynamic configuration discovery that makes layering a first-class concept.