Unified Agent + reusable Go agent core.
Feat: Add internal FSStore prerequisite documentation.
+336
+336
docs/feat/feat_20260206_internal_fsstore.md
+336
docs/feat/feat_20260206_internal_fsstore.md
···
1
+
---
2
+
date: 2026-02-06
3
+
title: Internal FSStore Prerequisite (MAEP + Contacts)
4
+
status: done
5
+
---
6
+
7
+
# Internal FSStore Prerequisites and Implementation Plan
8
+
9
+
## 1) Goal
10
+
11
+
This requirement does one thing only: provide a unified, reusable file persistence foundation for migrating the current `maep` implementation and for future reuse by contacts business storage.
12
+
13
+
Domain consumers:
14
+
- `maep` (already exists, migrate first).
15
+
- `guard` (already exists, migrate JSONL writer).
16
+
- `contacts` (future integration, reuse file capabilities).
17
+
18
+
Directory conventions (required):
19
+
- `maep` has its own `maep_dir` (already exists).
20
+
- `guard` has its own `guard_dir` (new requirement).
21
+
- `contacts` has its own `contacts_dir` (future implementation).
22
+
23
+
Hard objectives:
24
+
- Migrate low-level file I/O in `maep/file_store.go` to `internal/fsstore`.
25
+
- Migrate `guard/audit_jsonl.go` to `internal/fsstore.JSONLWriter` (no behavior change).
26
+
- Migrate `guard approvals` from SQLite to a file backend (based on `internal/fsstore`).
27
+
- Standardize audit persistence as JSONL (`maep` and `guard`).
28
+
- Provide atomic writes, locking, and JSONL append capabilities for future contacts usage.
29
+
- Provide minimal generic indexing (generic key-value index file) to satisfy basic "fast access by key" requirements.
30
+
31
+
Constraints:
32
+
- No database introduced.
33
+
- No new business abstraction layer introduced.
34
+
- Extract only file capabilities, not domain models.
35
+
36
+
## 2) Scope
37
+
38
+
### 2.1 in scope
39
+
- Directory creation and permission constraints.
40
+
- JSON read/write (read, atomic write).
41
+
- Text read/write (for `active.md` / `inactive.md`).
42
+
- JSONL append and rotation.
43
+
- Generic index file read-modify-write (key -> entry, no domain semantics).
44
+
- Cross-process locking (same host).
45
+
- Error semantics and basic testing utilities.
46
+
47
+
### 2.2 out of scope
48
+
- `maep.Store` domain interface definition.
49
+
- Contacts business schema (including `contacts/index.json` structure).
50
+
- Any business policy (scoring, eviction, trust rules, prompt decisions).
51
+
- Distributed locks.
52
+
53
+
## 3) Current Problems (Migration Motivation)
54
+
55
+
Sources of duplicated implementation:
56
+
- `maep/file_store.go`: JSON read/write, atomic write, directory permissions, in-process lock.
57
+
- `guard/audit_jsonl.go`: JSONL append, rotation, writer lifecycle.
58
+
59
+
Risks:
60
+
- Inconsistent permission and atomicity strategies.
61
+
- Inconsistent cross-process concurrency semantics (`sync.Mutex` is in-process only).
62
+
- Reimplementing for contacts would continue divergence.
63
+
64
+
## 4) Target Package and Suggested Layout
65
+
66
+
Frozen package name: `internal/fsstore`
67
+
68
+
Suggested file layout:
69
+
- `internal/fsstore/options.go`
70
+
- `internal/fsstore/atomic.go`
71
+
- `internal/fsstore/json.go`
72
+
- `internal/fsstore/text.go`
73
+
- `internal/fsstore/jsonl.go`
74
+
- `internal/fsstore/index.go`
75
+
- `internal/fsstore/lock.go`
76
+
- `internal/fsstore/errors.go`
77
+
78
+
Notes:
79
+
- Start with a functional API to avoid premature class hierarchy.
80
+
- Add object-oriented forms later (for example a `Store` struct) if needed, without blocking current migration.
81
+
82
+
## 5) API v1 (Frozen Draft)
83
+
84
+
```go
85
+
package fsstore
86
+
87
+
import (
88
+
"encoding/json"
89
+
"context"
90
+
"os"
91
+
"time"
92
+
)
93
+
94
+
type FileOptions struct {
95
+
DirPerm os.FileMode // default 0700
96
+
FilePerm os.FileMode // default 0600
97
+
}
98
+
99
+
type JSONLOptions struct {
100
+
DirPerm os.FileMode // default 0700
101
+
FilePerm os.FileMode // default 0600
102
+
RotateMaxBytes int64 // default 100 MiB
103
+
FlushEachWrite bool // default true
104
+
SyncEachWrite bool // default false
105
+
}
106
+
107
+
func EnsureDir(path string, perm os.FileMode) error
108
+
109
+
func ReadJSON(path string, out any) (exists bool, err error)
110
+
func WriteJSONAtomic(path string, v any, opts FileOptions) error
111
+
112
+
func ReadText(path string) (content string, exists bool, err error)
113
+
func WriteTextAtomic(path string, content string, opts FileOptions) error
114
+
115
+
type IndexEntry struct {
116
+
Ref string `json:"ref,omitempty"`
117
+
Rev uint64 `json:"rev,omitempty"`
118
+
Hash string `json:"hash,omitempty"`
119
+
UpdatedAt time.Time `json:"updated_at,omitempty"`
120
+
Meta json.RawMessage `json:"meta,omitempty"` // domain-defined metadata
121
+
}
122
+
123
+
type IndexFile struct {
124
+
Version int `json:"version"`
125
+
Entries map[string]IndexEntry `json:"entries"`
126
+
}
127
+
128
+
func ReadIndex(path string) (index IndexFile, exists bool, err error)
129
+
func WriteIndexAtomic(path string, index IndexFile, opts FileOptions) error
130
+
func MutateIndex(ctx context.Context, path string, lockPath string, opts FileOptions, fn func(*IndexFile) error) error
131
+
132
+
func BuildLockPath(lockRoot string, lockKey string) (string, error)
133
+
func WithLock(ctx context.Context, lockPath string, fn func() error) error
134
+
135
+
type JSONLWriter struct { /* unexported */ }
136
+
func NewJSONLWriter(path string, opts JSONLOptions) (*JSONLWriter, error)
137
+
func (w *JSONLWriter) AppendJSON(v any) error
138
+
func (w *JSONLWriter) AppendLine(line string) error // line must not contain '\n'
139
+
func (w *JSONLWriter) Close() error
140
+
```
141
+
142
+
API notes:
143
+
- `ReadJSON` / `ReadText`: when file does not exist, return `exists=false, err=nil`.
144
+
- `Write*Atomic`: guarantees the target path is either old version or new version, never partially written.
145
+
- `IndexFile` is a generic index container with no domain rules; domain fields go into `Meta`.
146
+
- `MutateIndex` provides integrated read-modify-write + lock to avoid repeated caller boilerplate.
147
+
- `BuildLockPath` validates lock key and maps it to a hidden lock directory path.
148
+
- `WithLock`: same-host cross-process mutual exclusion; lock scope is determined by `lockPath`.
149
+
150
+
## 6) Semantic Details (Must Follow)
151
+
152
+
### 6.1 Permissions
153
+
- Default directory permission `0700`, default file permission `0600`.
154
+
- For all writes, call `EnsureDir(filepath.Dir(path))` before writing.
155
+
- After atomic write completes, target file permissions must be `FilePerm`.
156
+
157
+
### 6.2 Atomic write flow
158
+
Consistent for `WriteJSONAtomic` / `WriteTextAtomic`:
159
+
1. Create a temp file in the target directory (same directory).
160
+
2. Write full content.
161
+
3. `fsync` the temp file.
162
+
4. Set temp file permission.
163
+
5. `rename(temp, target)` as atomic replacement.
164
+
6. Best-effort `fsync` directory (warn only on failure, no rollback).
165
+
7. Clean up temp file on failure paths.
166
+
167
+
### 6.3 JSON encoding
168
+
- Use UTF-8.
169
+
- Default to `MarshalIndent` (2 spaces) and append trailing `\n` to reduce manual review noise.
170
+
- No schema validation in `fsstore`; callers are responsible.
171
+
172
+
### 6.4 JSONL spec
173
+
- One JSON object per line, with trailing `\n`.
174
+
- Before append, if `current_size + incoming_bytes > RotateMaxBytes`, rotate first then write.
175
+
- Rotation filename: `<path>.YYYYMMDDTHHMMSSZ` (UTC).
176
+
- If rotation target already exists (same-second collision), auto-append suffix `.<n>` (starting at `1`) until success.
177
+
- With `FlushEachWrite=true`, flush after each append.
178
+
- With `SyncEachWrite=true`, `fsync` after each append (high-reliability audit scenarios only).
179
+
180
+
### 6.5 Lock strategy (decided)
181
+
- Domain-level primary lock:
182
+
- MAEP: key=`state.main`
183
+
- Contacts: key=`state.main`
184
+
- Audit-stream lock: one lock key per audit file (for example `audit.audit_events_jsonl`).
185
+
186
+
Lock key naming pattern (unified):
187
+
- Two-part format: `<scope>.<resource>`
188
+
- Common `scope`: `state`, `audit`, `index`
189
+
- Lock key represents a "resource mutex unit", not separated by business action.
190
+
- Actions like `append/rotate` must reuse the same audit-stream lock key to avoid races from dual locks.
191
+
192
+
Lock path rules (avoid exposing business names directly in filenames):
193
+
- First determine `lockRoot`:
194
+
- MAEP: `<maep_dir>/.fslocks`
195
+
- Guard: `<guard_dir>/.fslocks`
196
+
- Contacts: `<contacts_dir>/.fslocks`
197
+
- Lock key spec:
198
+
- Allowed charset only: `[a-z0-9._-]`
199
+
- Lowercase only
200
+
- No leading `.` and no trailing `.`
201
+
- Max length `<= 120`
202
+
- Final lock file path: `<lockRoot>/<lockKey>.lck`
203
+
- Lock key rules are frozen in this version. Any future rule change requires downtime migration and a guarantee that only one rule set runs at a time.
204
+
205
+
`.lck` file content convention:
206
+
- Correctness relies on OS file lock (`flock/fcntl`) + open file descriptor ownership, not file content.
207
+
- Content is diagnostic only. Recommended one-line JSON:
208
+
- `{"lock_key":"...","pid":1234,"hostname":"...","acquired_at":"...","owner":"<random-id>"}`
209
+
- File content is for observability/troubleshooting only, never for stale-lock decisions.
210
+
211
+
Examples:
212
+
- `lockKey = "state.main"` -> `<maep_dir>/.fslocks/state.main.lck`
213
+
- `lockKey = "audit.audit_events_jsonl"` -> `<maep_dir>/.fslocks/audit.audit_events_jsonl.lck`
214
+
- `lockKey = "audit.guard_audit_jsonl"` -> `<guard_dir>/.fslocks/audit.guard_audit_jsonl.lck`
215
+
216
+
Execution constraints:
217
+
- State mutations must run under the corresponding domain-level lock.
218
+
- Normal reads are lock-free by default. For cross-file strongly consistent reads, readers must use the same resource lock key as writers (for example `state.main`).
219
+
- Audit can be appended within the domain lock or independently. If one operation writes both state and audit, fixed order is "state first, audit second".
220
+
- Nested acquisition of the same lock path is forbidden to avoid self-deadlock.
221
+
222
+
Platform implementation constraints:
223
+
- Unix-like (Linux/macOS): implement `WithLock` using `flock/fcntl`.
224
+
- Windows (degraded implementation): lock-file occupancy via `os.O_CREATE|os.O_EXCL` + polling retry.
225
+
- `WithLock` wait duration is controlled only by `ctx`; no additional stale-lock auto-recovery.
226
+
- Under Windows degraded mode, lock files left by abnormal exits must be cleaned up manually (`.lck` file).
227
+
228
+
### 6.6 Error semantics
229
+
Suggested error categories (sentinel + wrap):
230
+
- `ErrInvalidPath`
231
+
- `ErrLockTimeout` (when `ctx` times out/cancels)
232
+
- `ErrLockUnavailable`
233
+
- `ErrEncodeFailed`
234
+
- `ErrDecodeFailed`
235
+
- `ErrAtomicWriteFailed`
236
+
237
+
Notes:
238
+
- Business layer can route retry/alert strategy via `errors.Is`.
239
+
240
+
## 7) Migration Targets and Steps
241
+
242
+
### 7.1 Phase A: deliver fsstore core capabilities (completed)
243
+
- Implement API v1.
244
+
- Complete unit tests and concurrency tests.
245
+
246
+
### 7.2 Phase B: migrate guard JSONL (completed)
247
+
- Add guard-specific directory convention:
248
+
- Config: `guard.dir_name` (default `guard`).
249
+
- Resolution: `guard_dir = <file_state_dir>/<guard.dir_name>`.
250
+
- Default audit path: `<guard_dir>/audit/guard_audit.jsonl`.
251
+
- Default approvals file path: `<guard_dir>/approvals/guard_approvals.json`.
252
+
- Refactor `guard/audit_jsonl.go` to reuse `fsstore.JSONLWriter`.
253
+
- Unify guard lock and path strategy through `BuildLockPath + WithLock`.
254
+
- Migrate guard approvals from SQLite to file backend (JSON + atomic write + lock).
255
+
- Freeze guard approvals file format:
256
+
- Top-level: `{ "version": 1, "records": { "<approval_id>": { ...ApprovalRecord... } } }`
257
+
- `records` uses `approval_id -> record` map for O(1) `Get/Resolve`.
258
+
- No automatic migration; historical SQLite data is migrated manually.
259
+
- `guard.approvals.sqlite_dsn` config key removed (no longer read).
260
+
- No separate approvals audit stream in this phase (no `guard_approvals_audit.jsonl`).
261
+
262
+
### 7.3 Phase C: migrate maep file store (completed)
263
+
- Migrate `readJSONFile`, `writeJSONFileAtomic`, and directory creation logic from `maep/file_store.go` to `fsstore`.
264
+
- Keep in-process `sync.Mutex` for object-level concurrency; add cross-process lock at file level.
265
+
266
+
### 7.4 Phase D: upgrade maep audit file to JSONL (completed)
267
+
- Migrate from `audit_events.json` to `audit_events.jsonl`.
268
+
- Compatibility strategy:
269
+
1. Prefer reading `.jsonl`.
270
+
2. If `.jsonl` does not exist and `.json` exists, import once and generate `.jsonl`.
271
+
3. After successful import, rename old `.json` to `audit_events.json.migrated.<ts>` (do not delete directly).
272
+
273
+
### 7.5 Phase E: contacts integration prep (pending)
274
+
- Contacts business layer uses `WriteTextAtomic` for `active.md` / `inactive.md`.
275
+
- Contacts audit uses `JSONLWriter` directly.
276
+
- Whether `contacts/index.json` exists and its structure are defined by contacts business docs, not frozen in this phase.
277
+
278
+
## 8) Test Matrix (Must Cover)
279
+
280
+
### 8.1 Atomic write
281
+
- After interrupted writes, target file remains readable (old or new version).
282
+
- High-frequency overwrite does not produce corrupted JSON.
283
+
284
+
### 8.2 Lock
285
+
- Two processes concurrently writing the same target are serialized.
286
+
- `ctx` cancellation exits lock wait in time.
287
+
288
+
### 8.3 JSONL
289
+
- Appended lines are complete, no partial line.
290
+
- Appending continues after rotation.
291
+
- Concurrent append has no interleaved corruption.
292
+
293
+
### 8.4 MAEP regression
294
+
- No behavior change in contact/inbox/dedupe/protocol_history.
295
+
- `maep audit list` output and filtering remain compatible (only storage medium changes).
296
+
297
+
## 9) TODO (Implementation Checklist)
298
+
299
+
### 9.1 fsstore package
300
+
- [x] Create `internal/fsstore` package layout.
301
+
- [x] Implement `EnsureDir`.
302
+
- [x] Implement `ReadJSON` / `WriteJSONAtomic`.
303
+
- [x] Implement `ReadText` / `WriteTextAtomic`.
304
+
- [x] Implement `ReadIndex` / `WriteIndexAtomic` / `MutateIndex`.
305
+
- [x] Implement `BuildLockPath` (validate lock key -> hidden lock file path).
306
+
- [x] Implement `WithLock(context, lockPath, fn)`.
307
+
- [x] Implement `JSONLWriter` and rotation rules.
308
+
- [x] Add sentinel errors and `errors.Is` semantics.
309
+
- [x] Add Windows degraded lock implementation and documentation.
310
+
311
+
### 9.2 migrate guard
312
+
- [x] Add `guard.dir_name` config (default `guard`).
313
+
- [x] Switch default paths to `guard_dir`-based (audit + approvals).
314
+
- [x] Refactor `guard/audit_jsonl.go` to reuse `internal/fsstore`.
315
+
- [x] Migrate approvals from SQLite to file backend (`guard_approvals.json`).
316
+
- [x] Remove `guard.approvals.sqlite_dsn` config key.
317
+
- [x] Do not add a separate approvals audit stream.
318
+
319
+
### 9.3 migrate maep
320
+
- [x] Replace low-level I/O helper functions in `maep/file_store.go`.
321
+
- [x] Add domain-level lock integration (`state.main` -> `BuildLockPath`).
322
+
- [x] Add `audit_events.json -> audit_events.jsonl` migration logic.
323
+
- [x] Regress `maep` related tests.
324
+
325
+
### 9.4 reserve for contacts
326
+
- [x] Verify `WriteTextAtomic` satisfies markdown main-file updates (when contacts integrates).
327
+
- [x] Verify contacts audit JSONL append capability (when contacts integrates).
328
+
329
+
## 10) Acceptance Criteria
330
+
331
+
This prerequisite is considered complete when all conditions below are met:
332
+
- `maep` and `guard` no longer maintain duplicate atomic-write/JSONL logic.
333
+
- `maep` and `guard` audits are both JSONL.
334
+
- `maep` file read/write has cross-process lock guarantees.
335
+
- Contacts can directly reuse `internal/fsstore` for file-based storage.
336
+
- `ReadIndex` / `WriteIndexAtomic` / `MutateIndex` are implemented with test coverage.