scratch: untrack stray files so .gitignore actually applies

-172

scratch/design-convey-error-wave0.md

··· 1 - # Design: convey-error-handling Wave 0 2 - 3 - This file is the lode-local decision log target for Wave 0. The final decision table will also be mirrored into the spec at commit time. 4 - 5 - ## Decision log 6 - 7 - | # | Decision | Chosen | Rationale | 8 - |---|----------|--------|-----------| 9 - | 1 | `api.js` housing | Add `convey/static/api.js`; insert it in `convey/templates/app.html` immediately after `websocket.js` and before the inline sidebar-state script. | That load point runs after `window.appEvents` exists and before any workspace/background consumers, while staying clear of `SurfaceState` and `AppServices` init ordering. | 10 - | 2 | `saveControl` housing | Keep `saveControl` in `convey/static/api.js` beside `ApiError` and `apiJson`. | The save helper depends on the fetch/error contract, so keeping them together avoids split ownership and duplicate imports. | 11 - | 3 | `appEvents.listen` overload | Keep the existing two-arg form and add `(tract, options, fn)` auto-detected by `typeof options === 'object' && typeof fn === 'function'`; the overloaded return stays the cleanup function and is augmented with `pending.track(corrId)` / `pending.clear(corrId)`. | Preserving the cleanup-function return keeps all existing listeners compatible. Attaching `pending` to the returned cleanup function avoids adding a second public API just for correlation tracking. | 12 - | 4 | `onParseError` routing | Add `appEvents.onParseError(fn)`; on websocket parse failure, keep the existing `console.warn`, call every registered parse-error handler, and call `window.logError(error, { context: 'websocket-parse' })` if defined. | This keeps current console visibility, adds explicit subscriber hooks, and routes parse failures into the existing owner-visible error log when available. It also means `error-handler.js` must expose `window.logError`. | 13 - | 5 | `replaceLoading` heuristic | Add `SurfaceState.replaceLoading(container, options)` with the exact heuristic from the scope: if `container.querySelector('.surface-state--loading')` is truthy, treat it as first paint and `replaceChildren(...)`; otherwise manage a singleton `.surface-state-refresh-error` sibling after the last non-error child. | The heuristic matches the audited failure split between first-paint emptiness and refresh-on-stale-content without forcing every caller to hand-roll placement rules. | 14 - | 6 | `registerTask` 403 behavior | On `ApiError.status === 403`, stop the interval, set `health.disabled = true`, do not tint the menu pip, do not notify, and retain the task record in `AppServices.getTaskHealth(appName)`. `registerTask` owns the no-auth-redirect behavior internally so consumers do not pass `noAuthRedirect`. | Support uses `403` as a real disabled state, not a broken-state signal. Centralizing that rule inside `registerTask` keeps background consumers simple and makes diagnostics consistent. | 15 - | 7 | `.menu-item-bg-failing` visual | Apply the class to the existing `<li class="menu-item">` root and render the failing indicator with a `::after` 8px amber pip using `var(--color-warning, #f59e0b)` and a `2px solid var(--facet-bg-primary, #fff)` border. No DOM mutation. | The menu already has a stable root for per-app state, and a pseudo-element avoids template churn while still reading on hover/focus/current states. | 16 - | 8 | Proof-of-adoption set | Adopt the proof set exactly as scoped: `convey/static/pairing.js`, starred-app toggle in `convey/static/app.js`, `apps/tokens/workspace.html`, and `apps/support/background.html`. Ship the `appEvents.listen` overload in Wave 0 without a real-site migration. | This gives one concrete adopter for each new primitive with low blast radius and good audit coverage. Deferring a live websocket-listener migration avoids binding timeout UX decisions into Wave 0 before later waves choose them per surface. | 17 - | 9 | Test form | Add self-contained static smoke pages under `convey/static/tests/` plus `convey/static/tests/README.md`; do not wire them into `tests/verify_browser.py` in this lode. | The repo currently has no `convey/static/tests/` harness. Static HTML pages are enough to prove the primitives without adding a pinchtab dependency to Wave 0. | 18 - | 10 | `ApiError` fields | `ApiError extends Error` with `status`, `statusText`, `serverMessage`, `url`, and optional `cause: 'parse'`; `message === serverMessage`; no `cause: 'network'`. | This keeps `instanceof ApiError` reliable, preserves the server text exactly, and leaves network failures on the existing global error path instead of inventing a second network-error contract. | 19 - | 11 | Error envelope robustness | `apiJson` extracts `payload?.error ?? payload?.message ?? 'Request failed (HTTP ${status})'` and documents that contract in JSDoc. | That catches the dominant server contract plus the existing todos `{"status":"error","message":"..."}` shape without requiring a server-side cleanup before Wave 1-3 adoption. | 20 - | 12 | Menu-item structural precondition | Do not add a new `position: relative` rule: the general `.menu-bar .menu-item` rule already has it in `convey/static/app.css`. | The pip already has a stable positioning anchor at the house-style rule level, so duplicating the property would be noise. | 21 - 22 - ## Primitive APIs (signatures only) 23 - 24 - ```js 25 - /** 26 - * @template T 27 - * @param {string} url 28 - * @param {RequestInit & { noAuthRedirect?: boolean }} [opts] 29 - * @returns {Promise<T>} 30 - * @throws {ApiError} 31 - */ 32 - window.apiJson = function apiJson(url, opts) {}; 33 - 34 - /** 35 - * @extends Error 36 - * @param {{ 37 - * status: number, 38 - * statusText: string, 39 - * serverMessage: string, 40 - * url: string, 41 - * cause?: 'parse' 42 - * }} init 43 - */ 44 - class ApiError extends Error {} 45 - 46 - /** 47 - * @template T 48 - * @param {{ 49 - * el: HTMLElement, 50 - * request: () => Promise<T>, 51 - * snapshot?: () => unknown, 52 - * revertOnError?: boolean | ((snapshot: unknown, error: Error) => void), 53 - * renderError?: (message: string, error: Error) => void, 54 - * clearError?: () => void, 55 - * onSuccess?: (result: T) => void 56 - * }} options 57 - * @returns {Promise<T>} 58 - */ 59 - window.saveControl = function saveControl(options) {}; 60 - 61 - /** 62 - * @param {{ 63 - * icon?: string, 64 - * heading?: string, 65 - * desc?: string, 66 - * action?: string | HTMLElement, 67 - * headingLevel?: string 68 - * }} [options] 69 - * @returns {HTMLElement} 70 - */ 71 - window.SurfaceState.errorCard = function errorCard(options) {}; 72 - 73 - /** 74 - * @param {Element} container 75 - * @param {{ 76 - * icon?: string, 77 - * heading?: string, 78 - * desc?: string, 79 - * action?: string | HTMLElement, 80 - * headingLevel?: string 81 - * }} [options] 82 - * @returns {HTMLElement} 83 - */ 84 - window.SurfaceState.replaceLoading = function replaceLoading(container, options) {}; 85 - 86 - /** 87 - * @param {string} tract 88 - * @param {(msg: any) => void} fn 89 - * @returns {() => void} 90 - */ 91 - window.appEvents.listen = function listen(tract, fn) {}; 92 - 93 - /** 94 - * @param {string} tract 95 - * @param {{ 96 - * schema?: (msg: any) => boolean, 97 - * timeout?: number, 98 - * onDrop?: (msg: any, error: Error) => void, 99 - * onTimeout?: (corrId: string) => void, 100 - * correlationKey?: string | ((msg: any) => string | null | undefined) 101 - * }} options 102 - * @param {(msg: any) => void} fn 103 - * @returns {(() => void) & { pending: { track(corrId: string): void, clear(corrId: string): void } }} 104 - */ 105 - window.appEvents.listen = function listen(tract, options, fn) {}; 106 - 107 - /** 108 - * @param {(error: Error, rawData: string) => void} fn 109 - * @returns {() => void} 110 - */ 111 - window.appEvents.onParseError = function onParseError(fn) {}; 112 - 113 - /** 114 - * @typedef {{ 115 - * disabled: boolean, 116 - * failing: boolean, 117 - * lastError: string | null, 118 - * lastRunAt: number | null, 119 - * lastSuccessAt: number | null, 120 - * consecutiveFailures: number 121 - * }} AppTaskHealth 122 - */ 123 - 124 - /** 125 - * @template T 126 - * @param {string} appName 127 - * @param {string} taskName 128 - * @param {{ 129 - * intervalMs: number, 130 - * run: (task: { apiJson: typeof window.apiJson }) => Promise<T>, 131 - * onSuccess?: (result: T) => void, 132 - * onError?: (error: Error) => void, 133 - * failuresBeforeFailing?: number 134 - * }} options 135 - * @returns {{ stop(): void, runNow(): Promise<void>, getHealth(): AppTaskHealth }} 136 - */ 137 - window.AppServices.registerTask = function registerTask(appName, taskName, options) {}; 138 - 139 - /** 140 - * @param {string} appName 141 - * @returns {Record<string, AppTaskHealth>} 142 - */ 143 - window.AppServices.getTaskHealth = function getTaskHealth(appName) {}; 144 - ``` 145 - 146 - ## Migration plan 147 - 148 - 1. `convey/static/error-handler.js` — expose `window.logError` while preserving the existing bottom-log behavior so websocket parse failures have an owner-visible sink. 149 - 2. `convey/templates/app.html` — insert `api.js` after `websocket.js`; no other load-order change. 150 - 3. `convey/static/api.js` — add `ApiError`, `apiJson`, and `saveControl`. 151 - 4. `convey/static/websocket.js` — add the `listen` overload, parse-error fanout, `onParseError`, and correlation timeout plumbing. 152 - 5. `convey/static/app.js` — add `SurfaceState.errorCard`, `SurfaceState.replaceLoading`, `AppServices.registerTask`, `AppServices.getTaskHealth`, starred-app toggle adoption, and menu failing-state class plumbing. 153 - 6. `convey/static/app.css` — add `.surface-state-refresh-error` and `.menu-item-bg-failing::after` styling. Do not add a new `.menu-item { position: relative; }` rule because it already exists. 154 - 7. `convey/static/pairing.js` — replace the local `fetchJson` implementation with a thin `apiJson(..., { noAuthRedirect: true })` wrapper and update both existing callers in that file. 155 - 8. `apps/tokens/workspace.html` — switch the load-failure catch block to `SurfaceState.errorCard` / `replaceLoading` and remove the Retry button per founder direction. 156 - 9. `apps/support/background.html` — migrate badge polling to `AppServices.registerTask` and delete the silent-swallow comment. 157 - 10. `convey/static/tests/README.md` plus `convey/static/tests/api.html`, `surface-state.html`, `ws-listen.html`, `register-task.html` — add static smoke coverage for the new primitives. 158 - 159 - No `convey/templates/menu_bar.html` change is needed; the existing `<li class="menu-item">` root is the anchor for the failing pip. 160 - 161 - ## Test plan 162 - 163 - - `convey/static/tests/api.html` — smoke `ApiError`, `apiJson`, error-envelope extraction, redirect suppression, and `saveControl` revert behavior with monkeypatched `window.fetch`. 164 - - `convey/static/tests/surface-state.html` — smoke `SurfaceState.errorCard` and `SurfaceState.replaceLoading` for both first-paint and refresh placement. 165 - - `convey/static/tests/ws-listen.html` — smoke `appEvents.listen` overload, pending tracking, timeout callbacks, parse-error fanout, and `onParseError`. 166 - - `convey/static/tests/register-task.html` — smoke `registerTask` lifecycle, success/failure transitions, 403 disable behavior, and `getTaskHealth`. 167 - 168 - ## Risks / open questions 169 - 170 - - `window.logError` is not currently public, so Wave 0 must expose it without regressing the existing `window.error` / `unhandledrejection` surfaces. 171 - - The websocket overload ships without a production adopter in Wave 0; the static smoke page is the only proof until later waves migrate a real listener. 172 - - Static smoke pages are manual in this wave and are not wired into CI or `verify_browser`, so enforcement still depends on reviewer discipline.

-788

scratch/design-convey-error-wave1.md

··· 1 - # Wave 1 design — convey error handling migration 2 - 3 - Scope: design only. No app/template implementation in this stage. Wave 0 shipped primitives are the ground truth. This wave migrates the 19 Tier-1 sites plus the importer listener contract. 4 - 5 - ## Files in scope 6 - 7 - - `apps/settings/workspace.html` 8 - - `apps/speakers/workspace.html` 9 - - `apps/home/workspace.html` 10 - - `apps/activities/_day.html` 11 - - `apps/import/workspace.html` 12 - - `apps/import/_detail.html` 13 - - `apps/observer/workspace.html` 14 - 15 - ## Implementation order 16 - 17 - 1. Add local CSS/helpers needed by multiple sites. 18 - 2. Migrate the 7 settings saves to `saveControl(...)`. 19 - 3. Migrate speakers to `apiJson(...)` + shared resolver. 20 - 4. Upgrade first-paint loaders and refresh errors in home / activities / observer / import. 21 - 5. Migrate importer WS listener to the options overload; add stalled state + load-more banner. 22 - 6. Split import detail error surfaces. 23 - 7. Audit with grep. 24 - 25 - ## D1. Settings `errorHost` 26 - 27 - **Decision** 28 - 29 - - Use `.settings-field small` as `saveControl`’s `errorHost`. 30 - - Add the amber override in `apps/settings/workspace.html`’s inline `<style>` block, not in `convey/static/app.css`. 31 - - Pre-clear the `<small>` before calling `saveControl(...)`. 32 - 33 - **Rationale** 34 - 35 - - The helper `<small>` is already where owners look. 36 - - `saveControl` only removes `[data-control-save-error]`; it does not clear arbitrary host text, so `Saved` / helper copy would otherwise coexist with the injected span. 37 - 38 - **Sketch** 39 - 40 - ```css 41 - .settings-field small .control-save-error { 42 - color: var(--color-warning, #f59e0b); 43 - margin-left: 0; 44 - font-size: inherit; 45 - display: inline; 46 - } 47 - ``` 48 - 49 - ```js 50 - function prepareFieldErrorHost(el) { 51 - const small = el.closest('.settings-field')?.querySelector('small'); 52 - if (!small) return null; 53 - if (!fieldHelperText.has(small)) fieldHelperText.set(small, small.textContent); 54 - const existingTimeout = fieldStatusTimeouts.get(small); 55 - if (existingTimeout) { 56 - clearTimeout(existingTimeout); 57 - fieldStatusTimeouts.delete(small); 58 - } 59 - small.textContent = ''; 60 - small.classList.remove('status-saved', 'status-error', 'status-fade'); 61 - return small; 62 - } 63 - ``` 64 - 65 - ## D2. Settings `onSuccess` 66 - 67 - **Decision** 68 - 69 - - Every settings `saveControl(...)` call includes `onSuccess: () => showFieldStatus(el, 'saved')` or the equivalent bound element. 70 - 71 - **Rationale** 72 - 73 - - Preserve the current success flash instead of making saves silent. 74 - 75 - **Sketch** 76 - 77 - ```js 78 - onSuccess: () => showFieldStatus(el, 'saved') 79 - ``` 80 - 81 - ## D3. Speakers `friendlyError` 82 - 83 - **Decision** 84 - 85 - - Keep `friendlyError(...)`. 86 - - Add local `resolveSpeakerError(err)` beside the three handlers. 87 - - If the mapping returns the generic fallback for an unknown server message, show the raw server message instead. 88 - - Keep the generic fallback only for the two intentional generic mappings: `Sentence embedding not found` and `No speaker labels found`. 89 - 90 - **Rationale** 91 - 92 - - Known failures stay owner-friendly. 93 - - Unknown failures remain legible instead of being flattened to the generic message. 94 - 95 - **Sketch** 96 - 97 - ```js 98 - function resolveSpeakerError(err) { 99 - const raw = err?.serverMessage || err?.message || ''; 100 - const friendly = friendlyError(raw); 101 - const GENERIC = 'something went wrong — try again'; 102 - const preserveGeneric = /Sentence embedding not found|No speaker labels found/i.test(raw); 103 - if (friendly === GENERIC && raw && !preserveGeneric) { 104 - return raw; 105 - } 106 - return friendly; 107 - } 108 - ``` 109 - 110 - Network/HTTP failures use the same resolver; `ApiError.serverMessage` is the canonical field. 111 - 112 - ## D4. Home refresh behavior 113 - 114 - **Decision** 115 - 116 - - `refreshSkills` and `refreshRoutines`: replace the content inside their container with `SurfaceState.errorCard(...)`. 117 - - `refreshBriefing`: use `SurfaceState.replaceLoading('pulse-briefing', ...)` so refresh failures render as a sibling `.surface-state-refresh-error` and the existing card stays visible. 118 - 119 - **Rationale** 120 - 121 - - Skills/routines are volatile sections; blanking them is acceptable. 122 - - Briefing is persistent owner-facing copy; preserve it in stale state. 123 - 124 - **Sketch** 125 - 126 - ```js 127 - container.innerHTML = window.SurfaceState.errorCard({ 128 - heading: "Couldn't refresh skills", 129 - desc: 'Reload the page to try again.', 130 - serverMessage: err?.serverMessage ?? err?.message, 131 - }); 132 - ``` 133 - 134 - ```js 135 - window.SurfaceState.replaceLoading('pulse-briefing', window.SurfaceState.errorCard({ 136 - heading: "Couldn't refresh briefing", 137 - desc: 'Reload the page to try again.', 138 - serverMessage: err?.serverMessage ?? err?.message, 139 - })); 140 - ``` 141 - 142 - ## D5. Custom loading scaffolds 143 - 144 - **Decision** 145 - 146 - - Replace the bespoke loading HTML at: 147 - - `apps/observer/workspace.html:512` 148 - - `apps/activities/_day.html:498` 149 - - `apps/import/workspace.html:610` 150 - - Hard-code the same DOM structure emitted by `SurfaceState.loading(...)` so `replaceLoading(...)` detects first-paint loading via `.surface-state--loading`. 151 - 152 - **Rationale** 153 - 154 - - Consistent first-paint markup lets the shared replacement heuristic work everywhere. 155 - 156 - **Sketch** 157 - 158 - ```html 159 - <div class="surface-state surface-state--loading" role="status" aria-busy="true"> 160 - <div class="surface-state-spinner" aria-hidden="true"></div> 161 - <span class="surface-state-text" data-role="loading-status">Loading observers...</span> 162 - </div> 163 - ``` 164 - 165 - Use the same structure with text adjusted to: 166 - 167 - - `Loading observers...` 168 - - `Loading activities...` 169 - - `Loading imports...` 170 - 171 - ## D6. Importer WS timeout 172 - 173 - Note: the `appEvents.listen` options-overload default `correlationKey` is `'use_id'` (confirmed at `convey/static/websocket.js:310`). Importer opts into `'import_id'` explicitly. This correction was logged at gate approval. 174 - 175 - **Decision** 176 - 177 - - Add `const IMPORT_STALL_TIMEOUT_MS = 10 * 60 * 1000;` near the top of `apps/import/workspace.html`. 178 - - Use the `appEvents.listen` options overload with `correlationKey: 'import_id'`. 179 - 180 - **Rationale** 181 - 182 - - Ten minutes is the pragmatic stall threshold. 183 - - The overload defaults to `use_id`; importer must opt into `import_id`. 184 - 185 - **Sketch** 186 - 187 - ```js 188 - const IMPORT_STALL_TIMEOUT_MS = 10 * 60 * 1000; 189 - const IMPORT_ROW_EVENTS = new Set(['started', 'status', 'completed', 'error']); 190 - const IMPORT_TERMINAL_EVENTS = new Set(['completed', 'error']); 191 - ``` 192 - 193 - ```js 194 - const importEventsCleanup = window.appEvents.listen('importer', { 195 - correlationKey: 'import_id', 196 - schema: ['import_id', 'event'], 197 - timeout: IMPORT_STALL_TIMEOUT_MS, 198 - onTimeout: markRowStalled, 199 - }, (eventData) => { 200 - if (!IMPORT_ROW_EVENTS.has(eventData.event)) return; 201 - updateImportRow(eventData.import_id, eventData); 202 - if (!IMPORT_TERMINAL_EVENTS.has(eventData.event)) { 203 - importEventsCleanup.pending.track(eventData.import_id); 204 - } 205 - }); 206 - ``` 207 - 208 - Also arm the timer after local synthetic starts and after `loadImports()` discovers already-running rows. 209 - 210 - ## D7. Importer stalled UI 211 - 212 - **Decision** 213 - 214 - - Add `import-row--stalled`. 215 - - Replace the row’s blue running badge with a non-spinning amber `stalled` badge. 216 - - Surface the `import_id` in the row. 217 - - Mirror the stalled state into `importEvents[importId]` so the progress route can show it too. 218 - 219 - **Rationale** 220 - 221 - - Stalled is non-terminal and must be visually distinct from running, failed, and completed. 222 - 223 - **Sketch** 224 - 225 - ```css 226 - .import-row--stalled { background: rgba(245, 158, 11, 0.08); } 227 - .import-row--stalled:hover { background: rgba(245, 158, 11, 0.12); } 228 - .import-status.stalled { background: rgba(245, 158, 11, 0.16); color: #92400e; } 229 - .import-stalled-meta { color: #92400e; } 230 - ``` 231 - 232 - ```js 233 - function markRowStalled(importId) { 234 - const row = document.querySelector(`tr[data-import-id="${importId}"]`); 235 - if (!row) return; 236 - row.classList.add('import-row--stalled'); 237 - row.querySelector('.status-cell').innerHTML = 238 - `<span class="import-status stalled">Stalled</span><div class="progress-detail import-stalled-meta">Stalled — no updates in 10 minutes. Import ID: ${escapeHtml(importId)}. Reload to retry.</div>`; 239 - importEvents[importId] = { ...(importEvents[importId] || {}), import_id: importId, event: 'stalled', stalled: true }; 240 - refreshInlineProgress(importId, importEvents[importId]); 241 - } 242 - ``` 243 - 244 - `updateImportRow(...)` clears the stalled row class when a later `started` or `status` arrives. 245 - 246 - ## D8. `loadMoreImports` banner 247 - 248 - **Decision** 249 - 250 - - On `loadMoreImports()` failure, insert a singleton `.surface-state-refresh-error` banner between the table and `#importLoadMore`. 251 - - Preserve existing rows and keep the load-more control. 252 - 253 - **Rationale** 254 - 255 - - The rows already on screen remain useful state. 256 - 257 - **Sketch** 258 - 259 - ```js 260 - function renderLoadMoreError(err) { 261 - document.getElementById('importLoadMoreError')?.remove(); 262 - const loadMore = document.getElementById('importLoadMore'); 263 - if (!loadMore?.parentNode) return; 264 - const banner = document.createElement('div'); 265 - banner.id = 'importLoadMoreError'; 266 - banner.className = 'surface-state-refresh-error'; 267 - banner.innerHTML = `<strong>Couldn't load more imports.</strong> ${escapeHtml(err?.serverMessage ?? err?.message ?? 'Reload the page to try again.')}`; 268 - loadMore.parentNode.insertBefore(banner, loadMore); 269 - } 270 - ``` 271 - 272 - Remove `#importLoadMoreError` on the next successful page append. 273 - 274 - ## D9. `_detail.html` failure split 275 - 276 - **Decision** 277 - 278 - - `#importMeta` becomes the primary error surface with a prominent `.surface-state-refresh-error` banner that includes `err.serverMessage`. 279 - - `#overviewContent`, `#importJsonContent`, and `#importedJsonContent` become neutral unavailable/no-data copy. 280 - - Do not use the literal word `Pending` in the error path. 281 - 282 - **Rationale** 283 - 284 - - `Pending` is a legitimate server state and must stay reserved for that state. 285 - - The meta strip is the most visible slot on the page. 286 - 287 - **Sketch** 288 - 289 - ```js 290 - document.getElementById('importMeta').innerHTML = 291 - `<div class="surface-state-refresh-error" role="alert"><strong>Couldn't load import details.</strong> ${escapeHtml(err?.serverMessage ?? err?.message ?? 'Reload the page to try again.')}</div>`; 292 - document.getElementById('overviewContent').innerHTML = '<div class="no-data">Import details are unavailable right now.</div>'; 293 - document.getElementById('importJsonContent').innerHTML = '<span class="no-data">Import metadata unavailable.</span>'; 294 - document.getElementById('importedJsonContent').innerHTML = '<span class="no-data">Processed metadata unavailable.</span>'; 295 - ``` 296 - 297 - ## D10. Observer `loadObservers` 298 - 299 - **Decision** 300 - 301 - - Migrate `loadObservers()` to `window.apiJson(...)`. 302 - - On failure, use `window.SurfaceState.replaceLoading('observersList', window.SurfaceState.errorCard(...))`. 303 - - Remove `showLocalError(..., { retry: true })` from this loader only. 304 - - Leave `showLocalError(...)` itself intact; do not delete the retry branch in this wave. 305 - 306 - **Rationale** 307 - 308 - - First-paint and refresh failures should look like tokens-style data errors, not like local action errors. 309 - 310 - **Sketch** 311 - 312 - ```js 313 - window.SurfaceState.replaceLoading('observersList', window.SurfaceState.errorCard({ 314 - heading: "Couldn't load observers", 315 - desc: 'Reload the page to try again.', 316 - serverMessage: err?.serverMessage ?? err?.message, 317 - })); 318 - ``` 319 - 320 - ## Per-site code sketches 321 - 322 - ### 1. `apps/settings/workspace.html:4769` Plaud sync toggle 323 - 324 - **Before** 325 - ```js 326 - const response = await fetch('api/sync', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ plaud: { enabled } }) }); 327 - const result = await response.json(); 328 - if (result.error) throw new Error(result.error); 329 - showFieldStatus(this, 'saved'); 330 - ``` 331 - 332 - **After** 333 - ```js 334 - const el = this; 335 - const errorHost = prepareFieldErrorHost(el); 336 - try { 337 - await window.saveControl({ 338 - el, errorHost, 339 - fetchArgs: ['api/sync', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ plaud: { enabled } }) }], 340 - onSuccess: () => showFieldStatus(el, 'saved'), 341 - }); 342 - } catch (_) {} 343 - ``` 344 - 345 - ### 2. `apps/settings/workspace.html:4786` Granola sync toggle 346 - 347 - **Before** 348 - ```js 349 - const response = await fetch('api/sync', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ granola: { enabled } }) }); 350 - const result = await response.json(); 351 - if (result.error) throw new Error(result.error); 352 - showFieldStatus(this, 'saved'); 353 - ``` 354 - 355 - **After** 356 - ```js 357 - const el = this; 358 - const errorHost = prepareFieldErrorHost(el); 359 - try { 360 - await window.saveControl({ 361 - el, errorHost, 362 - fetchArgs: ['api/sync', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ granola: { enabled } }) }], 363 - onSuccess: () => showFieldStatus(el, 'saved'), 364 - }); 365 - } catch (_) {} 366 - ``` 367 - 368 - ### 3. `apps/settings/workspace.html:4803` Obsidian sync toggle 369 - 370 - **Before** 371 - ```js 372 - const response = await fetch('api/sync', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ obsidian: { enabled } }) }); 373 - const result = await response.json(); 374 - if (result.error) throw new Error(result.error); 375 - showFieldStatus(this, 'saved'); 376 - ``` 377 - 378 - **After** 379 - ```js 380 - const el = this; 381 - const errorHost = prepareFieldErrorHost(el); 382 - try { 383 - await window.saveControl({ 384 - el, errorHost, 385 - fetchArgs: ['api/sync', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ obsidian: { enabled } }) }], 386 - onSuccess: () => showFieldStatus(el, 'saved'), 387 - }); 388 - } catch (_) {} 389 - ``` 390 - 391 - ### 4. `apps/settings/workspace.html:5897` `saveRetentionConfig(data)` 392 - 393 - **Before** 394 - ```js 395 - const response = await fetch('api/storage', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(data) }); 396 - const result = await response.json(); 397 - if (result.error) throw new Error(result.error); 398 - showFieldStatus(el, 'saved'); 399 - ``` 400 - 401 - **After** 402 - ```js 403 - const el = document.getElementById('retentionModeField'); 404 - const errorHost = el ? prepareFieldErrorHost(el) : null; 405 - try { 406 - await window.saveControl({ 407 - el, errorHost, 408 - fetchArgs: ['api/storage', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(data) }], 409 - onSuccess: () => showFieldStatus(el, 'saved'), 410 - }); 411 - } catch (_) {} 412 - ``` 413 - 414 - ### 5. `apps/settings/workspace.html:4310` provider/tier/backup loop 415 - 416 - **Before** 417 - ```js 418 - const response = await fetch('api/providers', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ [type]: { [field]: value } }) }); 419 - const result = await response.json(); 420 - if (result.error) throw new Error(result.error); 421 - providersData = result; 422 - showFieldStatus(el, 'saved'); 423 - ``` 424 - 425 - **After** 426 - ```js 427 - const errorHost = prepareFieldErrorHost(el); 428 - try { 429 - await window.saveControl({ 430 - el, errorHost, 431 - fetchArgs: ['api/providers', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ [type]: { [field]: value } }) }], 432 - onSuccess: (result) => { providersData = result; /* existing provider/tier side-effects */ showFieldStatus(el, 'saved'); }, 433 - }); 434 - } catch (_) {} 435 - ``` 436 - 437 - ### 6. `apps/settings/workspace.html:4342` cogitate auth change 438 - 439 - **Before** 440 - ```js 441 - const response = await fetch('api/providers', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ auth: { [provider]: authSelect.value } }) }); 442 - const result = await response.json(); 443 - if (result.error) throw new Error(result.error); 444 - providersData = result; 445 - showFieldStatus(authSelect, 'saved'); 446 - ``` 447 - 448 - **After** 449 - ```js 450 - const el = this; 451 - const errorHost = prepareFieldErrorHost(el); 452 - try { 453 - await window.saveControl({ 454 - el, errorHost, 455 - fetchArgs: ['api/providers', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ auth: { [provider]: el.value } }) }], 456 - onSuccess: (result) => { providersData = result; updateTypeProviderKeyWarning('cogitate', provider, result.api_keys, result.auth); showFieldStatus(el, 'saved'); }, 457 - }); 458 - } catch (_) {} 459 - ``` 460 - 461 - ### 7. `apps/settings/workspace.html:4364` google backend change 462 - 463 - **Before** 464 - ```js 465 - const response = await fetch('api/providers', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ google_backend: value }) }); 466 - const result = await response.json(); 467 - if (result.error) throw new Error(result.error); 468 - providersData = result; 469 - showFieldStatus(this, 'saved'); 470 - ``` 471 - 472 - **After** 473 - ```js 474 - const el = this; 475 - const errorHost = prepareFieldErrorHost(el); 476 - try { 477 - await window.saveControl({ 478 - el, errorHost, 479 - fetchArgs: ['api/providers', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ google_backend: value }) }], 480 - onSuccess: (result) => { providersData = result; /* existing vertex/google field toggles */ showFieldStatus(el, 'saved'); }, 481 - }); 482 - } catch (_) {} 483 - ``` 484 - 485 - ### 8. `apps/speakers/workspace.html:1985` `confirmAttribution` 486 - 487 - **Before** 488 - ```js 489 - fetch('/app/speakers/api/confirm-attribution', { ... }) 490 - .then(r => r.json()) 491 - .then(data => { if (data.error) { showStatusBySentence(sentenceId, friendlyError(data.error), 'error'); return; } ... }) 492 - .catch(() => { showStatusBySentence(sentenceId, 'Failed to confirm attribution — try again', 'error'); }); 493 - ``` 494 - 495 - **After** 496 - ```js 497 - try { 498 - const data = await window.apiJson('/app/speakers/api/confirm-attribution', { ... }); 499 - if (data.error) { showStatusBySentence(sentenceId, resolveSpeakerError({ serverMessage: data.error, message: data.error }), 'error'); return; } 500 - ... 501 - } catch (err) { 502 - showStatusBySentence(sentenceId, resolveSpeakerError(err), 'error'); 503 - } 504 - ``` 505 - 506 - ### 9. `apps/speakers/workspace.html:2020` `correctAttribution` 507 - 508 - **Before** 509 - ```js 510 - fetch('/app/speakers/api/correct-attribution', { ... }) 511 - .then(r => r.json()) 512 - .then(data => { if (data.error) { showStatusBySentence(sentenceId, friendlyError(data.error), 'error'); return; } ... }) 513 - .catch(() => { showStatusBySentence(sentenceId, 'Failed to correct attribution — try again', 'error'); }); 514 - ``` 515 - 516 - **After** 517 - ```js 518 - try { 519 - const data = await window.apiJson('/app/speakers/api/correct-attribution', { ... }); 520 - if (data.error) { showStatusBySentence(sentenceId, resolveSpeakerError({ serverMessage: data.error, message: data.error }), 'error'); return; } 521 - ... 522 - } catch (err) { 523 - showStatusBySentence(sentenceId, resolveSpeakerError(err), 'error'); 524 - } 525 - ``` 526 - 527 - ### 10. `apps/speakers/workspace.html:2047` `assignAttribution` 528 - 529 - **Before** 530 - ```js 531 - fetch('/app/speakers/api/assign-attribution', { ... }) 532 - .then(r => r.json()) 533 - .then(data => { if (data.error) { showStatusBySentence(sentenceId, friendlyError(data.error), 'error'); return; } ... }) 534 - .catch(() => { showStatusBySentence(sentenceId, 'Failed to assign attribution — try again', 'error'); }); 535 - ``` 536 - 537 - **After** 538 - ```js 539 - try { 540 - const data = await window.apiJson('/app/speakers/api/assign-attribution', { ... }); 541 - if (data.error) { showStatusBySentence(sentenceId, resolveSpeakerError({ serverMessage: data.error, message: data.error }), 'error'); return; } 542 - ... 543 - } catch (err) { 544 - showStatusBySentence(sentenceId, resolveSpeakerError(err), 'error'); 545 - } 546 - ``` 547 - 548 - ### 11. `apps/home/workspace.html:1639` `refreshRoutines` 549 - 550 - **Before** 551 - ```js 552 - fetch('/app/home/api/pulse') 553 - .then(r => r.json()) 554 - .then(data => { ... }) 555 - .catch(function(err) { console.error('home: refreshRoutines failed', err); }); 556 - ``` 557 - 558 - **After** 559 - ```js 560 - try { 561 - const data = await window.apiJson('/app/home/api/pulse'); 562 - ... 563 - } catch (err) { 564 - console.error('home: refreshRoutines failed', err); 565 - document.getElementById('pulse-routines')?.replaceChildren(); 566 - document.getElementById('pulse-routines')?.insertAdjacentHTML('afterbegin', window.SurfaceState.errorCard({ heading: "Couldn't refresh routines", desc: 'Reload the page to try again.', serverMessage: err?.serverMessage ?? err?.message })); 567 - } 568 - ``` 569 - 570 - ### 12. `apps/home/workspace.html:1702` `refreshSkills` 571 - 572 - **Before** 573 - ```js 574 - fetch('/app/home/api/pulse') 575 - .then(r => r.json()) 576 - .then(data => { ... }) 577 - .catch(function(err) { console.error('home: refreshSkills failed', err); }); 578 - ``` 579 - 580 - **After** 581 - ```js 582 - try { 583 - const data = await window.apiJson('/app/home/api/pulse'); 584 - ... 585 - } catch (err) { 586 - console.error('home: refreshSkills failed', err); 587 - document.getElementById('pulse-skills')?.replaceChildren(); 588 - document.getElementById('pulse-skills')?.insertAdjacentHTML('afterbegin', window.SurfaceState.errorCard({ heading: "Couldn't refresh skills", desc: 'Reload the page to try again.', serverMessage: err?.serverMessage ?? err?.message })); 589 - } 590 - ``` 591 - 592 - ### 13. `apps/home/workspace.html:1772` `refreshBriefing` 593 - 594 - **Before** 595 - ```js 596 - fetch('/app/home/api/briefing') 597 - .then(function(r) { return r.json(); }) 598 - .then(function(data) { ... }) 599 - .catch(function(err) { console.error('home: refreshBriefing failed', err); }); 600 - ``` 601 - 602 - **After** 603 - ```js 604 - try { 605 - const data = await window.apiJson('/app/home/api/briefing'); 606 - ... 607 - } catch (err) { 608 - console.error('home: refreshBriefing failed', err); 609 - window.SurfaceState.replaceLoading('pulse-briefing', window.SurfaceState.errorCard({ heading: "Couldn't refresh briefing", desc: 'Reload the page to try again.', serverMessage: err?.serverMessage ?? err?.message })); 610 - } 611 - ``` 612 - 613 - ### 14. `apps/activities/_day.html:896` `loadData` 614 - 615 - **Before** 616 - ```js 617 - fetch(`/app/activities/api/day/${day}/activities`).then(r => r.json()).then(acts => { 618 - allActivities = acts || []; 619 - ... 620 - }).catch(() => { 621 - document.getElementById('timelineView').innerHTML = '<div class="timeline-empty">Failed to load data.</div>'; 622 - }); 623 - ``` 624 - 625 - **After** 626 - ```js 627 - try { 628 - const acts = await window.apiJson(`/app/activities/api/day/${day}/activities`); 629 - allActivities = acts || []; 630 - ... 631 - } catch (err) { 632 - window.SurfaceState.replaceLoading('timelineView', window.SurfaceState.errorCard({ heading: "Couldn't load activities", desc: 'Reload the page to try again.', serverMessage: err?.serverMessage ?? err?.message })); 633 - } 634 - ``` 635 - 636 - ### 15. `apps/import/workspace.html:978` `loadImports` 637 - 638 - **Before** 639 - ```js 640 - const response = await fetch('/app/import/api/list?page=1&per_page=25'); 641 - const data = await response.json(); 642 - ... 643 - document.getElementById('importListContent').innerHTML = `...Retry...`; 644 - ``` 645 - 646 - **After** 647 - ```js 648 - const data = await window.apiJson('/app/import/api/list?page=1&per_page=25'); 649 - ... 650 - document.getElementById('importLoadMoreError')?.remove(); 651 - document.querySelectorAll('.import-table tbody tr').forEach(row => { 652 - if (row.querySelector('.import-status.running')) importEventsCleanup.pending.track(row.dataset.importId); 653 - }); 654 - // catch: 655 - window.SurfaceState.replaceLoading('importListContent', window.SurfaceState.errorCard({ heading: "Couldn't load import history", desc: 'Reload the page to try again.', serverMessage: err?.serverMessage ?? err?.message })); 656 - ``` 657 - 658 - ### 16. `apps/import/workspace.html:1052` `loadMoreImports` 659 - 660 - **Before** 661 - ```js 662 - const response = await fetch(`/app/import/api/list?page=${currentPage}&per_page=25`); 663 - const data = await response.json(); 664 - ... 665 - console.error('Failed to load more imports:', err); 666 - ``` 667 - 668 - **After** 669 - ```js 670 - const data = await window.apiJson(`/app/import/api/list?page=${currentPage}&per_page=25`); 671 - document.getElementById('importLoadMoreError')?.remove(); 672 - ... 673 - // catch: 674 - console.error('Failed to load more imports:', err); 675 - renderLoadMoreError(err); 676 - ``` 677 - 678 - ### 17. `apps/import/workspace.html:1791` importer listener 679 - 680 - **Before** 681 - ```js 682 - appEvents.listen('importer', eventData => updateImportRow(eventData.import_id, eventData)); 683 - ``` 684 - 685 - **After** 686 - ```js 687 - const importEventsCleanup = window.appEvents.listen('importer', { 688 - correlationKey: 'import_id', 689 - schema: ['import_id', 'event'], 690 - timeout: IMPORT_STALL_TIMEOUT_MS, 691 - onTimeout: markRowStalled, 692 - }, (eventData) => { 693 - if (!IMPORT_ROW_EVENTS.has(eventData.event)) return; 694 - updateImportRow(eventData.import_id, eventData); 695 - if (!IMPORT_TERMINAL_EVENTS.has(eventData.event)) importEventsCleanup.pending.track(eventData.import_id); 696 - }); 697 - ``` 698 - 699 - ### 18. `apps/import/_detail.html:437` detail bootstrap load 700 - 701 - **Before** 702 - ```js 703 - fetch('/app/import/api/{{ timestamp }}') 704 - .then(r => r.json()) 705 - .then(data => { ... }) 706 - .catch(err => { 707 - document.getElementById('importMeta').innerHTML = '<span style="color:red;">Error loading import details</span>'; 708 - document.getElementById('overviewContent').innerHTML = '<div class="no-data">Error loading data</div>'; 709 - document.getElementById('importJsonContent').innerHTML = '<span class="no-data">Error loading data</span>'; 710 - document.getElementById('importedJsonContent').innerHTML = '<span class="no-data">Error loading data</span>'; 711 - }); 712 - ``` 713 - 714 - **After** 715 - ```js 716 - window.apiJson('/app/import/api/{{ timestamp }}') 717 - .then(data => { ... }) 718 - .catch(err => { 719 - document.getElementById('importMeta').innerHTML = `<div class="surface-state-refresh-error" role="alert"><strong>Couldn't load import details.</strong> ${escapeHtml(err?.serverMessage ?? err?.message ?? 'Reload the page to try again.')}</div>`; 720 - document.getElementById('overviewContent').innerHTML = '<div class="no-data">Import details are unavailable right now.</div>'; 721 - document.getElementById('importJsonContent').innerHTML = '<span class="no-data">Import metadata unavailable.</span>'; 722 - document.getElementById('importedJsonContent').innerHTML = '<span class="no-data">Processed metadata unavailable.</span>'; 723 - }); 724 - ``` 725 - 726 - ### 19. `apps/observer/workspace.html:743` `loadObservers` 727 - 728 - **Before** 729 - ```js 730 - const response = await fetch('/app/observer/api/list'); 731 - const payload = await response.json(); 732 - ... 733 - showLocalError('couldn\'t load observers — the server may be unreachable. it will retry automatically, or you can retry now.', { retry: true }); 734 - ``` 735 - 736 - **After** 737 - ```js 738 - const payload = await window.apiJson('/app/observer/api/list'); 739 - ... 740 - // catch: 741 - window.SurfaceState.replaceLoading('observersList', window.SurfaceState.errorCard({ heading: "Couldn't load observers", desc: 'Reload the page to try again.', serverMessage: err?.serverMessage ?? err?.message })); 742 - ``` 743 - 744 - ## Audit grep checklist 745 - 746 - Repo-local scope doc with the original audit regexes is not present in this worktree. Use this derived checklist in the audit stage. 747 - 748 - **Legacy patterns that must disappear** 749 - 750 - - `apps/settings/workspace.html`: `fetch$'api/(sync|providers|storage)'` 751 - - `apps/speakers/workspace.html`: `fetch\('/app/speakers/api/(confirm-attribution|correct-attribution|assign-attribution)'` 752 - - `apps/home/workspace.html`: `console\.error\('home: refresh(Routines|Skills|Briefing) failed'` 753 - - `apps/activities/_day.html`: `fetch\(`/app/activities/api/day/\$\{day\}/activities`$` 754 - - `apps/import/workspace.html`: `fetch$'/app/import/api/list\?page=1&per_page=25'` 755 - - `apps/import/workspace.html`: `fetch\(`/app/import/api/list\?page=\$\{currentPage\}&per_page=25`$` 756 - - `apps/import/workspace.html`: `appEvents\.listen$'importer', eventData => updateImportRow\(eventData\.import_id, eventData$\)` 757 - - `apps/import/_detail.html`: `fetch$'/app/import/api/\{\{ timestamp \}\}'` 758 - - `apps/observer/workspace.html`: `showLocalError\([^)]*\{ retry: true \}$` 759 - - `apps/observer/workspace.html`: `fetch$'/app/observer/api/list'` 760 - 761 - **New patterns that must appear** 762 - 763 - - `apps/settings/workspace.html`: `prepareFieldErrorHost\(` 764 - - `apps/settings/workspace.html`: `errorHost` 765 - - `apps/settings/workspace.html`: `showFieldStatus\(.*'saved'` 766 - - `apps/speakers/workspace.html`: `function resolveSpeakerError\(err$` 767 - - `apps/home/workspace.html`: `window\.SurfaceState\.errorCard` 768 - - `apps/home/workspace.html`: `window\.SurfaceState\.replaceLoading\('pulse-briefing'` 769 - - `apps/activities/_day.html`: `window\.SurfaceState\.replaceLoading\('timelineView'` 770 - - `apps/import/workspace.html`: `const IMPORT_STALL_TIMEOUT_MS = 10 \* 60 \* 1000` 771 - - `apps/import/workspace.html`: `correlationKey: 'import_id'` 772 - - `apps/import/workspace.html`: `onTimeout: markRowStalled` 773 - - `apps/import/workspace.html`: `import-row--stalled` 774 - - `apps/import/workspace.html`: `surface-state-refresh-error` 775 - - `apps/import/_detail.html`: `surface-state-refresh-error` 776 - - `apps/observer/workspace.html`: `window\.SurfaceState\.replaceLoading\('observersList'` 777 - 778 - ## Risks and open questions 779 - 780 - - Importer dedup can emit `started` without a terminal event; Wave 1 only surfaces that as stalled UI. 781 - - `file_imported` / `enrichment_ready` must be ignored by row-state caching or they will clobber stage/elapsed fields. 782 - - `prepareFieldErrorHost(...)` must seed `fieldHelperText` before clearing the `<small>`, or the original helper text is lost after the first save. 783 - - Skills/routines containers are conditionally server-rendered; when absent, the refresh-error path no-ops instead of synthesizing a new section. 784 - - `showLocalError(...)` may end this wave with no `{ retry: true }` callers; keep the retry branch for a later cleanup wave. 785 - 786 - ## CPO follow-up 787 - 788 - The importer dedup path at `think/importers/cli.py:726-741` can emit `importer:started` without a terminal `completed`/`error` event. Wave 1 surfaces this client-side as the stalled UI. The server-side fix (always emit a terminal event, or a distinct `deduped` short-circuit event) is out of scope for this wave — flag to CPO as a separate ticket after Wave 1 ships.

-384

scratch/design-convey-error-wave2.md

··· 1 - # Convey error handling — Wave 2 design 2 - 3 - See `scratch/recon-convey-error-wave2.md` for the call-site inventory and backend contracts. This doc records the migration choices you will implement. 4 - 5 - ## Scope (recap) 6 - 7 - Wave 2 migrates 17 Tier-2 sites plus 1 prerequisite scaffold upgrade across 8 app files. 8 - 9 - D9 Sol updated-days is deferred. Do not touch that client path in Wave 2. See Out-of-scope follow-ups. 10 - 11 - ## Conventions 12 - 13 - - Keep `window.logError(err, { context: ... })` alongside every owner-visible surface. Logging and UI are separate requirements. 14 - - Do not add retry buttons. Owner action is reload unless the site already has its own mutation button. 15 - - Reuse existing Wave 1 helpers in-place. Do not add overlapping helpers for home, settings, or speakers. 16 - - Use `window.apiJson` for every migrated HTTP path. Raw `fetch(...).then(r => r.json())` leaves non-2xx and malformed JSON silent. 17 - - Use `window.SurfaceState.errorCard(...)` for first-paint section failures and `surface-state-refresh-error` only when stale content is intentionally preserved. 18 - - Use `window.appEvents.listen(..., options, handler)` for websocket timeout/drop handling. The overload contract is at `convey/static/websocket.js:300-322`. 19 - - **UI patience, not backend SLA.** Client-side timeouts on long-running async work (websocket listeners, polling loops) are owner-patience thresholds — the point at which the UI stops claiming "still working" and tells the owner to reload. They are not statements about how long the backend should take. Backend work may continue past the UI timeout; the client simply stops lying about its state. This framing is reusable for Wave 3 background-task health. 20 - 21 - ## Site-by-site decisions 22 - 23 - ### D1 — Home refreshVitals + refreshNarrative 24 - 25 - Targets: `apps/home/workspace.html:1582-1630` 26 - 27 - Use `window.apiJson('/app/home/api/pulse')` in both refreshers. Keep `malformedHomeResponse()` for 200-shape failures instead of inventing a second parse helper. 28 - 29 - For `refreshVitals`, preserve the existing `#pulse-vitals` content on failure and append a singleton refresh-error sibling after `#pulse-vitals`. Copy: `Couldn't refresh vitals — showing last known state.` Keep the current render path untouched on success. 30 - 31 - For `refreshNarrative`, preserve the existing `#pulse-narrative` content on failure and append a singleton refresh-error sibling after `#pulse-narrative`. Copy: `Couldn't refresh narrative — showing last known state.` Do not blank the rendered markdown on refresh failure. 32 - 33 - These are refresh-only paths. There is already first-paint content in the HTML, so this is a stale-preserving migration, not a loading-scaffold migration. 34 - 35 - ### D2 — Link refreshStatus + refreshDevices 36 - 37 - Targets: `apps/link/workspace.html:171-199` 38 - 39 - Add a stable container id to the existing status row: use `#link-status-panel` on the current `.link-status-row`. `refreshStatus()` should call `window.apiJson('/app/link/api/status', ...)`, shape-validate `typeof data.enrolled === 'boolean'`, and throw `new window.ApiError({ cause: 'parse', status: 200, serverMessage: 'Unexpected status shape' })` on mismatch. 40 - 41 - On `refreshStatus()` failure, do not call `setStatus(...)`. Leave the last visible status text and LAN nudge state unchanged. Render a refresh-error sibling after `#link-status-panel` with copy `Can't reach pairing service — reload to try again.` This is PD9. 42 - 43 - `refreshDevices()` should also move to `window.apiJson('/app/link/api/devices', ...)`. 44 - 45 - Use a split surface for devices: 46 - 47 - - First failure before the first successful device render: replace `#link-devices-list` contents with `SurfaceState.errorCard({ heading: "Couldn't load paired devices", desc: "Reload to try again.", serverMessage: err.serverMessage })`. 48 - - Later refresh failure after at least one successful render: leave the stale device list visible and append a `surface-state-refresh-error` sibling after `#link-devices-list` with copy `Couldn't refresh paired devices — showing last known state.` 49 - 50 - Track first-success with a small local boolean. Do not treat transport failure as `offline`. 51 - 52 - ### D3 — Chat listener 53 - 54 - Target: `apps/chat/workspace.html:60-167` 55 - 56 - Keep the live listener only for `today`, but switch it to the options overload: 57 - 58 - - `schema: ['kind']` 59 - - `correlationKey: 'use_id'` 60 - - `timeout: 3 * 60 * 1000` 61 - - `onDrop: window.logError` 62 - - `onTimeout: handleChatTalentTimeout` 63 - 64 - PD3 applies. Do not implement an owner-reply watchdog in this wave. 65 - 66 - Only track spawned talent cards. Call `cleanup.pending.track(useId)` inside `appendEventFromLive()` when `kind === 'talent_spawned'` and `use_id` exists. Do not track `owner_message`, `sol_message`, or `reflection_ready`. 67 - 68 - Add a stalled visual state for talent cards: 69 - 70 - - New variant class: `chat-talent-card--stalled` 71 - - New status value: `data-talent-status="stalled"` 72 - - New copy: `Talent stopped responding — reload to retry` 73 - 74 - On timeout, find the matching active talent card by `data-talent-use-id`, convert it in place to the stalled variant, and leave the rest of the transcript unchanged. If the card is already finished or errored, the timeout callback is a no-op. 75 - 76 - ### D4 — Entities cortex listener 77 - 78 - Target: `apps/entities/workspace.html:3216-3312` 79 - 80 - Switch `window.appEvents.listen('cortex', ...)` to the options overload with: 81 - 82 - - `schema: ['event', 'use_id']` 83 - - `timeout: 2 * 60 * 1000` 84 - - `correlationKey: 'use_id'` 85 - - `onDrop: window.logError` 86 - - `onTimeout: handleEntitiesTimeout` 87 - 88 - Use 2 minutes as the UI stall threshold. There is no shorter backend contract in `apps/entities/routes.py:498-574`, and cortex still allows much longer work by default (`think/cortex.py:335-339` defaults to 10 minutes). Two minutes is a UI patience threshold for these owner-triggered, single-agent actions, not a backend SLA. 89 - 90 - Track ids in both paths: 91 - 92 - - `submitEntityAssist()` tracks the real `use_id` immediately after the POST succeeds and the temp id is remapped. 93 - - `listenForAgentCompletion()` tracks the `agentId` when description generation starts. 94 - 95 - Timeout behavior must fail both maps cleanly and only once: 96 - 97 - - If `pendingEntities.has(useId)`, call the existing failure path with copy `Entity assist timed out — reload to retry`. 98 - - If `pendingAgentCallbacks.has(useId)`, remove the callback entry first, then invoke the callback with a timeout-shaped failure so the textarea/button unlock and the description inline error renders. 99 - 100 - Ignore late websocket `finish`/`error` events after timeout by deleting the pending entry before surfacing the timeout. 101 - 102 - ### D5 — Settings saveField 103 - 104 - Targets: `apps/settings/workspace.html:3270-3512` plus all `[data-section][data-key]` autosave controls 105 - 106 - Keep `saveField()` as the debounced orchestrator. Do not move debounce into `saveControl()`. 107 - 108 - PD5: when `el` is null, or when the caller is the auto-detected `identity.timezone` path, bypass `saveControl()` and call `window.apiJson('api/config', ...)` directly inside the existing debounce. On failure, log with `window.logError(...)`. There is no UI control to revert in that path. 109 - 110 - For normal controls, migrate `saveField()` to `prepareFieldErrorHost(el)` plus `window.saveControl(...)`. 111 - 112 - PD7 sub-choice: use a local last-known-good snapshot per element. This is required by `saveControl()` internals: 113 - 114 - - `convey/static/api.js:208-213` reads `readValue` before the request and uses that as `previousValue`. 115 - - `convey/static/api.js:220` writes `savedControlValues` from the live DOM value after success. 116 - - `convey/static/api.js:227-228` writes `previousValue` back on failure. 117 - 118 - That ordering means: 119 - 120 - - default `getInitialControlValue()` is wrong for JS-populated settings fields because it falls back to `defaultValue/defaultChecked` (`convey/static/api.js:98-113`) 121 - - a per-call snapshot of the changed DOM value would also be wrong, because `previousValue` is captured before the fetch starts 122 - 123 - Design: 124 - 125 - - Seed `el.__lastKnownValue` for every `saveField`-managed control during `populateFields()`, after the DOM has been filled from config. 126 - - Pass `readValue: () => el.__lastKnownValue` for all non-env controls. 127 - - Update `el.__lastKnownValue` in `onSuccess` to the control’s current logical value. 128 - - Keep the default `writeValue`. The field types in scope are text, textarea, select, checkbox, and password, so the stock writer is enough. 129 - 130 - PD6 env-key nuance (revised): **do not route env fields through `saveControl()` at all.** Reason: pre-clearing the field before `saveControl()` starts is a UX regression — on failure the user's typed secret is lost and they must re-type. Post-success clear is current UX; a pre-save clear is new behavior. Instead, env fields use a direct path: 131 - 132 - - `prepareFieldErrorHost(el)` to seed the `<small>` host 133 - - `window.apiJson('api/config', { method: 'PUT', ... })` with the debounced `value` 134 - - On success: clear `el.value = ''` (preserve current post-success UX), run existing key-validation and env-status refresh, call `showFieldStatus(el, 'saved')` 135 - - On failure: render the inline error directly into the `<small>` host using the same `[data-control-save-error]` markup shape that `saveControl()` produces (copy exactly so `.settings-field small .control-save-error` CSS applies), keep the typed secret in `el.value`, call `window.logError(...)` 136 - - No weakmap interaction anywhere 137 - 138 - This sidesteps the secret-caching problem entirely while preserving retry-without-retype for env saves. Non-env controls still use the `saveControl()` path per PD7. 139 - 140 - Error behavior: 141 - 142 - - Success path still calls `showFieldStatus(el, 'saved')` 143 - - Failure path relies on `saveControl()` to render the field-local error in the prepared `<small>` 144 - - Every catch still logs via `window.logError` 145 - 146 - ### D6 — Entities confirmEntityDelete 147 - 148 - Targets: `apps/entities/workspace.html:3436-3465` 149 - 150 - Drop the optimistic delete. The order becomes: 151 - 152 - 1. Disable the confirm button. 153 - 2. Call `window.apiJson(...)` on the DELETE route. 154 - 3. On success, clear `detected-action-error`, close the modal, then call `removeEntityFromUI(entityName)`. 155 - 4. On failure, re-enable the confirm button, close the modal, and show `showInlineError('detected-action-error', err.serverMessage || "Couldn't delete entity")`. 156 - 157 - Do not fall back to `loadEntities()` as the primary repair path. Preserve the current local success path by keeping `removeEntityFromUI()` as the post-success side effect. 158 - 159 - ### D7 — Entities loadEntities + loadJournalEntities 160 - 161 - Targets: `apps/entities/workspace.html:1258-1261`, `2636-2703` 162 - 163 - PD4 applies. Upgrade the loading scaffold first so `SurfaceState.replaceLoading()` can replace in place. 164 - 165 - Before: 166 - 167 - ```html 168 - <div id="entities-loading" class="entities-loading"> 169 - <div class="spinner"></div> 170 - <p>Loading entities...</p> 171 - </div> 172 - ``` 173 - 174 - After: 175 - 176 - ```html 177 - <div id="entities-loading" class="entities-loading"> 178 - <div class="surface-state surface-state--loading" role="status" aria-busy="true"> 179 - <div class="surface-state-spinner" aria-hidden="true"></div> 180 - <span class="surface-state-text" data-role="loading-status">Loading entities...</span> 181 - </div> 182 - </div> 183 - ``` 184 - 185 - Keep `#entities-loading` as the outer id so callers do not change. The important change is the inner `.surface-state--loading` child, because `replaceLoading()` only replaces in place when that marker exists (`convey/static/app.js:1606-1635`). 186 - 187 - Then migrate both loaders to `window.apiJson(...)` and replace the catch-time scaffold pollution with: 188 - 189 - - `window.logError(...)` 190 - - `window.SurfaceState.replaceLoading('entities-loading', window.SurfaceState.errorCard({ heading: "Couldn't load entities", desc: "Reload to try again.", serverMessage: err.serverMessage }))` 191 - 192 - Use the same pattern for `loadJournalEntities()` with journal-specific copy only if it materially improves clarity. Otherwise keep one consistent `Couldn't load entities` surface. 193 - 194 - ### D8 — Settings four loaders 195 - 196 - Targets: 197 - 198 - - `loadTranscribeBackends()` at `apps/settings/workspace.html:4531-4554` 199 - - `loadObserve()` at `4675-4683` 200 - - `loadSync()` at `4760-4768` 201 - - `loadStorage()` at `5800-5808` 202 - 203 - These sections do not have loading scaffolds, so `replaceLoading()` is the wrong primitive. Add one top-of-section status slot per section, immediately under the section description: 204 - 205 - - `#transcriptionLoadState` inside `#section-transcription` 206 - - `#observerLoadState` inside `#section-observer` 207 - - `#syncLoadState` inside `#section-sync` 208 - - `#storageLoadState` inside `#section-storage` 209 - 210 - Use `SurfaceState.errorCard(...)` in those slots. 211 - 212 - Decisions by loader: 213 - 214 - - `loadTranscribeBackends()`: targeted transcription-only surface. Render `Couldn't load transcription backends` in `#transcriptionLoadState`, disable the backend selector until success, and do not let this failure abort `loadConfig()`. `loadConfig()` should still populate the rest of the settings page. 215 - - `loadObserve()`: render `Couldn't load observer settings` in `#observerLoadState` and disable `#field-tmux-enabled` plus `#field-tmux-capture-interval` until success. 216 - - `loadSync()`: render `Couldn't load sync settings` in `#syncLoadState` and disable the Plaud, Granola, and Obsidian toggles until success. 217 - - `loadStorage()`: render `Couldn't load storage settings` in `#storageLoadState`. On first paint, disable retention controls until success. On later refreshes after cleanup actions, keep stale storage numbers visible and use the same slot for refresh-error copy instead of blanking the section. 218 - 219 - All four loaders still log failures. Clear the section slot on the next successful load. 220 - 221 - ### D9 — Sol updated-days — DEFERRED 222 - 223 - Target: `apps/sol/workspace.html:1363-1385` 224 - 225 - PD1 applies. Do not change this client path in Wave 2. 226 - 227 - Reason: `apps/sol/routes.py:637-644` converts backend failure into `[]`, so the client cannot distinguish empty from error. Leave the existing `.catch(() => { banner.style.display = 'none'; })` unchanged. 228 - 229 - Flag the backend-contract follow-up in Out-of-scope follow-ups and in implementation-stage gate output. 230 - 231 - ### D10 — Sol loadIdentity 232 - 233 - Target: `apps/sol/workspace.html:1197-1231` 234 - 235 - PD2 applies. 236 - 237 - Move to `window.apiJson('/app/sol/api/identity')`. Remove the dead `if (data.error)` branch, because the route returns real `500 + {error: ...}` on failure and has no 200-side disabled contract. 238 - 239 - On failure, replace `#sol-identity` contents with: 240 - 241 - - heading: `Couldn't load identity` 242 - - desc: `Reload to try again.` 243 - - server message: `err.serverMessage` 244 - 245 - Do not hide the section anymore. A missing identity card should stop meaning “backend failed silently.” 246 - 247 - ### D11 — Health retry-import 248 - 249 - Target: `apps/health/workspace.html:1739-1759` 250 - 251 - PD8 applies. 252 - 253 - Move the click handler to `window.apiJson('/app/health/api/retry-import', ...)`. Remove the `data.status === 'not_implemented'` branch entirely. 254 - 255 - Use the existing row-local importer card as the surface: 256 - 257 - - while pending, keep the current button-disabled `Retrying...` state 258 - - on success, clear the row error text and show `Retry sent` 259 - - on failure, restore the button label, re-enable the button, and write `err.serverMessage || 'Retry failed'` into that card’s `.activity-card-error` 260 - 261 - This keeps the failure local to the affected import row and lets the server’s 501 message surface unchanged. 262 - 263 - ## Additional sites 264 - 265 - ### Speakers checkOwnerStatus + submitOwnerChoice 266 - 267 - Targets: `apps/speakers/workspace.html:1259-1364` 268 - 269 - Use `window.apiJson(...)` for the owner-status GET, the nested owner-detect POST, and both confirm/reject POSTs. 270 - 271 - Keep `resolveSpeakerError()` as the message normalizer. Do not add a second speakers error helper. 272 - 273 - Surface choices: 274 - 275 - - `checkOwnerStatus()` top-level failure: render an owner-banner error state instead of hiding both banners. Copy baseline: `Couldn't load owner status`. 276 - - nested detect failure: clear `ownerDetectionInFlight`, keep the owner area visible, and render `Couldn't analyze voice patterns` through the same banner surface. 277 - - `submitOwnerChoice()` failure: re-enable the buttons and render the normalized server message in the owner banner. 278 - 279 - On success, preserve the existing `checkOwnerStatus()` refresh behavior. 280 - 281 - ### Speakers loadReview 282 - 283 - Target: `apps/speakers/workspace.html:1725-1752` 284 - 285 - PD10 applies. 286 - 287 - Keep first-paint and refresh separate: 288 - 289 - - First-paint path: when `#spkSentences` still contains the `Loading...` placeholder and no `.spk-sentence`, replace that area with `SurfaceState.errorCard({ heading: "Couldn't load review", desc: "Reload to try again.", serverMessage: err.serverMessage })`. 290 - - Refresh path: when `.spk-sentence` rows already exist, preserve them and render one refresh-error slot at the top of the review panel with copy `Couldn't reload review — showing last known state.` 291 - 292 - Because there is no existing top-of-panel status slot, add one in the detail render above `#spkSentences`: `#spkReviewStatus`. Clear it on the next successful `loadReview()`. 293 - 294 - ### Speakers loadUntilFound 295 - 296 - Target: `apps/speakers/workspace.html:1461-1474` 297 - 298 - Move to `window.apiJson(...)`. Add a catch. Do not let the recursive hash-hunt die as an unhandled rejection. 299 - 300 - On failure: 301 - 302 - - log the error 303 - - stop recursion 304 - - preserve the current segment list 305 - - render a segment-list refresh error in a new lightweight slot above the list, not a global modal 306 - 307 - This is a refresh/deep-link recovery failure, not a first-paint full-panel failure. 308 - 309 - ### Entities generate-description fetch 310 - 311 - Target: `apps/entities/workspace.html:2416-2444` 312 - 313 - Move the kickoff POST to `window.apiJson(...)`. Validate that the response includes a usable `use_id`; otherwise throw a parse-style `ApiError`. 314 - 315 - On kickoff failure: 316 - 317 - - re-enable the textarea 318 - - remove `.generating` 319 - - call `showInlineError('description-save-error', err.serverMessage || "Couldn't start description generation")` 320 - 321 - On websocket timeout/error from D4, route the failure through the same inline surface so the owner sees one consistent description-generation error path. 322 - 323 - ### Entities previewEntityDelete 324 - 325 - Target: `apps/entities/workspace.html:3402-3434` 326 - 327 - Move the preview GET to `window.apiJson(...)`. Keep `showInlineError('detected-action-error', ...)` as the local surface. 328 - 329 - Preserve the existing modal population flow on success. On failure, keep the modal closed and show the server message inline. Do not add a second delete-preview helper. 330 - 331 - ## Implementation order 332 - 333 - 1. Entities scaffold upgrade + `loadEntities()` + `loadJournalEntities()` 334 - 2. Entities `previewEntityDelete()` + `confirmEntityDelete()` + generate-description kickoff 335 - 3. Entities cortex listener timeout/drop handling 336 - 4. Settings `saveField()` 337 - 5. Settings four loaders 338 - 6. Speakers `checkOwnerStatus()` + `submitOwnerChoice()` 339 - 7. Speakers `loadReview()` + `loadUntilFound()` 340 - 8. Home `refreshVitals()` + `refreshNarrative()` 341 - 9. Link `refreshStatus()` + `refreshDevices()` 342 - 10. Sol `loadIdentity()` 343 - 11. Health retry-import 344 - 12. Chat websocket listener 345 - 13. Audit and finalize 346 - 347 - ## Decision log table 348 - 349 - | Decision | Choice | Status | Rationale | 350 - | --- | --- | --- | --- | 351 - | PD1 / D9 | Defer Sol updated-days | Shipped 2026-04-23 | Backend collapses failure into `[]`; client cannot distinguish empty from error | 352 - | PD2 / D10 | `apiJson` + identity `errorCard`; remove `data.error` branch | Shipped 2026-04-23 | Route only has success or real 500 failure | 353 - | PD3 / D3 | Track only `talent_spawned`; 3-minute chat stall state | Shipped 2026-04-23 | `owner_message` has no `use_id`; owner watchdog is separate work | 354 - | D4 timeout | 2-minute entities UI stall threshold | Shipped 2026-04-23 | Interactive entity actions should not hang forever; backend still has a longer timeout | 355 - | PD4 / D7 | Retrofit `#entities-loading` with `.surface-state--loading` child | Shipped 2026-04-23 | `replaceLoading()` only replaces in place when that marker exists | 356 - | PD5 / D5 | Keep `saveField()` debounce; null-el timezone uses direct `apiJson` | Shipped 2026-04-23 | `saveControl()` requires an element | 357 - | PD6 / D5 | Env fields bypass `saveControl()`; use `apiJson` + `prepareFieldErrorHost` directly | Shipped 2026-04-23 | Avoids weakmap secret caching and preserves retry-without-retype UX on failure | 358 - | PD7 / D5 | Per-element `__lastKnownValue` snapshots | Shipped 2026-04-23 | Default snapshot is wrong for JS-populated controls | 359 - | PD8 / D11 | Treat 501 like any other error and surface server message | Shipped 2026-04-23 | Simpler and consistent; backend message already has the right copy | 360 - | PD9 / D2 | Preserve stale link status; do not synthesize `offline` on transport failure | Shipped 2026-04-23 | `offline` is not the same as “pairing service failed” | 361 - | PD10 / speakers | Split `loadReview()` by first paint vs refresh | Shipped 2026-04-23 | First-paint can be replaced; mutation refresh should preserve current sentences | 362 - | PD11 / all | Keep `logError` with every visible surface | Shipped 2026-04-23 | Telemetry and owner-facing state serve different purposes | 363 - 364 - ## As-implemented deviations 365 - 366 - - Bundle 4: `saveField()` uses a local `saveConfigValue()` wrapper and passes `fetchArgs` as a function so `saveControl()` treats `200 { success: false, error: ... }` config saves as failures instead of false positives. 367 - - Bundle 4: the request body stays on the live route contract, `{ section, data: { [key]: value } }`, rather than the speculative `{ section, key, value, runtime_env }` shape from the early draft. 368 - - Bundle 7: first-paint vs refresh detection keys off the actual rendered `.spk-sentence` rows plus the literal `Loading...` placeholder; the design draft’s `.spk-sentence-row` selector does not exist in the template. 369 - - Bundle 7: `loadReview()` still converts legacy `200 { error: ... }` payloads into thrown errors so the same first-paint / refresh error surfaces handle both transport failures and envelope failures. 370 - - Bundle 8: home adds a local `clearPulseRefreshError()` helper because `SurfaceState.replaceLoading()` appends refresh-error siblings but does not remove them after a later successful refresh. 371 - - Bundle 11: no behavioral deviation from the approved design; the importer retry handler kept the existing delegated click-handler shape. 372 - - Bundle 12: chat uses a local `<style>` block in `apps/chat/workspace.html` for the stalled-card visuals and normalizes the server-rendered finished-card `data-talent-status` in `apps/chat/_chat_event.html` so the live and initial DOM states match. 373 - 374 - ## Out-of-scope follow-ups 375 - 376 - - D9 Sol updated-days banner. Backend must return a detectable error envelope or non-2xx status before the client can migrate. 377 - - Chat owner-reply watchdog. `owner_message` has no `use_id`, so this needs a separate correlation design. 378 - - Any `convey/static/api.js` primitive expansion. Wave 2 must stay within existing primitives. 379 - - Shared helper consolidation across app files. Reuse local helpers now; consolidate later if the pattern repeats again. 380 - 381 - ## Open questions needing Jer's confirmation (flag these at gate time) 382 - 383 - 1. D5 env-field UX (resolved by senior): env fields bypass `saveControl()`. Use `apiJson` + `prepareFieldErrorHost` directly. Clear on success; preserve the typed secret on failure so the owner can retry without re-typing. See revised D5 env-key section. 384 - 2. D4 timeout constant: 2 minutes is the proposed UI stall threshold for entity assist and description generation. Owner-facing threshold, not a backend SLA. Senior approves; flag to Jer at gate for visibility only.

-127

scratch/design-convey-error-wave3.md

··· 1 - # Convey error handling - Wave 3 design 2 - 3 - ## Overview 4 - 5 - Wave 3 closes the remaining Tier 3 shell, onboarding, and background-task error-handling gaps across 10 sites: `init.html` observers/finalize, shell background registration, chat-bar hydrate, month stats shell, todos background badge/nudges, support background verification, support ticket detail, and support announcements. 6 - 7 - This wave is mostly shell-template work. Reuse the Wave 0 primitives, add one tiny `AppServices` helper for non-task background failures, and keep the support badge backend contract explicitly out of scope. 8 - 9 - ## Conventions carried forward 10 - 11 - - Keep `window.logError(err, { context: ... })` alongside every owner-visible failure surface. Logging and UI are separate requirements. 12 - - Use `err.serverMessage` as the primary owner copy when present; fall back to the explicit strings below. 13 - - Do not add retry buttons. Reload is the recovery path unless the site already has its own action button. 14 - - Reuse Wave 0 primitives in place: `window.apiJson`, `window.SurfaceState.errorCard`, `window.SurfaceState.replaceLoading`, and `AppServices.registerTask`. 15 - - UI timeouts and background-task failure thresholds are owner-patience signals, not backend SLAs. 16 - 17 - ## Decision log 18 - 19 - | Decision | Chosen | Rationale | Rejected alternatives | Final copy | 20 - | --- | --- | --- | --- | --- | 21 - | D-init | Shipped 2026-04-23. Add `<script src="error-handler.js">` and `<script src="api.js">` to `convey/templates/init.html`; do not load `app.js` or `websocket.js`; add local `.error-message` styling in `init.html` because onboarding does not load `app.css`. Verified: `convey/static/api.js` is standalone and `convey/static/error-handler.js` is safe without shell DOM or `appEvents`. | Smallest DRY path. Avoids micro-inlining shared helpers into onboarding. | Inline `apiJson`/`logError`; add `app.js`; add `websocket.js`; extract a shared partial just for two scripts. | None. | 22 - | D-finalize | Shipped 2026-04-23. Move `finalize()` to `window.apiJson('/init/finalize', ...)`; add `#finalize-error` next to the CTA; log with context `init-finalize`. Keep the password field-local `showFieldStatus(...)` path for the current backend 400 password validation case and do not double-surface that error in the new slot. | `apiJson` fixes non-2xx and malformed JSON; the new slot fixes the current "button does nothing" failure. Current backend only returns 400 for password length, so field-local handling remains precise. | Password-only surfacing; silent catch; redirect on failure; modal/global error. | Password field on 400: current server string `Password must be at least 8 characters` via `err.serverMessage`. Finalize slot fallback: `Couldn't finalize setup. Check your connection and try again.` | 23 - | D-observers | Shipped 2026-04-23. Add `#observer-error` as a separate node from `#observer-empty`; switch `loadObservers()` to `window.apiJson('/init/observers')`; hide `#observer-error` on success and hide `#observer-empty` while an error is showing; log with context `init-observers`. | Empty state and error state must stop sharing the same element. `apiJson` gives one transport contract. | Reuse `#observer-empty`; inline `fetch(...).json()`; silent keep-empty behavior. | Primary: `err.serverMessage`. Fallback: `Couldn't check for observers - reload to try again.` | 24 - | D-bg-register | Shipped 2026-04-23. Add `AppServices.markBackgroundFailing(appName, error)` beside `registerTask()` / `getTaskHealth()` in `convey/static/app.js`. `convey/templates/app.html` background catch calls that helper plus `window.logError(err, { context: 'app-bg-register', app: '{{ app_name }}' })`. Helper only adds `.menu-item-bg-failing` to `.menu-item[data-app-name="<app>"]` and no-ops if missing. | Centralizes shell-side failing-pip behavior for background scripts that fail before they can register a task. Keeps the app loop running. | Direct DOM mutation inside template catch; fake `registerTask()` records; popup notifications for registration errors. | None - pip only. | 25 - | D-chat-hydrate | Shipped 2026-04-23. On hydrate failure, call `setPendingState(true)`, `setStatus("Couldn't load recent chat session. Reload to try again.", "Couldn't load recent chat session. Reload to try again.")`, and `window.logError(err, { context: 'chat-hydrate' })`. Verified: `setStatus()` only accepts `(text, title)` and has no error variant. Do not add a new chat-bar state. | Reuses the existing disabled affordance on `#chatBarInput` / `#chatBarSend` and keeps the change local to `convey/templates/app.html`. | Invent a chat-bar error variant/class; leave the bar interactive; blank status on error. | `Couldn't load recent chat session. Reload to try again.` | 26 - | D-month-shell | Shipped 2026-04-23. Use a normalized provider contract: `date_nav.html` providers return `{ data, error }`, with `window.apiJson(...)` inside the provider. `convey/static/month-picker.js` caches `{ data, error, facet }`, preserves stale `data` on same-month/same-facet refetch failure, logs with context `month-stats`, adds a warning glyph on `#date-nav-label` via CSS state when `error` is present, and renders a small inline dropdown error above the grid only when the current month has no cached data for the active facet. Clear glyph/title/inline error on next success. | Current picker has no internal header to tint and must remain non-blocking. A normalized `{ data, error }` contract keeps the provider and picker responsibilities explicit. | Block with `SurfaceState.errorCard`; add a new header row inside the picker; keep collapsing failures to empty months. | Label tooltip: `err.serverMessage || "Couldn't load month stats."` Inline first-open copy: `Couldn't load month stats.` plus `err.serverMessage` as secondary text. | 27 - | D-todos-bg | Shipped 2026-04-23. Migrate `updateBadge()` to `AppServices.registerTask('todos', 'update-badge', { intervalMs: 5 * 60 * 1000, run, onSuccess })`. Migrate `checkNudges()` to `AppServices.registerTask('todos', 'check-nudges', { run })` with no `intervalMs`; verified `registerTask()` still performs the initial run when `intervalMs` is absent. Both task `run` functions use the task-scoped `apiJson`; validate `count` and `nudges` shape before mutating badge state or scheduling timers. Keep `_nudgeTimers` dedupe exactly as-is. | Matches support for badge cadence, fixes init-only dark failures, and avoids adding nudge re-fetch dedupe state. The current `registerTask()` contract already supports init-only work cleanly. | Poll nudges; fake init-only with a 24h interval; keep raw `fetch(...)`; add new background-task framework. | No new site-specific copy. Shared `registerTask()` failure notification remains `todos background task failing` with the thrown message. | 28 - | D-support-bg | Shipped 2026-04-23. No client change. Keep `apps/support/background.html` as-is and record that Wave 0 already migrated it to `registerTask(intervalMs=5m)`. Flag the backend contract gap instead. | The client already uses the intended primitive. The missing behavior is server-side: the badge route never returns failure or 403 to the task. | Client-side heuristics on `count === 0`; server fix in this wave. | None. | 29 - | D-open-ticket | Shipped 2026-04-23. Move `openTicket()` to `window.apiJson('/app/support/api/tickets/' + id)`; log with context `support-open-ticket`; on failure render a `support-empty` card into `#ticket-detail` that matches `loadTickets()`'s support-local error shape, but keep the existing back affordance/button. | `loadTickets()` is already the gold-standard pattern in this workspace. Reuse the support-local card instead of mixing in a second visual language. | Keep raw `fetch()` and sparse-ticket rendering; switch to a generic `SurfaceState` card; hide the detail pane on error. | Heading fallback: `Couldn't load ticket.` Hint: `Go back and select it again.` Button: `Back to tickets`. | 30 - | D-announcements | Shipped 2026-04-23. Add local `announcementsFirstPaintDone` and `announcementsLastSuccessAt` state in `apps/support/workspace.html`. Move `loadAnnouncements()` to `window.apiJson(...)`; on first-paint failure render inline error copy in the banner slot and log with context `support-announcements`; on later failure after a successful load, preserve the prior banner content and append a singleton stale-indicator sibling with timestamp text. On success clear stale UI and set the flags. Note: current HEAD only calls `loadAnnouncements()` once, so the stale branch is forward-compatible and does not add a new refresh trigger. | Stops `!ok` from disappearing silently and preserves stale content if the function is ever re-run. The extra flags are the minimum state needed for the split surface. | Silent `!ok` return; always replace banner on failure; add a retry button or periodic polling. | First paint: `Couldn't load announcements. Reload to try again.` Refresh stale: `Couldn't refresh announcements - showing last known state.` Timestamp suffix: `Last updated {local time}.` | 31 - 32 - ## Implementation order 33 - 34 - ### Bundle 1 35 - 36 - Target commit message: `convey: load api helpers in init shell` 37 - 38 - | File | Change | 39 - | --- | --- | 40 - | `convey/templates/init.html` | Add `error-handler.js` and `api.js` includes; add local `.error-message` styles; add `#observer-error` and `#finalize-error` slots. | 41 - 42 - ### Bundle 2 43 - 44 - Target commit message: `convey: surface init observer and finalize failures` 45 - 46 - | File | Change | 47 - | --- | --- | 48 - | `convey/templates/init.html` | Migrate `loadObservers()` and `finalize()` to `window.apiJson`; wire the new slots; keep password validation field-local; add `window.logError` calls. | 49 - 50 - ### Bundle 3 51 - 52 - Target commit message: `convey: mark shell background and chat hydrate failures` 53 - 54 - | File | Change | 55 - | --- | --- | 56 - | `convey/static/app.js` | Add `AppServices.markBackgroundFailing(appName, error)` beside `registerTask()` / `getTaskHealth()`. | 57 - | `convey/templates/app.html` | Use the new helper in the per-app background catch; migrate chat hydrate failure to disabled input + status copy + `logError`. | 58 - 59 - ### Bundle 4 60 - 61 - Target commit message: `convey: signal month stats failures in date nav` 62 - 63 - | File | Change | 64 - | --- | --- | 65 - | `convey/templates/date_nav.html` | Change provider to `window.apiJson(...)` and return `{ data, error }`. | 66 - | `convey/static/month-picker.js` | Normalize/cache `{ data, error, facet }`; preserve stale data on failure; render inline picker error and warning state; log failures. | 67 - | `convey/static/app.css` | Add label warning-glyph state and small picker-error styles. | 68 - 69 - ### Bundle 5 70 - 71 - Target commit message: `todos: migrate badge and nudge background fetches` 72 - 73 - | File | Change | 74 - | --- | --- | 75 - | `apps/todos/background.html` | Wrap badge polling and nudge scheduling in `AppServices.registerTask`; keep badge polling at 5 minutes; keep nudges init-only; add light shape validation. | 76 - 77 - ### Bundle 6 78 - 79 - Target commit message: `support: harden ticket detail loads` 80 - 81 - | File | Change | 82 - | --- | --- | 83 - | `apps/support/workspace.html` | Move `openTicket()` to `window.apiJson`; render support-local error card with back affordance; add `window.logError`. | 84 - | `apps/support/background.html` | Verify only - no edit expected. | 85 - | `apps/support/routes.py` | Reference only - follow-up, no edit in Wave 3. | 86 - 87 - ### Bundle 7 88 - 89 - Target commit message: `support: split announcements first-paint and stale errors` 90 - 91 - | File | Change | 92 - | --- | --- | 93 - | `apps/support/workspace.html` | Add `announcementsFirstPaintDone` / `announcementsLastSuccessAt`; move `loadAnnouncements()` to `window.apiJson`; split first-paint vs stale refresh surfaces; add `window.logError`. | 94 - 95 - ## Validation plan 96 - 97 - - Run `make ci`. 98 - - Run `make test`. 99 - - Grep sweeps after implementation: 100 - - confirm the migrated sites no longer use raw `fetch(...).json()` at `init observers/finalize`, chat hydrate, ticket detail, announcements, and todos background. 101 - - confirm `AppServices.markBackgroundFailing` is the only template-side path for non-task background registration failure. 102 - - confirm month-picker code now references the normalized `{ data, error }` contract and the warning-state selector. 103 - - Screenshot plan: 104 - - capture `/init` with forced observers failure and forced finalize failure to verify the new inline slots do not collide with `#observer-empty` or the password field. 105 - - capture one date-nav app with month-picker first-open failure and one stale-month failure to verify the label glyph plus inline picker error. 106 - - capture `/app/support` with ticket-detail failure and with announcements first-paint failure / stale indicator. 107 - - Manual smoke: 108 - - open `convey/static/tests/register-task.html` to confirm the shared background-task behavior still passes after adding `markBackgroundFailing`. 109 - 110 - ## Out-of-scope follow-ups 111 - 112 - 1. `apps/support/routes.py:256-272` collapses errors to `200 {"count": 0}`. Server contract change needed so the support `registerTask` sees real failures / 403-disabled. 113 - 2. `init.html` gains `logError` calls but onboarding has no shell-level `#error-log` sink. Inline slots are the owner-visible surface; logging is telemetry-only at init. 114 - 3. Wave 2 D9 - `apps/sol/workspace.html:1371` `sol updated-days` deferred because the server collapses failure to `[]`. 115 - 116 - ### Also noted during audit 117 - 118 - - `apps/activities/_day.html:953` - Wave 1 activity loader missing `window.logError`. 119 - - `apps/settings/workspace.html:3564` - pre-existing undocumented empty catch after `saveControl`; `onError` at `:3562` logs, but the suppressing catch is undocumented. 120 - 121 - ## Audit evidence 122 - 123 - - Grep sweeps passed after audit fixup: migrated Wave 3 sites no longer have raw `fetch(...).json()` at the migrated lines; recursive empty-catch sweep is empty; no `console.error` remains in the touched Wave 3 areas; all expected `logError` contexts are present (`init-observers`, `init-finalize`, `app-bg-register`, `chat-hydrate`, `month-stats`, `support-open-ticket`, `support-announcements`). 124 - - `make test`: `3915 passed, 5 skipped, 1 warning`. 125 - - `make test-app APP=todos`: `85 passed`; `apps/support/tests/` absent. 126 - - `make verify-browser`: not run because `tests/verify_browser.py` does not cover the Wave 3 failure paths. 127 - - Screenshots: not captured; direct `sol screenshot` required a running stack, and the sandbox reported ready but exited before screenshots could connect.

-95

scratch/entity-observer-prototype/results/tier2-flash-minimal-jsonl-nothink.json

··· 1 - { 2 - "label": "tier2-flash-minimal-jsonl-nothink", 3 - "config": { 4 - "facet": "solstone", 5 - "day": "20260414", 6 - "strategy": "minimal", 7 - "context_style": "compact", 8 - "system_key": "observer_v2_terse", 9 - "format_key": "jsonl", 10 - "model": "gemini-3-flash-preview", 11 - "thinking_budget": 0 12 - }, 13 - "context_stats": { 14 - "total_attached": 129, 15 - "active_count": 39, 16 - "estimated_tokens": 12909, 17 - "system_prompt_chars": 337, 18 - "user_prompt_chars": 46689 19 - }, 20 - "generation": { 21 - "elapsed_seconds": 5.49, 22 - "usage": { 23 - "prompt_tokens": 11439, 24 - "output_tokens": 647, 25 - "total_tokens": 12086, 26 - "thinking_tokens": null 27 - }, 28 - "output_chars": 2923, 29 - "finish_reason": "FinishReason.STOP" 30 - }, 31 - "parsing": { 32 - "parse_success": true, 33 - "observation_count": 8, 34 - "entity_count": 7, 35 - "errors": [] 36 - }, 37 - "evaluation": { 38 - "total_observations": 8, 39 - "entities_with_observations": 7, 40 - "duplicates": 0, 41 - "quality_flags": [], 42 - "quality_score": 1.0 43 - }, 44 - "raw_output": "{\"entity_id\": \"jeremie_miller\", \"entity_name\": \"Jeremie Miller\", \"content\": \"Advocates for a 'tenure-weighted customer standing' model in corporate governance, where long-term customers gain formal standing in covenant enforcement decisions.\", \"reasoning\": \"Jeremie explicitly proposed this concept for Sol PBC to align customer incentives with mission preservation.\"}\n{\"entity_id\": \"sol_pbc\", \"entity_name\": \"Sol PBC\", \"content\": \"Adopts a 'cessation-over-violation' principle, where the entity is structured to shut down rather than violate its core privacy and mission covenants.\", \"reasoning\": \"This governance failure mode was identified as a key architectural constraint for the PBC's legal framework.\"}\n{\"entity_id\": \"solstone\", \"entity_name\": \"Solstone\", \"content\": \"Utilizes a 'status-based routing' architecture for user onboarding, directing users to specific post-login states such as 'applied', 'approved', or 'revoked' based on their Scout portal status.\", \"reasoning\": \"This was documented as the core logic for the Scouts application and user entry flow.\"}\n{\"entity_id\": \"solstone\", \"entity_name\": \"Solstone\", \"content\": \"Architecture for data merging includes a 'staged review' workflow where high-confidence matches are auto-merged while ambiguous data is held in a 'staged_entities' directory for human-in-the-loop audit.\", \"reasoning\": \"Jeremie designed this 'sol transfer merge' architecture to handle the complexity of merging distributed journals.\"}\n{\"entity_id\": \"glenn_otis_brown\", \"entity_name\": \"Glenn Otis Brown\", \"content\": \"Acts as a strategic bridge between Solstone and the Electronic Frontier Foundation (EFF) community, specifically facilitating introductions to leadership like Cindy Cohn.\", \"reasoning\": \"Glenn's role shifted from general advisor to a specific relationship bridge for EFF/legal outreach.\"}\n{\"entity_id\": \"extro\", \"entity_name\": \"Extro\", \"content\": \"Employs a 'draft-approve-send' separation protocol for external communications to ensure an audit trail and maintain a consistent organizational voice.\", \"reasoning\": \"This protocol was codified in the CEO weekly reflection as a necessary governance fix for the organization.\"}\n{\"entity_id\": \"soulstone_observer\", \"entity_name\": \"Soulstone Observer\", \"content\": \"Requires local GUI trust authorization and full Xcode installation on macOS to facilitate self-signed code signing for local data capture services.\", \"reasoning\": \"Technical discovery during onboarding confirmed that headless or custom-keychain workarounds are insufficient for macOS security requirements.\"}\n{\"entity_id\": \"plaud\", \"entity_name\": \"Plaud\", \"content\": \"Lacks a public API for direct integration, requiring manual extraction of Cloud API tokens via browser developer tools to sync recordings into external systems like Solstone.\", \"reasoning\": \"Jeremie documented this specific workaround during a live onboarding and troubleshooting session.\"}", 45 - "observations": [ 46 - { 47 - "entity_id": "jeremie_miller", 48 - "entity_name": "Jeremie Miller", 49 - "content": "Advocates for a 'tenure-weighted customer standing' model in corporate governance, where long-term customers gain formal standing in covenant enforcement decisions.", 50 - "reasoning": "Jeremie explicitly proposed this concept for Sol PBC to align customer incentives with mission preservation." 51 - }, 52 - { 53 - "entity_id": "sol_pbc", 54 - "entity_name": "Sol PBC", 55 - "content": "Adopts a 'cessation-over-violation' principle, where the entity is structured to shut down rather than violate its core privacy and mission covenants.", 56 - "reasoning": "This governance failure mode was identified as a key architectural constraint for the PBC's legal framework." 57 - }, 58 - { 59 - "entity_id": "solstone", 60 - "entity_name": "Solstone", 61 - "content": "Utilizes a 'status-based routing' architecture for user onboarding, directing users to specific post-login states such as 'applied', 'approved', or 'revoked' based on their Scout portal status.", 62 - "reasoning": "This was documented as the core logic for the Scouts application and user entry flow." 63 - }, 64 - { 65 - "entity_id": "solstone", 66 - "entity_name": "Solstone", 67 - "content": "Architecture for data merging includes a 'staged review' workflow where high-confidence matches are auto-merged while ambiguous data is held in a 'staged_entities' directory for human-in-the-loop audit.", 68 - "reasoning": "Jeremie designed this 'sol transfer merge' architecture to handle the complexity of merging distributed journals." 69 - }, 70 - { 71 - "entity_id": "glenn_otis_brown", 72 - "entity_name": "Glenn Otis Brown", 73 - "content": "Acts as a strategic bridge between Solstone and the Electronic Frontier Foundation (EFF) community, specifically facilitating introductions to leadership like Cindy Cohn.", 74 - "reasoning": "Glenn's role shifted from general advisor to a specific relationship bridge for EFF/legal outreach." 75 - }, 76 - { 77 - "entity_id": "extro", 78 - "entity_name": "Extro", 79 - "content": "Employs a 'draft-approve-send' separation protocol for external communications to ensure an audit trail and maintain a consistent organizational voice.", 80 - "reasoning": "This protocol was codified in the CEO weekly reflection as a necessary governance fix for the organization." 81 - }, 82 - { 83 - "entity_id": "soulstone_observer", 84 - "entity_name": "Soulstone Observer", 85 - "content": "Requires local GUI trust authorization and full Xcode installation on macOS to facilitate self-signed code signing for local data capture services.", 86 - "reasoning": "Technical discovery during onboarding confirmed that headless or custom-keychain workarounds are insufficient for macOS security requirements." 87 - }, 88 - { 89 - "entity_id": "plaud", 90 - "entity_name": "Plaud", 91 - "content": "Lacks a public API for direct integration, requiring manual extraction of Cloud API tokens via browser developer tools to sync recordings into external systems like Solstone.", 92 - "reasoning": "Jeremie documented this specific workaround during a live onboarding and troubleshooting session." 93 - } 94 - ] 95 - }

-490

scratch/entity-observer-prototype/run_experiment.py

··· 1 - #!/usr/bin/env python3 2 - """Run entity_observer generate prototype experiments. 3 - 4 - Tests different prompt strategies, output formats, and model tiers against 5 - real pre-computed context from the journal. READ-ONLY on journal data; 6 - writes results to scratch/entity-observer-prototype/results/. 7 - 8 - Usage: 9 - cd /home/jer/projects/solstone 10 - python3 scratch/entity-observer-prototype/run_experiment.py \ 11 - --facet solstone --day 20260414 \ 12 - --prompt structured_json \ 13 - --model gemini-2.5-flash-lite \ 14 - --label lite-structured-json 15 - """ 16 - 17 - import argparse 18 - import json 19 - import os 20 - import sys 21 - import time 22 - from pathlib import Path 23 - 24 - sys.path.insert(0, str(Path(__file__).resolve().parents[2])) 25 - os.environ.setdefault("SOL_JOURNAL", str(Path(__file__).resolve().parents[2] / "journal")) 26 - 27 - from assemble_context import assemble_full_context, format_prompt_context 28 - 29 - RESULTS_DIR = Path(__file__).parent / "results" 30 - 31 - # --- Prompt templates --- 32 - 33 - SYSTEM_PROMPTS = { 34 - "observer_v1": """You are an entity observation agent for a personal knowledge system called Solstone. 35 - Your task: extract durable factoids about entities from today's journal content. 36 - 37 - An observation is a lasting fact about WHO or WHAT an entity IS — not what happened today. 38 - 39 - Good observations: 40 - - "Advocates for Socratic questioning in mentorship" 41 - - "Based in Seattle, previously worked at Google" 42 - - "Prefers async communication over meetings" 43 - 44 - NOT observations (these are activity logs): 45 - - "Discussed migration today" 46 - - "Sent contract for review" 47 - - "Uses v2.1.50" (expires) 48 - 49 - Rules: 50 - 1. One fact per observation — no compound sentences 51 - 2. Must pass BOTH litmus tests: 52 - a) "Would this be true and useful 6 months from now?" 53 - b) "Would this help someone who's never met this entity?" 54 - 3. Check for semantic duplicates against existing observations 55 - 4. If existing observations are already rich, restraint is correct — zero new observations is valid 56 - 5. Skip entities where today's content reveals nothing durable""", 57 - 58 - "observer_v2_terse": """Entity observation agent. Extract durable factoids from today's journal content. 59 - 60 - Observation = lasting fact about WHO/WHAT an entity IS. NOT activity logs, NOT ephemeral state. 61 - 62 - Litmus: (1) true in 6 months? (2) useful to a stranger? Both must be yes. 63 - One fact per observation. No duplicates of existing observations. Zero new is valid.""", 64 - } 65 - 66 - OUTPUT_FORMAT_INSTRUCTIONS = { 67 - "json_array": """Output format: JSON array of observation objects. 68 - ```json 69 - [ 70 - { 71 - "entity_id": "entity_slug", 72 - "entity_name": "Full Name", 73 - "content": "The durable observation text", 74 - "reasoning": "Why this qualifies as a durable observation (1 sentence)" 75 - } 76 - ] 77 - ``` 78 - Output ONLY the JSON array. No markdown, no commentary.""", 79 - 80 - "jsonl": """Output format: one JSON object per line (JSONL), no wrapping array. 81 - {"entity_id": "entity_slug", "entity_name": "Full Name", "content": "The observation", "reasoning": "Why"} 82 - {"entity_id": "other_entity", "entity_name": "Other Name", "content": "Another observation", "reasoning": "Why"} 83 - 84 - Output ONLY the JSONL lines. No markdown, no commentary, no blank lines between entries.""", 85 - 86 - "markdown_structured": """Output format: Markdown with one section per entity that has new observations. 87 - 88 - ## Entity Name (entity_id) 89 - - **Observation:** The durable factoid 90 - - **Reasoning:** Why this is durable (1 sentence) 91 - 92 - ## Another Entity (another_id) 93 - - **Observation:** Another factoid 94 - - **Reasoning:** Why 95 - 96 - Skip entities with no new observations entirely. End with a summary line: 97 - "Observed X entities, Y new observations total." 98 - """, 99 - 100 - "json_grouped": """Output format: JSON object grouped by entity_id. 101 - ```json 102 - { 103 - "observations": { 104 - "entity_slug": [ 105 - {"content": "The observation", "reasoning": "Why"} 106 - ], 107 - "other_entity": [ 108 - {"content": "Another observation", "reasoning": "Why"} 109 - ] 110 - }, 111 - "skipped": ["entity_ids_with_no_new_observations"], 112 - "summary": "Observed X entities, Y new observations total." 113 - } 114 - ``` 115 - Output ONLY the JSON. No markdown wrapping.""", 116 - } 117 - 118 - 119 - def build_prompt( 120 - context: dict, 121 - *, 122 - system_key: str = "observer_v1", 123 - format_key: str = "json_array", 124 - context_style: str = "structured", 125 - ) -> tuple[str, str]: 126 - """Build system + user prompt for an experiment. 127 - 128 - Returns (system_prompt, user_prompt). 129 - """ 130 - system = SYSTEM_PROMPTS[system_key] 131 - format_inst = OUTPUT_FORMAT_INSTRUCTIONS[format_key] 132 - 133 - # Build user prompt with pre-computed context 134 - context_text = format_prompt_context(context, style=context_style) 135 - 136 - user_prompt = f"""{format_inst} 137 - 138 - --- 139 - 140 - {context_text}""" 141 - 142 - return system, user_prompt 143 - 144 - 145 - def call_gemini( 146 - system_prompt: str, 147 - user_prompt: str, 148 - model: str, 149 - *, 150 - max_output_tokens: int = 8192, 151 - temperature: float = 0.3, 152 - thinking_budget: int | None = None, 153 - ) -> dict: 154 - """Call Gemini API and return result dict.""" 155 - from google import genai 156 - from google.genai import types 157 - 158 - api_key = os.environ.get("GOOGLE_API_KEY") 159 - if not api_key: 160 - # Try loading from vault 161 - vault_path = Path(__file__).resolve().parents[2].parent / "extro" / "cso" / "vault" / "api-keys" / "google-ai-studio.json" 162 - if vault_path.exists(): 163 - vault_data = json.loads(vault_path.read_text()) 164 - api_key = vault_data.get("api_key", "") 165 - if api_key: 166 - os.environ["GOOGLE_API_KEY"] = api_key 167 - 168 - if not api_key: 169 - raise RuntimeError("No GOOGLE_API_KEY found") 170 - 171 - client = genai.Client(api_key=api_key) 172 - 173 - config_kwargs = { 174 - "max_output_tokens": max_output_tokens, 175 - "temperature": temperature, 176 - } 177 - if thinking_budget is not None: 178 - config_kwargs["thinking_config"] = types.ThinkingConfig( 179 - thinking_budget=thinking_budget 180 - ) 181 - 182 - config = types.GenerateContentConfig( 183 - system_instruction=system_prompt, 184 - **config_kwargs, 185 - ) 186 - 187 - start = time.time() 188 - response = client.models.generate_content( 189 - model=model, 190 - contents=user_prompt, 191 - config=config, 192 - ) 193 - elapsed = time.time() - start 194 - 195 - # Extract usage 196 - usage = {} 197 - if response.usage_metadata: 198 - um = response.usage_metadata 199 - usage = { 200 - "prompt_tokens": getattr(um, "prompt_token_count", 0), 201 - "output_tokens": getattr(um, "candidates_token_count", 0), 202 - "total_tokens": getattr(um, "total_token_count", 0), 203 - "thinking_tokens": getattr(um, "thoughts_token_count", 0), 204 - } 205 - 206 - return { 207 - "text": response.text or "", 208 - "usage": usage, 209 - "elapsed_seconds": round(elapsed, 2), 210 - "model": model, 211 - "finish_reason": str(getattr(response.candidates[0], "finish_reason", "")) if response.candidates else "", 212 - } 213 - 214 - 215 - def parse_output(text: str, format_key: str) -> dict: 216 - """Attempt to parse the model output and validate structure.""" 217 - result = { 218 - "raw_text": text, 219 - "parse_success": False, 220 - "observation_count": 0, 221 - "entity_count": 0, 222 - "observations": [], 223 - "errors": [], 224 - } 225 - 226 - # Strip markdown code fences if present 227 - cleaned = text.strip() 228 - if cleaned.startswith("```"): 229 - lines = cleaned.split("\n") 230 - # Remove first and last lines (fences) 231 - lines = lines[1:] 232 - if lines and lines[-1].strip() == "```": 233 - lines = lines[:-1] 234 - cleaned = "\n".join(lines).strip() 235 - 236 - try: 237 - if format_key == "json_array": 238 - observations = json.loads(cleaned) 239 - if isinstance(observations, list): 240 - result["parse_success"] = True 241 - result["observations"] = observations 242 - result["observation_count"] = len(observations) 243 - result["entity_count"] = len(set(o.get("entity_id", "") for o in observations)) 244 - 245 - elif format_key == "jsonl": 246 - observations = [] 247 - parse_errors = 0 248 - for line in cleaned.split("\n"): 249 - line = line.strip() 250 - if not line: 251 - continue 252 - try: 253 - observations.append(json.loads(line)) 254 - except json.JSONDecodeError: 255 - parse_errors += 1 256 - result["parse_success"] = len(observations) > 0 257 - result["observations"] = observations 258 - result["observation_count"] = len(observations) 259 - result["entity_count"] = len(set(o.get("entity_id", "") for o in observations)) 260 - if parse_errors: 261 - result["errors"].append(f"{parse_errors} lines failed to parse (likely truncation)") 262 - 263 - elif format_key == "json_grouped": 264 - data = json.loads(cleaned) 265 - if isinstance(data, dict) and "observations" in data: 266 - result["parse_success"] = True 267 - all_obs = [] 268 - for entity_id, obs_list in data["observations"].items(): 269 - for obs in obs_list: 270 - all_obs.append({"entity_id": entity_id, **obs}) 271 - result["observations"] = all_obs 272 - result["observation_count"] = len(all_obs) 273 - result["entity_count"] = len(data["observations"]) 274 - result["skipped_count"] = len(data.get("skipped", [])) 275 - 276 - elif format_key == "markdown_structured": 277 - # Count ## headers and **Observation:** lines 278 - import re 279 - entities = re.findall(r"^## (.+?)(?:\s*\(|$)", cleaned, re.MULTILINE) 280 - observations = re.findall(r"\*\*Observation:\*\*\s*(.+)", cleaned) 281 - result["parse_success"] = len(observations) > 0 or "0 new observations" in cleaned.lower() 282 - result["observation_count"] = len(observations) 283 - result["entity_count"] = len(entities) 284 - result["observations"] = [ 285 - {"content": o, "entity_name": entities[i] if i < len(entities) else "?"} 286 - for i, o in enumerate(observations) 287 - ] 288 - 289 - except (json.JSONDecodeError, KeyError, TypeError) as e: 290 - result["errors"].append(str(e)) 291 - 292 - return result 293 - 294 - 295 - def evaluate_observations(parsed: dict, context: dict) -> dict: 296 - """Evaluate observation quality against the pre-computed context.""" 297 - eval_result = { 298 - "total_observations": parsed["observation_count"], 299 - "entities_with_observations": parsed["entity_count"], 300 - "duplicates": 0, 301 - "quality_flags": [], 302 - } 303 - 304 - # Build existing observation index 305 - existing_obs = {} 306 - for ec in context["entity_contexts"]: 307 - existing_obs[ec["id"]] = set( 308 - o.lower().strip() for o in ec.get("observations", ec.get("recent_observations", [])) 309 - ) 310 - 311 - # Check each observation 312 - for obs in parsed.get("observations", []): 313 - entity_id = obs.get("entity_id", "") 314 - content = obs.get("content", "") 315 - content_lower = content.lower().strip() 316 - 317 - # Check for exact duplicates 318 - if entity_id in existing_obs: 319 - for existing in existing_obs[entity_id]: 320 - if content_lower == existing or content_lower in existing or existing in content_lower: 321 - eval_result["duplicates"] += 1 322 - eval_result["quality_flags"].append( 323 - f"DUPLICATE: {entity_id}: '{content[:60]}...'" 324 - ) 325 - break 326 - 327 - # Check for temporal language (not durable) 328 - temporal_markers = ["today", "currently", "as of", "this week", "yesterday", "right now"] 329 - for marker in temporal_markers: 330 - if marker in content_lower: 331 - eval_result["quality_flags"].append( 332 - f"TEMPORAL: {entity_id}: '{content[:60]}...' (contains '{marker}')" 333 - ) 334 - break 335 - 336 - # Check for activity-log patterns 337 - activity_markers = ["discussed", "sent", "reviewed", "filed", "submitted", "scheduled"] 338 - for marker in activity_markers: 339 - if content_lower.startswith(marker): 340 - eval_result["quality_flags"].append( 341 - f"ACTIVITY_LOG: {entity_id}: '{content[:60]}...' (starts with '{marker}')" 342 - ) 343 - break 344 - 345 - eval_result["quality_score"] = max(0, 1.0 - ( 346 - eval_result["duplicates"] * 0.15 + 347 - len(eval_result["quality_flags"]) * 0.05 348 - )) 349 - 350 - return eval_result 351 - 352 - 353 - def run_experiment( 354 - facet: str, 355 - day: str, 356 - *, 357 - strategy: str = "focused", 358 - context_style: str = "structured", 359 - system_key: str = "observer_v1", 360 - format_key: str = "json_array", 361 - model: str = "gemini-2.5-flash-lite", 362 - thinking_budget: int | None = None, 363 - label: str = "", 364 - ) -> dict: 365 - """Run a single experiment and return full results.""" 366 - print(f"\n{'='*60}") 367 - print(f"Experiment: {label or 'unnamed'}") 368 - print(f" Model: {model}") 369 - print(f" Strategy: {strategy}, Style: {context_style}") 370 - print(f" System: {system_key}, Format: {format_key}") 371 - if thinking_budget: 372 - print(f" Thinking budget: {thinking_budget}") 373 - print(f"{'='*60}") 374 - 375 - # Assemble context 376 - print("Assembling context...") 377 - context = assemble_full_context(facet, day, strategy=strategy) 378 - print(f" Active entities: {context['active_count']}, Est tokens: {context['estimated_tokens']:,}") 379 - 380 - # Build prompt 381 - system_prompt, user_prompt = build_prompt( 382 - context, 383 - system_key=system_key, 384 - format_key=format_key, 385 - context_style=context_style, 386 - ) 387 - print(f" System prompt: {len(system_prompt):,} chars") 388 - print(f" User prompt: {len(user_prompt):,} chars") 389 - 390 - # Call model 391 - print(f"Calling {model}...") 392 - gen_result = call_gemini( 393 - system_prompt, 394 - user_prompt, 395 - model, 396 - thinking_budget=thinking_budget, 397 - ) 398 - print(f" Elapsed: {gen_result['elapsed_seconds']}s") 399 - print(f" Usage: {gen_result['usage']}") 400 - print(f" Output length: {len(gen_result['text']):,} chars") 401 - 402 - # Parse output 403 - print("Parsing output...") 404 - parsed = parse_output(gen_result["text"], format_key) 405 - print(f" Parse success: {parsed['parse_success']}") 406 - print(f" Observations: {parsed['observation_count']}") 407 - print(f" Entities: {parsed['entity_count']}") 408 - if parsed["errors"]: 409 - print(f" Errors: {parsed['errors']}") 410 - 411 - # Evaluate quality 412 - print("Evaluating quality...") 413 - evaluation = evaluate_observations(parsed, context) 414 - print(f" Duplicates: {evaluation['duplicates']}") 415 - print(f" Quality flags: {len(evaluation['quality_flags'])}") 416 - print(f" Quality score: {evaluation['quality_score']:.2f}") 417 - if evaluation["quality_flags"][:5]: 418 - for flag in evaluation["quality_flags"][:5]: 419 - print(f" - {flag}") 420 - 421 - # Compile result 422 - result = { 423 - "label": label, 424 - "config": { 425 - "facet": facet, 426 - "day": day, 427 - "strategy": strategy, 428 - "context_style": context_style, 429 - "system_key": system_key, 430 - "format_key": format_key, 431 - "model": model, 432 - "thinking_budget": thinking_budget, 433 - }, 434 - "context_stats": { 435 - "total_attached": context["total_attached"], 436 - "active_count": context["active_count"], 437 - "estimated_tokens": context["estimated_tokens"], 438 - "system_prompt_chars": len(system_prompt), 439 - "user_prompt_chars": len(user_prompt), 440 - }, 441 - "generation": { 442 - "elapsed_seconds": gen_result["elapsed_seconds"], 443 - "usage": gen_result["usage"], 444 - "output_chars": len(gen_result["text"]), 445 - "finish_reason": gen_result["finish_reason"], 446 - }, 447 - "parsing": { 448 - "parse_success": parsed["parse_success"], 449 - "observation_count": parsed["observation_count"], 450 - "entity_count": parsed["entity_count"], 451 - "errors": parsed["errors"], 452 - }, 453 - "evaluation": evaluation, 454 - "raw_output": gen_result["text"], 455 - "observations": parsed["observations"], 456 - } 457 - 458 - # Save result 459 - RESULTS_DIR.mkdir(parents=True, exist_ok=True) 460 - result_file = RESULTS_DIR / f"{label or 'unnamed'}.json" 461 - result_file.write_text(json.dumps(result, indent=2, ensure_ascii=False)) 462 - print(f"\nResult saved to {result_file}") 463 - 464 - return result 465 - 466 - 467 - if __name__ == "__main__": 468 - parser = argparse.ArgumentParser() 469 - parser.add_argument("--facet", default="solstone") 470 - parser.add_argument("--day", default="20260414") 471 - parser.add_argument("--strategy", default="focused") 472 - parser.add_argument("--context-style", default="structured") 473 - parser.add_argument("--system", default="observer_v1") 474 - parser.add_argument("--format", default="json_array") 475 - parser.add_argument("--model", default="gemini-2.5-flash-lite") 476 - parser.add_argument("--thinking-budget", type=int, default=None) 477 - parser.add_argument("--label", default="") 478 - args = parser.parse_args() 479 - 480 - run_experiment( 481 - args.facet, 482 - args.day, 483 - strategy=args.strategy, 484 - context_style=args.context_style, 485 - system_key=args.system, 486 - format_key=args.format, 487 - model=args.model, 488 - thinking_budget=args.thinking_budget, 489 - label=args.label, 490 - )

-2

scratch/link-blindness-grep.txt

··· 1 - $ tail -n 2000 scratch/link-blindness-journal/health/supervisor.log scratch/link-blindness-journal/link/*.log 2>/dev/null | grep -iE 'authorization|bearer |cookie|payload|ca private|BEGIN PRIVATE|client_cert|home_attestation' || echo "no matches" 2 - no matches

Configure Feed

Configure Feed