Navigate a directory full of directories, identifying repos and worktrees
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

add Seedling and Flowline phase design docs for planner rollout

Refs: is-tree-scan-priority
Refs: is-tree-fuzzel-pipeline

rektide 87f4857f 7fb578d9

+581 -2
+235
doc/plan/demand-sense-loops.md
··· 1 + # Demand-Driven Sense Loops for `is-tree` 2 + 3 + This document links the `is-tree` scan/picker work to the loop architecture in [`rektide/iso-chill` `doc/design.md`](https://github.com/rektide/iso-chill/blob/main/doc/design.md), especially: 4 + 5 + - [The Loop System](https://github.com/rektide/iso-chill/blob/main/doc/design.md#the-loop-system) 6 + - [Queue](https://github.com/rektide/iso-chill/blob/main/doc/design.md#queue) 7 + - [SignalEvent](https://github.com/rektide/iso-chill/blob/main/doc/design.md#signalevent) 8 + - [Control Loop Design](https://github.com/rektide/iso-chill/blob/main/doc/design.md#control-loop-design) 9 + 10 + Related local docs: 11 + 12 + - [`/doc/pick-iter.md`](/doc/pick-iter.md) 13 + - [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md) 14 + 15 + ## Intent 16 + 17 + Move from one-shot "scan everything, compute everything" execution to a looped model: 18 + 19 + - `scan` is a generator loop that emits candidates quickly. 20 + - downstream loops are consumers with explicit demands. 21 + - sensing pools fulfill those demands at the right cost tier. 22 + 23 + Instead of hardcoding one runtime path, we declare process demands and let a sense planner satisfy them. 24 + 25 + ## Why this model fits 26 + 27 + The `iso-chill` model gives us three useful ideas to borrow directly: 28 + 29 + 1. **Consumer-initiated subscriptions**: producers do not need to know all consumers. 30 + 2. **Shared state + signal events**: events coordinate; state is source of truth. 31 + 3. **Micro-batched loop ticks**: good traceability, bounded work, easier tuning. 32 + 33 + For `is-tree`, this means scan can emit candidate references, while multiple consumers (picker, staleness view, renderer, sync planner) independently subscribe and ask for more detail only when needed. 34 + 35 + ## Core model 36 + 37 + ### Candidate context store 38 + 39 + Maintain a shared per-path context store, analogous to `ProcessContext` in `iso-chill`. 40 + 41 + Each candidate keeps: 42 + 43 + - `directory` identity 44 + - known columns (`status`, `change-date`, `ahead`, ...) 45 + - freshness timestamps per column 46 + - in-flight demand state 47 + 48 + ### Signal events 49 + 50 + Loops emit small events (not full payload copies), such as: 51 + 52 + - `candidate.discovered` 53 + - `candidate.updated(column=change-date)` 54 + - `candidate.ready(demand=picker.fast)` 55 + 56 + Consumers react to events, then read required data from the shared store. 57 + 58 + ### Sense pools 59 + 60 + A sense pool is a loop (or strategy set) that can fill a class of columns. 61 + 62 + Initial pools: 63 + 64 + - **identity pool**: `status`, `directory`, worktree/basic type 65 + - **fs metadata pool**: `change-date`, directory mtime, file-count-lite 66 + - **history pool**: `commit-date` 67 + - **remote state pool**: `ahead` 68 + - **deep file pool**: per-file age/size/modified (future `--files`) 69 + 70 + Pools run on demand, not eagerly for all candidates. 71 + 72 + ## Demand declarations 73 + 74 + Each process declares what it needs as a demand contract. 75 + 76 + Example schema: 77 + 78 + ```rust 79 + struct Demand { 80 + id: String, 81 + columns: Vec<String>, 82 + max_staleness: std::time::Duration, 83 + priority: u8, 84 + max_candidates: Option<usize>, 85 + ordering: Vec<SortKey>, 86 + } 87 + ``` 88 + 89 + Examples: 90 + 91 + - `picker.fast` 92 + - columns: `directory`, `status`, `change-date` 93 + - priority: high 94 + - ordering: `change-date-` 95 + - `render.detail` 96 + - columns: full format projection 97 + - priority: medium 98 + - `stale.files` 99 + - columns: `directory`, per-file age columns 100 + - priority: low 101 + 102 + The planner computes the minimal pool work needed to satisfy active demands. 103 + 104 + ## Loop topology 105 + 106 + ```mermaid 107 + flowchart LR 108 + subgraph ScanLoop[Scan Generator Loop] 109 + enum[enumerate roots] 110 + detect[cheap identity detect] 111 + emit_discovered[emit candidate.discovered] 112 + enum --> detect --> emit_discovered 113 + end 114 + 115 + subgraph Store[CandidateContext Store] 116 + ctx[(candidate state)] 117 + end 118 + 119 + subgraph SensePools[Sense Pools] 120 + p_id[identity pool] 121 + p_fs[fs metadata pool] 122 + p_hist[history pool] 123 + p_remote[remote state pool] 124 + p_deep[deep file pool] 125 + end 126 + 127 + subgraph Consumers[Consumer Loops] 128 + pick[picker loop] 129 + render[render loop] 130 + stale[staleness loop] 131 + sync[reforrest prep loop] 132 + end 133 + 134 + emit_discovered --> ctx 135 + ctx --> p_id 136 + ctx --> p_fs 137 + ctx --> p_hist 138 + ctx --> p_remote 139 + ctx --> p_deep 140 + 141 + p_id --> ctx 142 + p_fs --> ctx 143 + p_hist --> ctx 144 + p_remote --> ctx 145 + p_deep --> ctx 146 + 147 + ctx --> pick 148 + ctx --> render 149 + ctx --> stale 150 + ctx --> sync 151 + ``` 152 + 153 + ## How this changes `--scan` 154 + 155 + `--scan` becomes the primary candidate-generator loop, not a one-off mode. 156 + 157 + - It emits candidates immediately from cheap detection. 158 + - It can apply early ordering (`directory`, `change-date`) when available. 159 + - It does not wait for expensive pools. 160 + 161 + Then downstream consumers trigger deeper sensing for selected candidates only. 162 + 163 + Pipeline intent remains: 164 + 165 + ```bash 166 + is-tree --scan --format directory | fuzzel --dmenu --multi | is-tree --stdin --format all 167 + ``` 168 + 169 + But internally this is not "run two unrelated commands"; it is two demand profiles over the same sensing model: 170 + 171 + - first command asks for `picker.fast` 172 + - second command asks for `render.detail` on a reduced candidate set 173 + 174 + ## Planning rules (demand-first) 175 + 176 + 1. Union active demand columns. 177 + 2. Determine cheapest pool set that can satisfy union. 178 + 3. Execute pools in stage order (cheap to expensive). 179 + 4. Emit events as each demand reaches readiness. 180 + 5. Recompute when demand set changes (new consumer, canceled consumer, narrowed candidate set). 181 + 182 + This generalizes pushdown from static CLI parsing to live loop operation. 183 + 184 + ## Practical example 185 + 186 + ### Step 1: picker demand 187 + 188 + Consumer requests: 189 + 190 + - columns: `directory`, `status`, `change-date` 191 + - ordering: `change-date-` 192 + - target: first 200 candidates quickly 193 + 194 + Planner schedules: 195 + 196 + - scan loop + identity pool + fs metadata pool 197 + - no history/remote/deep pools yet 198 + 199 + ### Step 2: user selects 8 paths 200 + 201 + Renderer requests: 202 + 203 + - columns: `status`, `directory`, `commit-date`, `ahead` 204 + - scope: selected 8 paths 205 + 206 + Planner schedules: 207 + 208 + - history + remote pools, but only for selected 8 209 + 210 + This is where the large performance win comes from. 211 + 212 + ## Minimal implementation path 213 + 214 + 1. Keep current CLI behavior and planner from [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md). 215 + 2. Add a small in-memory candidate store abstraction. 216 + 3. Introduce one event channel and one consumer loop (`picker.fast`). 217 + 4. Split existing probes into pool-like executors with declared column coverage. 218 + 5. Expand to multiple consumer loops and scoped demand recomputation. 219 + 220 + ## Design constraints 221 + 222 + - Preserve Unix composability at the CLI surface. 223 + - Keep deterministic output for non-streaming commands. 224 + - Make pool scheduling observable (debug logs per demand and pool run). 225 + - Avoid hidden expensive upgrades unless policy explicitly allows it. 226 + 227 + ## Decision 228 + 229 + Adopt a demand-driven sense-loop architecture: 230 + 231 + - scan loop generates candidates 232 + - consumers declare demands 233 + - sense pools fulfill demands by cost tier 234 + 235 + This ties `is-tree` directly to proven `iso-chill` loop patterns while staying focused on repository candidate generation and selective deep inspection.
+146
doc/plan/flowline-phase-2-staged-runtime.md
··· 1 + # Flowline: Phase 2 Staged Runtime Design 2 + 3 + Plan name: **Flowline**. 4 + 5 + Flowline is Phase 2 of the planner/pushdown roadmap from [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md). It consumes Seedling planner outputs and executes queries in staged runtime order. 6 + 7 + Related references: 8 + 9 + - [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md) 10 + - [`/doc/plan/seedling-phase-1-query-planner.md`](/doc/plan/seedling-phase-1-query-planner.md) 11 + - [`/doc/pick-iter.md`](/doc/pick-iter.md) 12 + - [`/src/main.rs`](/src/main.rs) 13 + - [`/src/plugin.rs`](/src/plugin.rs) 14 + 15 + ## Why this phase exists 16 + 17 + Seedling can tell us what work is needed and when it is available. Flowline makes runtime follow that plan so early columns are computed and emitted before expensive columns. 18 + 19 + This is the phase that turns planning into user-visible speed improvements. 20 + 21 + ## Goals 22 + 23 + - Execute query work in explicit stages. 24 + - Support early streaming when plan allows. 25 + - Apply early filters/sorts before late probes. 26 + - Keep full-mode correctness equivalent to current behavior. 27 + - Route directory-only optimization through planner output, not bespoke branch logic. 28 + 29 + ## Non-goals 30 + 31 + - No plugin metadata API changes yet (still use planner metadata map). 32 + - No per-file recursion implementation yet. 33 + - No long-running daemon behavior. 34 + 35 + ## Runtime stage model 36 + 37 + ```mermaid 38 + flowchart LR 39 + Enumerate[Enumerate paths] --> EarlyProbe[Early probe columns] 40 + EarlyProbe --> EarlyOps[Early filter and sort] 41 + EarlyOps --> LateProbe{Need late probe?} 42 + LateProbe -- No --> Render[Render rows] 43 + LateProbe -- Yes --> LateCollect[Plugin late collection] 44 + LateCollect --> FinalOps[Final filter and sort] 45 + FinalOps --> Render 46 + ``` 47 + 48 + ## Proposed module layout 49 + 50 + Create a runtime domain with stage-specific units: 51 + 52 + - `src/runtime/mod.rs` — execution entrypoint 53 + - `src/runtime/enumerate.rs` — path enumeration and base row init 54 + - `src/runtime/early.rs` — early probe and early predicate/sort application 55 + - `src/runtime/late.rs` — plugin streaming/merge for late columns 56 + - `src/runtime/render.rs` — text/json rendering adapters 57 + 58 + Planner integration: 59 + 60 + - `src/main.rs` builds `LogicalQuery` and `PhysicalPlan`, then calls `runtime::execute(plan, args)`. 61 + 62 + ## Execution contract 63 + 64 + `runtime::execute` consumes: 65 + 66 + - `PhysicalPlan` (from Seedling) 67 + - parsed CLI flags 68 + - plugin registry 69 + 70 + It returns a fully rendered output side effect (stdout/stderr) with the same user-visible semantics as current runtime. 71 + 72 + ## Stage behavior details 73 + 74 + ### Stage 1: Enumerate 75 + 76 + - Build candidate path list from positional args or `--all` roots. 77 + - Initialize row records with `directory` column available. 78 + - If plan is `FastPath::DirectoryOnly`, render immediately and return. 79 + 80 + ### Stage 2: Early probe 81 + 82 + - Compute early columns only (`status`, `change-date`, etc.) for surviving rows. 83 + - Apply early-eligible filters. 84 + - Apply early sort keys. 85 + - If plan supports streaming, emit rows progressively. 86 + 87 + ### Stage 3: Late probe (conditional) 88 + 89 + - Run plugin streaming only if `plan.needs_late_probe`. 90 + - Request only late-required columns from the registry. 91 + - Merge patches into existing rows. 92 + 93 + ### Stage 4: Finalize + render 94 + 95 + - Apply deferred filters and final sort keys. 96 + - Render text/json through existing formatting semantics. 97 + 98 + ## Scan policy integration 99 + 100 + Flowline introduces policy handling for scan mode with late requirements: 101 + 102 + - `upgrade` (default): execute late stage to preserve correctness. 103 + - `defer`: skip late stage, warn on stderr, render early-only results. 104 + - `error`: fail with actionable message listing unsupported keys. 105 + 106 + Policy should be pluggable in runtime config so future flags can expose it. 107 + 108 + ## Compatibility expectations 109 + 110 + - Existing non-scan commands produce equivalent results as before. 111 + - Existing format and json rendering stay stable. 112 + - Existing plugin toggles and plugin-selected columns continue to work. 113 + 114 + ## Acceptance criteria 115 + 116 + - Runtime executes through staged entrypoint for all command paths. 117 + - `FastPath::DirectoryOnly` uses planner decision and preserves current directory-only output semantics. 118 + - Early-stage query shapes avoid late probe/plugin collection. 119 + - Mixed-stage queries still produce correct final ordering and output. 120 + - Scan mode obeys selected late-key policy (`upgrade`/`defer`/`error`). 121 + - Text and JSON output remain compatible with current format semantics. 122 + 123 + ## Verification 124 + 125 + - Integration tests for staged vs baseline equivalence: 126 + - `--all --format all` 127 + - `--all --format "{status} {directory}"` 128 + - `--all --format directory` 129 + - `--all --json --format "{directory} {status}"` 130 + - Policy tests for scan mode: 131 + - scan + early-only keys 132 + - scan + late keys under each policy 133 + - Performance checks on large directory sets to confirm reduced late probe work. 134 + 135 + ## Risks and mitigations 136 + 137 + - **Risk:** staged sort behavior differs from one-shot sort. 138 + - **Mitigation:** enforce stable sort semantics and add deterministic order tests. 139 + - **Risk:** filter placement bugs leak/omit rows. 140 + - **Mitigation:** explicit stage-tagged filter evaluation tests. 141 + - **Risk:** runtime complexity increases maintenance cost. 142 + - **Mitigation:** domain-grouped runtime modules with narrow responsibilities. 143 + 144 + ## Done when 145 + 146 + Flowline is complete when runtime follows planner stage decisions end-to-end, delivering early responsiveness for scan-friendly queries while preserving full-mode correctness and output compatibility.
+196
doc/plan/seedling-phase-1-query-planner.md
··· 1 + # Seedling: Phase 1 Query Planner Design 2 + 3 + Plan name: **Seedling**. 4 + 5 + Seedling is Phase 1 of the planner/pushdown roadmap from [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md). It introduces planner metadata and rule evaluation without changing final runtime behavior. 6 + 7 + Related references: 8 + 9 + - [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md) 10 + - [`/doc/pick-iter.md`](/doc/pick-iter.md) 11 + - [`/src/main.rs`](/src/main.rs) 12 + - [`/src/plugin.rs`](/src/plugin.rs) 13 + 14 + ## Why this phase exists 15 + 16 + Current fast wins (for example `--all --format directory`) are implemented as direct special cases in [`/src/main.rs`](/src/main.rs). Seedling generalizes this into a planner that can decide when fast paths are valid. 17 + 18 + This keeps optimization logic centralized and predictable. 19 + 20 + ## Goals 21 + 22 + - Introduce a `LogicalQuery -> PhysicalPlan` conversion path. 23 + - Classify known columns by availability stage and cost. 24 + - Apply projection/filter/sort pushdown rules in planning only. 25 + - Emit a deterministic `PhysicalPlan` object that runtime can consume later. 26 + - Preserve current runtime behavior by default (planner can be sidecar at first). 27 + 28 + ## Non-goals 29 + 30 + - No runtime stage executor yet (that is Flowline / Phase 2). 31 + - No plugin API changes yet. 32 + - No user-facing behavior changes except optional debug output. 33 + 34 + ## Proposed module layout 35 + 36 + Create a planner domain (not flat): 37 + 38 + - `src/planner/mod.rs` — public planner API (`build_plan`) 39 + - `src/planner/query.rs` — `LogicalQuery`, sort/filter parsing structures 40 + - `src/planner/meta.rs` — column capability metadata map 41 + - `src/planner/rules.rs` — pushdown rule evaluation 42 + - `src/planner/plan.rs` — `PhysicalPlan` stage requirements 43 + 44 + Integration point in existing runtime: 45 + 46 + - `src/main.rs` calls `planner::build_plan(...)` after argument parsing. 47 + 48 + ## Data contracts 49 + 50 + ```rust 51 + #[derive(Debug, Clone, Copy, PartialEq, Eq)] 52 + pub enum ExecMode { 53 + Full, 54 + Scan, 55 + } 56 + 57 + #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] 58 + pub enum AvailabilityStage { 59 + Enumerate, 60 + EarlyProbe, 61 + LateProbe, 62 + Finalize, 63 + } 64 + 65 + #[derive(Debug, Clone, Copy, PartialEq, Eq)] 66 + pub enum CostClass { 67 + Free, 68 + Cheap, 69 + Expensive, 70 + } 71 + 72 + #[derive(Debug, Clone, PartialEq, Eq)] 73 + pub struct ColumnPlanMeta { 74 + pub key: &'static str, 75 + pub stage: AvailabilityStage, 76 + pub cost: CostClass, 77 + } 78 + 79 + #[derive(Debug, Clone, PartialEq, Eq)] 80 + pub struct LogicalQuery { 81 + pub mode: ExecMode, 82 + pub roots: Vec<std::path::PathBuf>, 83 + pub projection: Vec<String>, 84 + pub filters: Vec<FilterExpr>, 85 + pub sort_keys: Vec<SortKey>, 86 + pub emit_json: bool, 87 + } 88 + 89 + #[derive(Debug, Clone, PartialEq, Eq)] 90 + pub struct PhysicalPlan { 91 + pub required_columns: Vec<String>, 92 + pub earliest_render_stage: AvailabilityStage, 93 + pub needs_late_probe: bool, 94 + pub can_stream_early: bool, 95 + pub fast_path: FastPath, 96 + } 97 + 98 + #[derive(Debug, Clone, Copy, PartialEq, Eq)] 99 + pub enum FastPath { 100 + None, 101 + DirectoryOnly, 102 + } 103 + ``` 104 + 105 + ## Initial column capability table 106 + 107 + Seedling uses a hardcoded map in `src/planner/meta.rs`: 108 + 109 + | Column | Stage | Cost | 110 + |---|---|---| 111 + | `directory` | `Enumerate` | `Free` | 112 + | `status` | `EarlyProbe` | `Cheap` | 113 + | `change-date` | `EarlyProbe` | `Cheap` | 114 + | `workparent` | `LateProbe` | `Cheap` | 115 + | `commit-date` | `LateProbe` | `Expensive` | 116 + | `ahead` | `LateProbe` | `Expensive` | 117 + 118 + Unknown columns default to `LateProbe` + `Expensive` (safe fallback). 119 + 120 + ## Planning rules 121 + 122 + ### Projection pushdown 123 + 124 + `required_columns` is the union of: 125 + 126 + - projection keys 127 + - filter-referenced keys 128 + - sort keys 129 + 130 + If the union is exactly `{directory}`, planner returns `FastPath::DirectoryOnly`. 131 + 132 + ### Filter stage assignment 133 + 134 + Each filter is assigned to the earliest stage where its column is available. 135 + 136 + - early filters can reduce rows before expensive work 137 + - late filters are marked and deferred 138 + 139 + ### Sort stage assignment 140 + 141 + Sort keys are split by stage: 142 + 143 + - early-sort keys (`Enumerate`/`EarlyProbe`) 144 + - final-sort keys (`LateProbe`) 145 + 146 + Planner marks whether staged sort is needed in Phase 2. 147 + 148 + ### Mode + policy decision 149 + 150 + For `ExecMode::Scan`, late requirements are recorded so runtime can decide policy in Phase 2 (`upgrade`, `defer`, `error`). 151 + 152 + Seedling does not enforce the policy; it only computes requirements. 153 + 154 + ## Integration sequence 155 + 156 + 1. Parse CLI args in [`/src/main.rs`](/src/main.rs). 157 + 2. Build `LogicalQuery` from parsed args. 158 + 3. Call `planner::build_plan(&query)`. 159 + 4. Use plan for diagnostics and fast-path selection (initially `DirectoryOnly`). 160 + 5. Continue existing runtime path unchanged for non-fast-path cases. 161 + 162 + This gives us planner coverage without runtime churn. 163 + 164 + ## Acceptance criteria 165 + 166 + - `planner::build_plan` exists and is covered by unit tests. 167 + - Planner metadata table exists for all currently documented output columns. 168 + - Directory-only requests (`directory` or `{directory}`) resolve to `FastPath::DirectoryOnly`. 169 + - Mixed projection/sort/filter queries produce deterministic `required_columns` regardless of input ordering. 170 + - Scan queries with late keys are detected and marked in the plan. 171 + - Existing output behavior remains unchanged for non-fast-path queries. 172 + 173 + ## Verification 174 + 175 + - Unit tests in planner module for: 176 + - projection union logic 177 + - fast-path detection 178 + - sort/filter stage assignment 179 + - unknown column fallback behavior 180 + - CLI smoke checks still pass for: 181 + - `is-tree --all --format directory` 182 + - `is-tree --all --format "{status} {directory}"` 183 + - `is-tree --all --format all --json` 184 + 185 + ## Risks and mitigations 186 + 187 + - **Risk:** planner and runtime diverge. 188 + - **Mitigation:** keep planner outputs explicit and tested; only consume planner decisions through well-defined fields. 189 + - **Risk:** unknown columns accidentally break planning. 190 + - **Mitigation:** safe fallback to late/expensive. 191 + - **Risk:** overfitting to current columns. 192 + - **Mitigation:** isolate metadata table so plugin metadata can replace it in a later phase. 193 + 194 + ## Done when 195 + 196 + Seedling is complete when `is-tree` has a test-backed planning layer that can accurately identify fast-path and stage requirements, while preserving all existing runtime behavior.
+4 -2
doc/planner-pushdown.md
··· 5 5 Related docs and code: 6 6 7 7 - [`/doc/pick-iter.md`](/doc/pick-iter.md) 8 + - [`/doc/plan/seedling-phase-1-query-planner.md`](/doc/plan/seedling-phase-1-query-planner.md) 9 + - [`/doc/plan/flowline-phase-2-staged-runtime.md`](/doc/plan/flowline-phase-2-staged-runtime.md) 8 10 - [`/README.md`](/README.md) 9 11 - [`/src/main.rs`](/src/main.rs) 10 12 - [`/src/plugin.rs`](/src/plugin.rs) ··· 219 221 220 222 ## Implementation plan 221 223 222 - ### Phase 1: planner metadata and rule engine 224 + ### Phase 1: Seedling (planner metadata and rule engine) 223 225 224 226 - Add `LogicalQuery`, `PhysicalPlan`, and column metadata table. 225 227 - Build planner from existing CLI args (`format`, `sort`, `filter`, `json`, `all`). ··· 230 232 - Planner returns deterministic stage assignment for projection/filter/sort keys. 231 233 - `--all --format directory` is represented as enumerate-only plan. 232 234 233 - ### Phase 2: staged execution runtime 235 + ### Phase 2: Flowline (staged execution runtime) 234 236 235 237 - Introduce execution stages in `run()` path: 236 238 - enumerate