···11+# Demand-Driven Sense Loops for `is-tree`
22+33+This document links the `is-tree` scan/picker work to the loop architecture in [`rektide/iso-chill` `doc/design.md`](https://github.com/rektide/iso-chill/blob/main/doc/design.md), especially:
44+55+- [The Loop System](https://github.com/rektide/iso-chill/blob/main/doc/design.md#the-loop-system)
66+- [Queue](https://github.com/rektide/iso-chill/blob/main/doc/design.md#queue)
77+- [SignalEvent](https://github.com/rektide/iso-chill/blob/main/doc/design.md#signalevent)
88+- [Control Loop Design](https://github.com/rektide/iso-chill/blob/main/doc/design.md#control-loop-design)
99+1010+Related local docs:
1111+1212+- [`/doc/pick-iter.md`](/doc/pick-iter.md)
1313+- [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md)
1414+1515+## Intent
1616+1717+Move from one-shot "scan everything, compute everything" execution to a looped model:
1818+1919+- `scan` is a generator loop that emits candidates quickly.
2020+- downstream loops are consumers with explicit demands.
2121+- sensing pools fulfill those demands at the right cost tier.
2222+2323+Instead of hardcoding one runtime path, we declare process demands and let a sense planner satisfy them.
2424+2525+## Why this model fits
2626+2727+The `iso-chill` model gives us three useful ideas to borrow directly:
2828+2929+1. **Consumer-initiated subscriptions**: producers do not need to know all consumers.
3030+2. **Shared state + signal events**: events coordinate; state is source of truth.
3131+3. **Micro-batched loop ticks**: good traceability, bounded work, easier tuning.
3232+3333+For `is-tree`, this means scan can emit candidate references, while multiple consumers (picker, staleness view, renderer, sync planner) independently subscribe and ask for more detail only when needed.
3434+3535+## Core model
3636+3737+### Candidate context store
3838+3939+Maintain a shared per-path context store, analogous to `ProcessContext` in `iso-chill`.
4040+4141+Each candidate keeps:
4242+4343+- `directory` identity
4444+- known columns (`status`, `change-date`, `ahead`, ...)
4545+- freshness timestamps per column
4646+- in-flight demand state
4747+4848+### Signal events
4949+5050+Loops emit small events (not full payload copies), such as:
5151+5252+- `candidate.discovered`
5353+- `candidate.updated(column=change-date)`
5454+- `candidate.ready(demand=picker.fast)`
5555+5656+Consumers react to events, then read required data from the shared store.
5757+5858+### Sense pools
5959+6060+A sense pool is a loop (or strategy set) that can fill a class of columns.
6161+6262+Initial pools:
6363+6464+- **identity pool**: `status`, `directory`, worktree/basic type
6565+- **fs metadata pool**: `change-date`, directory mtime, file-count-lite
6666+- **history pool**: `commit-date`
6767+- **remote state pool**: `ahead`
6868+- **deep file pool**: per-file age/size/modified (future `--files`)
6969+7070+Pools run on demand, not eagerly for all candidates.
7171+7272+## Demand declarations
7373+7474+Each process declares what it needs as a demand contract.
7575+7676+Example schema:
7777+7878+```rust
7979+struct Demand {
8080+ id: String,
8181+ columns: Vec<String>,
8282+ max_staleness: std::time::Duration,
8383+ priority: u8,
8484+ max_candidates: Option<usize>,
8585+ ordering: Vec<SortKey>,
8686+}
8787+```
8888+8989+Examples:
9090+9191+- `picker.fast`
9292+ - columns: `directory`, `status`, `change-date`
9393+ - priority: high
9494+ - ordering: `change-date-`
9595+- `render.detail`
9696+ - columns: full format projection
9797+ - priority: medium
9898+- `stale.files`
9999+ - columns: `directory`, per-file age columns
100100+ - priority: low
101101+102102+The planner computes the minimal pool work needed to satisfy active demands.
103103+104104+## Loop topology
105105+106106+```mermaid
107107+flowchart LR
108108+ subgraph ScanLoop[Scan Generator Loop]
109109+ enum[enumerate roots]
110110+ detect[cheap identity detect]
111111+ emit_discovered[emit candidate.discovered]
112112+ enum --> detect --> emit_discovered
113113+ end
114114+115115+ subgraph Store[CandidateContext Store]
116116+ ctx[(candidate state)]
117117+ end
118118+119119+ subgraph SensePools[Sense Pools]
120120+ p_id[identity pool]
121121+ p_fs[fs metadata pool]
122122+ p_hist[history pool]
123123+ p_remote[remote state pool]
124124+ p_deep[deep file pool]
125125+ end
126126+127127+ subgraph Consumers[Consumer Loops]
128128+ pick[picker loop]
129129+ render[render loop]
130130+ stale[staleness loop]
131131+ sync[reforrest prep loop]
132132+ end
133133+134134+ emit_discovered --> ctx
135135+ ctx --> p_id
136136+ ctx --> p_fs
137137+ ctx --> p_hist
138138+ ctx --> p_remote
139139+ ctx --> p_deep
140140+141141+ p_id --> ctx
142142+ p_fs --> ctx
143143+ p_hist --> ctx
144144+ p_remote --> ctx
145145+ p_deep --> ctx
146146+147147+ ctx --> pick
148148+ ctx --> render
149149+ ctx --> stale
150150+ ctx --> sync
151151+```
152152+153153+## How this changes `--scan`
154154+155155+`--scan` becomes the primary candidate-generator loop, not a one-off mode.
156156+157157+- It emits candidates immediately from cheap detection.
158158+- It can apply early ordering (`directory`, `change-date`) when available.
159159+- It does not wait for expensive pools.
160160+161161+Then downstream consumers trigger deeper sensing for selected candidates only.
162162+163163+Pipeline intent remains:
164164+165165+```bash
166166+is-tree --scan --format directory | fuzzel --dmenu --multi | is-tree --stdin --format all
167167+```
168168+169169+But internally this is not "run two unrelated commands"; it is two demand profiles over the same sensing model:
170170+171171+- first command asks for `picker.fast`
172172+- second command asks for `render.detail` on a reduced candidate set
173173+174174+## Planning rules (demand-first)
175175+176176+1. Union active demand columns.
177177+2. Determine cheapest pool set that can satisfy union.
178178+3. Execute pools in stage order (cheap to expensive).
179179+4. Emit events as each demand reaches readiness.
180180+5. Recompute when demand set changes (new consumer, canceled consumer, narrowed candidate set).
181181+182182+This generalizes pushdown from static CLI parsing to live loop operation.
183183+184184+## Practical example
185185+186186+### Step 1: picker demand
187187+188188+Consumer requests:
189189+190190+- columns: `directory`, `status`, `change-date`
191191+- ordering: `change-date-`
192192+- target: first 200 candidates quickly
193193+194194+Planner schedules:
195195+196196+- scan loop + identity pool + fs metadata pool
197197+- no history/remote/deep pools yet
198198+199199+### Step 2: user selects 8 paths
200200+201201+Renderer requests:
202202+203203+- columns: `status`, `directory`, `commit-date`, `ahead`
204204+- scope: selected 8 paths
205205+206206+Planner schedules:
207207+208208+- history + remote pools, but only for selected 8
209209+210210+This is where the large performance win comes from.
211211+212212+## Minimal implementation path
213213+214214+1. Keep current CLI behavior and planner from [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md).
215215+2. Add a small in-memory candidate store abstraction.
216216+3. Introduce one event channel and one consumer loop (`picker.fast`).
217217+4. Split existing probes into pool-like executors with declared column coverage.
218218+5. Expand to multiple consumer loops and scoped demand recomputation.
219219+220220+## Design constraints
221221+222222+- Preserve Unix composability at the CLI surface.
223223+- Keep deterministic output for non-streaming commands.
224224+- Make pool scheduling observable (debug logs per demand and pool run).
225225+- Avoid hidden expensive upgrades unless policy explicitly allows it.
226226+227227+## Decision
228228+229229+Adopt a demand-driven sense-loop architecture:
230230+231231+- scan loop generates candidates
232232+- consumers declare demands
233233+- sense pools fulfill demands by cost tier
234234+235235+This ties `is-tree` directly to proven `iso-chill` loop patterns while staying focused on repository candidate generation and selective deep inspection.
+146
doc/plan/flowline-phase-2-staged-runtime.md
···11+# Flowline: Phase 2 Staged Runtime Design
22+33+Plan name: **Flowline**.
44+55+Flowline is Phase 2 of the planner/pushdown roadmap from [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md). It consumes Seedling planner outputs and executes queries in staged runtime order.
66+77+Related references:
88+99+- [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md)
1010+- [`/doc/plan/seedling-phase-1-query-planner.md`](/doc/plan/seedling-phase-1-query-planner.md)
1111+- [`/doc/pick-iter.md`](/doc/pick-iter.md)
1212+- [`/src/main.rs`](/src/main.rs)
1313+- [`/src/plugin.rs`](/src/plugin.rs)
1414+1515+## Why this phase exists
1616+1717+Seedling can tell us what work is needed and when it is available. Flowline makes runtime follow that plan so early columns are computed and emitted before expensive columns.
1818+1919+This is the phase that turns planning into user-visible speed improvements.
2020+2121+## Goals
2222+2323+- Execute query work in explicit stages.
2424+- Support early streaming when plan allows.
2525+- Apply early filters/sorts before late probes.
2626+- Keep full-mode correctness equivalent to current behavior.
2727+- Route directory-only optimization through planner output, not bespoke branch logic.
2828+2929+## Non-goals
3030+3131+- No plugin metadata API changes yet (still use planner metadata map).
3232+- No per-file recursion implementation yet.
3333+- No long-running daemon behavior.
3434+3535+## Runtime stage model
3636+3737+```mermaid
3838+flowchart LR
3939+ Enumerate[Enumerate paths] --> EarlyProbe[Early probe columns]
4040+ EarlyProbe --> EarlyOps[Early filter and sort]
4141+ EarlyOps --> LateProbe{Need late probe?}
4242+ LateProbe -- No --> Render[Render rows]
4343+ LateProbe -- Yes --> LateCollect[Plugin late collection]
4444+ LateCollect --> FinalOps[Final filter and sort]
4545+ FinalOps --> Render
4646+```
4747+4848+## Proposed module layout
4949+5050+Create a runtime domain with stage-specific units:
5151+5252+- `src/runtime/mod.rs` — execution entrypoint
5353+- `src/runtime/enumerate.rs` — path enumeration and base row init
5454+- `src/runtime/early.rs` — early probe and early predicate/sort application
5555+- `src/runtime/late.rs` — plugin streaming/merge for late columns
5656+- `src/runtime/render.rs` — text/json rendering adapters
5757+5858+Planner integration:
5959+6060+- `src/main.rs` builds `LogicalQuery` and `PhysicalPlan`, then calls `runtime::execute(plan, args)`.
6161+6262+## Execution contract
6363+6464+`runtime::execute` consumes:
6565+6666+- `PhysicalPlan` (from Seedling)
6767+- parsed CLI flags
6868+- plugin registry
6969+7070+It returns a fully rendered output side effect (stdout/stderr) with the same user-visible semantics as current runtime.
7171+7272+## Stage behavior details
7373+7474+### Stage 1: Enumerate
7575+7676+- Build candidate path list from positional args or `--all` roots.
7777+- Initialize row records with `directory` column available.
7878+- If plan is `FastPath::DirectoryOnly`, render immediately and return.
7979+8080+### Stage 2: Early probe
8181+8282+- Compute early columns only (`status`, `change-date`, etc.) for surviving rows.
8383+- Apply early-eligible filters.
8484+- Apply early sort keys.
8585+- If plan supports streaming, emit rows progressively.
8686+8787+### Stage 3: Late probe (conditional)
8888+8989+- Run plugin streaming only if `plan.needs_late_probe`.
9090+- Request only late-required columns from the registry.
9191+- Merge patches into existing rows.
9292+9393+### Stage 4: Finalize + render
9494+9595+- Apply deferred filters and final sort keys.
9696+- Render text/json through existing formatting semantics.
9797+9898+## Scan policy integration
9999+100100+Flowline introduces policy handling for scan mode with late requirements:
101101+102102+- `upgrade` (default): execute late stage to preserve correctness.
103103+- `defer`: skip late stage, warn on stderr, render early-only results.
104104+- `error`: fail with actionable message listing unsupported keys.
105105+106106+Policy should be pluggable in runtime config so future flags can expose it.
107107+108108+## Compatibility expectations
109109+110110+- Existing non-scan commands produce equivalent results as before.
111111+- Existing format and json rendering stay stable.
112112+- Existing plugin toggles and plugin-selected columns continue to work.
113113+114114+## Acceptance criteria
115115+116116+- Runtime executes through staged entrypoint for all command paths.
117117+- `FastPath::DirectoryOnly` uses planner decision and preserves current directory-only output semantics.
118118+- Early-stage query shapes avoid late probe/plugin collection.
119119+- Mixed-stage queries still produce correct final ordering and output.
120120+- Scan mode obeys selected late-key policy (`upgrade`/`defer`/`error`).
121121+- Text and JSON output remain compatible with current format semantics.
122122+123123+## Verification
124124+125125+- Integration tests for staged vs baseline equivalence:
126126+ - `--all --format all`
127127+ - `--all --format "{status} {directory}"`
128128+ - `--all --format directory`
129129+ - `--all --json --format "{directory} {status}"`
130130+- Policy tests for scan mode:
131131+ - scan + early-only keys
132132+ - scan + late keys under each policy
133133+- Performance checks on large directory sets to confirm reduced late probe work.
134134+135135+## Risks and mitigations
136136+137137+- **Risk:** staged sort behavior differs from one-shot sort.
138138+ - **Mitigation:** enforce stable sort semantics and add deterministic order tests.
139139+- **Risk:** filter placement bugs leak/omit rows.
140140+ - **Mitigation:** explicit stage-tagged filter evaluation tests.
141141+- **Risk:** runtime complexity increases maintenance cost.
142142+ - **Mitigation:** domain-grouped runtime modules with narrow responsibilities.
143143+144144+## Done when
145145+146146+Flowline is complete when runtime follows planner stage decisions end-to-end, delivering early responsiveness for scan-friendly queries while preserving full-mode correctness and output compatibility.
+196
doc/plan/seedling-phase-1-query-planner.md
···11+# Seedling: Phase 1 Query Planner Design
22+33+Plan name: **Seedling**.
44+55+Seedling is Phase 1 of the planner/pushdown roadmap from [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md). It introduces planner metadata and rule evaluation without changing final runtime behavior.
66+77+Related references:
88+99+- [`/doc/planner-pushdown.md`](/doc/planner-pushdown.md)
1010+- [`/doc/pick-iter.md`](/doc/pick-iter.md)
1111+- [`/src/main.rs`](/src/main.rs)
1212+- [`/src/plugin.rs`](/src/plugin.rs)
1313+1414+## Why this phase exists
1515+1616+Current fast wins (for example `--all --format directory`) are implemented as direct special cases in [`/src/main.rs`](/src/main.rs). Seedling generalizes this into a planner that can decide when fast paths are valid.
1717+1818+This keeps optimization logic centralized and predictable.
1919+2020+## Goals
2121+2222+- Introduce a `LogicalQuery -> PhysicalPlan` conversion path.
2323+- Classify known columns by availability stage and cost.
2424+- Apply projection/filter/sort pushdown rules in planning only.
2525+- Emit a deterministic `PhysicalPlan` object that runtime can consume later.
2626+- Preserve current runtime behavior by default (planner can be sidecar at first).
2727+2828+## Non-goals
2929+3030+- No runtime stage executor yet (that is Flowline / Phase 2).
3131+- No plugin API changes yet.
3232+- No user-facing behavior changes except optional debug output.
3333+3434+## Proposed module layout
3535+3636+Create a planner domain (not flat):
3737+3838+- `src/planner/mod.rs` — public planner API (`build_plan`)
3939+- `src/planner/query.rs` — `LogicalQuery`, sort/filter parsing structures
4040+- `src/planner/meta.rs` — column capability metadata map
4141+- `src/planner/rules.rs` — pushdown rule evaluation
4242+- `src/planner/plan.rs` — `PhysicalPlan` stage requirements
4343+4444+Integration point in existing runtime:
4545+4646+- `src/main.rs` calls `planner::build_plan(...)` after argument parsing.
4747+4848+## Data contracts
4949+5050+```rust
5151+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
5252+pub enum ExecMode {
5353+ Full,
5454+ Scan,
5555+}
5656+5757+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
5858+pub enum AvailabilityStage {
5959+ Enumerate,
6060+ EarlyProbe,
6161+ LateProbe,
6262+ Finalize,
6363+}
6464+6565+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
6666+pub enum CostClass {
6767+ Free,
6868+ Cheap,
6969+ Expensive,
7070+}
7171+7272+#[derive(Debug, Clone, PartialEq, Eq)]
7373+pub struct ColumnPlanMeta {
7474+ pub key: &'static str,
7575+ pub stage: AvailabilityStage,
7676+ pub cost: CostClass,
7777+}
7878+7979+#[derive(Debug, Clone, PartialEq, Eq)]
8080+pub struct LogicalQuery {
8181+ pub mode: ExecMode,
8282+ pub roots: Vec<std::path::PathBuf>,
8383+ pub projection: Vec<String>,
8484+ pub filters: Vec<FilterExpr>,
8585+ pub sort_keys: Vec<SortKey>,
8686+ pub emit_json: bool,
8787+}
8888+8989+#[derive(Debug, Clone, PartialEq, Eq)]
9090+pub struct PhysicalPlan {
9191+ pub required_columns: Vec<String>,
9292+ pub earliest_render_stage: AvailabilityStage,
9393+ pub needs_late_probe: bool,
9494+ pub can_stream_early: bool,
9595+ pub fast_path: FastPath,
9696+}
9797+9898+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
9999+pub enum FastPath {
100100+ None,
101101+ DirectoryOnly,
102102+}
103103+```
104104+105105+## Initial column capability table
106106+107107+Seedling uses a hardcoded map in `src/planner/meta.rs`:
108108+109109+| Column | Stage | Cost |
110110+|---|---|---|
111111+| `directory` | `Enumerate` | `Free` |
112112+| `status` | `EarlyProbe` | `Cheap` |
113113+| `change-date` | `EarlyProbe` | `Cheap` |
114114+| `workparent` | `LateProbe` | `Cheap` |
115115+| `commit-date` | `LateProbe` | `Expensive` |
116116+| `ahead` | `LateProbe` | `Expensive` |
117117+118118+Unknown columns default to `LateProbe` + `Expensive` (safe fallback).
119119+120120+## Planning rules
121121+122122+### Projection pushdown
123123+124124+`required_columns` is the union of:
125125+126126+- projection keys
127127+- filter-referenced keys
128128+- sort keys
129129+130130+If the union is exactly `{directory}`, planner returns `FastPath::DirectoryOnly`.
131131+132132+### Filter stage assignment
133133+134134+Each filter is assigned to the earliest stage where its column is available.
135135+136136+- early filters can reduce rows before expensive work
137137+- late filters are marked and deferred
138138+139139+### Sort stage assignment
140140+141141+Sort keys are split by stage:
142142+143143+- early-sort keys (`Enumerate`/`EarlyProbe`)
144144+- final-sort keys (`LateProbe`)
145145+146146+Planner marks whether staged sort is needed in Phase 2.
147147+148148+### Mode + policy decision
149149+150150+For `ExecMode::Scan`, late requirements are recorded so runtime can decide policy in Phase 2 (`upgrade`, `defer`, `error`).
151151+152152+Seedling does not enforce the policy; it only computes requirements.
153153+154154+## Integration sequence
155155+156156+1. Parse CLI args in [`/src/main.rs`](/src/main.rs).
157157+2. Build `LogicalQuery` from parsed args.
158158+3. Call `planner::build_plan(&query)`.
159159+4. Use plan for diagnostics and fast-path selection (initially `DirectoryOnly`).
160160+5. Continue existing runtime path unchanged for non-fast-path cases.
161161+162162+This gives us planner coverage without runtime churn.
163163+164164+## Acceptance criteria
165165+166166+- `planner::build_plan` exists and is covered by unit tests.
167167+- Planner metadata table exists for all currently documented output columns.
168168+- Directory-only requests (`directory` or `{directory}`) resolve to `FastPath::DirectoryOnly`.
169169+- Mixed projection/sort/filter queries produce deterministic `required_columns` regardless of input ordering.
170170+- Scan queries with late keys are detected and marked in the plan.
171171+- Existing output behavior remains unchanged for non-fast-path queries.
172172+173173+## Verification
174174+175175+- Unit tests in planner module for:
176176+ - projection union logic
177177+ - fast-path detection
178178+ - sort/filter stage assignment
179179+ - unknown column fallback behavior
180180+- CLI smoke checks still pass for:
181181+ - `is-tree --all --format directory`
182182+ - `is-tree --all --format "{status} {directory}"`
183183+ - `is-tree --all --format all --json`
184184+185185+## Risks and mitigations
186186+187187+- **Risk:** planner and runtime diverge.
188188+ - **Mitigation:** keep planner outputs explicit and tested; only consume planner decisions through well-defined fields.
189189+- **Risk:** unknown columns accidentally break planning.
190190+ - **Mitigation:** safe fallback to late/expensive.
191191+- **Risk:** overfitting to current columns.
192192+ - **Mitigation:** isolate metadata table so plugin metadata can replace it in a later phase.
193193+194194+## Done when
195195+196196+Seedling is complete when `is-tree` has a test-backed planning layer that can accurately identify fast-path and stage requirements, while preserving all existing runtime behavior.
+4-2
doc/planner-pushdown.md
···55Related docs and code:
6677- [`/doc/pick-iter.md`](/doc/pick-iter.md)
88+- [`/doc/plan/seedling-phase-1-query-planner.md`](/doc/plan/seedling-phase-1-query-planner.md)
99+- [`/doc/plan/flowline-phase-2-staged-runtime.md`](/doc/plan/flowline-phase-2-staged-runtime.md)
810- [`/README.md`](/README.md)
911- [`/src/main.rs`](/src/main.rs)
1012- [`/src/plugin.rs`](/src/plugin.rs)
···219221220222## Implementation plan
221223222222-### Phase 1: planner metadata and rule engine
224224+### Phase 1: Seedling (planner metadata and rule engine)
223225224226- Add `LogicalQuery`, `PhysicalPlan`, and column metadata table.
225227- Build planner from existing CLI args (`format`, `sort`, `filter`, `json`, `all`).
···230232- Planner returns deterministic stage assignment for projection/filter/sort keys.
231233- `--all --format directory` is represented as enumerate-only plan.
232234233233-### Phase 2: staged execution runtime
235235+### Phase 2: Flowline (staged execution runtime)
234236235237- Introduce execution stages in `run()` path:
236238 - enumerate