Action Plan to Calibrate the KLBR Agent Memory Architecture#

Executive summary#

KLBR already has a strong conceptual shape: a semantic four-layer memory hierarchy, classifier-first routing, L1 time-windowed retrieval, ANN candidate generation followed by cross-encoder reranking, provenance backlinks, and an append-only archive. The immediate problem is not missing ideas; it is missing operational artifacts. The highest-value next steps are to freeze the lifecycle contract and schemas, build a reproducible benchmark harness, establish a flat-memory baseline, and only then calibrate routing, time windows, ANN depth, reranker thresholds, backlink policies, and consolidation triggers. That sequencing matches what the most relevant literature says matters in assistant memory systems: LongMemEval explicitly decomposes performance into indexing, retrieval, and reading; PerLTQA isolates routing/classification as a separate task and reports strong gains from BERT-style classifiers; BEIR shows why retrieve-then-rerank is usually better than one-stage dense retrieval but more expensive; and AgentPoison and MEXTRA show that memory quality work should not be separated from security and privacy work. citeturn1search0turn1search2turn1search3turn2search3turn2search14

For implementation, the safest path is prototype locally, benchmark aggressively, and delay backend commitment until access patterns are measured. The backend landscape has shifted since many early SQLite-vector assumptions were written down: SQLite now has an official vec1 extension that provides ANN search using IVFADC and trained models, while sqlite-vec has added ANN-related code and benchmarking support but still describes itself as pre-v1. Qdrant is a strong default when filtered search, persistence, and operational simplicity matter; Milvus is better when distributed scale is already a requirement; Faiss remains the best research control because it is a high-performance library rather than a full database; and DiskANN becomes compelling only when SSD-scale vector search is truly needed. citeturn0search0turn0search1turn5search9turn6search13turn6search2turn5search4

My recommended default for the next 8–12 weeks is this: use a flat baseline first with one strong open embedding model, one fast reranker, and either SQLite vec1 or Qdrant; collect traceable evaluation data; then add the KLBR-specific mechanisms one at a time in the order of routing, L1 windowing, backlinks, and consolidation. That will let you answer the only questions that matter right now: whether layering actually helps your query mix, whether backlinks improve exact recovery enough to justify storage amplification, and whether the classifier meaningfully reduces cost without hurting recall. citeturn1search0turn1search1turn1search2turn11search2

Assumptions and gap inventory#

This report assumes that the only currently specified design elements are the ones in your prompt: the semantic layers, classifier-first routing, L1 time-window narrowing, ANN-plus-rerank retrieval, provenance backlinks, and append-only archives. It also assumes that no executable code, schemas, representative traces, model selections, calibrated thresholds, concurrency rules, namespace model, deletion semantics, or benchmark scorecards have yet been provided. Under that assumption, the first goal is to convert KLBR from a concept into a system with falsifiable interfaces and measurable behavior.

Two of those gaps are more dangerous than they may look: namespace design and deletion semantics. Production memory systems increasingly make both explicit. LangGraph’s long-term memory is organized by namespace and key rather than as one undifferentiated store, and OpenAI’s memory controls emphasize that users should be able to inspect, delete, or disable memory explicitly. KLBR should copy that governance instinct even if its internal design is more sophisticated than those product patterns. citeturn10search0turn10search1turn10search13

A good working assumption for the prototype is an event-sourced memory model: L1 episodic records are immutable source events; L2–L4 are versioned derived views; every derived memory has provenance edges to lower-level evidence; deletes create tombstones immediately and schedule asynchronous re-materialization of any affected higher-layer memories. That assumption is technically conservative and is the cleanest way to preserve both provenance and erasure semantics.

Prioritized calibration backlog#

The table below is the practical backlog I would run. Owners are role-based placeholders rather than named people.

The logic of this backlog is deliberate. Do not tune layered retrieval before flat retrieval is strong; do not tune thresholds before models are frozen; do not tune concurrency before lifecycle semantics are explicit; and do not ship memory before delete and poisoning tests exist. LongMemEval, LoCoMo, and PerLTQA all punish systems that optimize one narrow axis while leaving the rest underspecified. citeturn1search0turn1search1turn1search2

Backend and model choices#

The most practical model plan is to use a two-tier roster: one low-risk baseline stack and one higher-ambition stack. For embeddings, multilingual E5 is a strong conservative baseline because it is well documented and available in multiple sizes; BGE-M3 is more ambitious because it supports dense, sparse, and multi-vector retrieval in one model and handles long inputs up to 8,192 tokens; jina-embeddings-v3 is also strong for long-context multilingual retrieval and offers task-specific adapters. For reranking, a fast English baseline such as cross-encoder/ms-marco-MiniLM-L6-v2 is still useful for quick design loops, while bge-reranker-v2-m3 is the better multilingual and higher-accuracy option when latency permits. citeturn3search0turn3search1turn3search2turn3search4turn3search8

For routing, start with the simplest deployable thing that can win: logistic regression over the same embedding vectors used for retrieval, exported to ONNX Runtime. That keeps training and deployment simple, lets you score on CPU, and makes ablations easy. If that misses the target, switch to a compact BERT/MiniLM classifier. PerLTQA is the strongest direct evidence here because it treats memory classification as a first-class task and reports that BERT-based classifiers outperform LLM-based routing on that subproblem. citeturn1search2turn9search1turn9search0

The backend comparison below is intentionally qualitative and workload-dependent. The ratings are expected relative behavior for KLBR-like filtered semantic retrieval, synthesized from the algorithms and deployment models in the cited primary sources. They should be validated against your own traces before any long-term commitment. citeturn0search0turn0search1turn6search0turn6search13turn6search2turn5search4turn5search5

If I had to choose one starting stack today, I would use multilingual-E5-base or BGE-M3 for embeddings, MiniLM cross-encoder for the first design loop, Qdrant for the first operational pilot, and SQLite vec1 as the local deterministic reference. That combination gives you one service backend, one embedded local backend, one conservative retrieval baseline, and one higher-ceiling retrieval option. Only move to Milvus when you know you need distributed operations, and only move to DiskANN when your measured cold tier is large enough that SSD density is the dominant economic constraint. citeturn3search0turn3search1turn3search4turn6search13turn0search0turn5search5

Measurement and CI/CD runbook#

Benchmarking should mirror the structure used in LongMemEval: indexing, retrieval, and reading are separate parts of the system and should be measured separately. For KLBR, that means at least five score groups: retrieval quality, answer quality, provenance quality, systems performance, and security/privacy. BEIR and MTEB are the right tools to screen retrievers and embeddings before they are embedded in the full assistant-memory stack; LongMemEval, LoCoMo, and PerLTQA are the right tools to test the full memory architecture; YCSB is the right storage-style workload generator for backend stress; and MLPerf matters only if model inference on the target hardware becomes the bottleneck rather than memory design itself. citeturn1search0turn1search1turn1search2turn1search3turn2search0turn8search0turn8search5turn8search8

The most important sweep ranges should be explicit and finite. My recommended initial grid is: L1 default windows {3, 7, 14, 30, 60, 180} days; ANN candidate sizes {20, 50, 100, 200}; reranker acceptance by percentile and by score margin; backlink depth {0,1,2} with depth 1 as the default; classifier confidence using entropy and margin gates; and consolidation triggers across {time-based nightly, count-based every N inserts, density-based cluster threshold}. Avoid “magic thresholds.” Every threshold should have a sweep report, reliability plot, and error bucket analysis.

flowchart LR
    A[Gold traces and benchmark adapters] --> B[Schema validation and deterministic ingest]
    B --> C[Flat retrieval baseline]
    C --> D[Router ablation]
    D --> E[L1 time-window sweep]
    E --> F[ANN K sweep]
    F --> G[Reranker threshold sweep]
    G --> H[Backlink policy sweep]
    H --> I[Consolidation trigger sweep]
    I --> J[Security and deletion tests]
    J --> K[Final scorecard and operating point]

flowchart TD
    PR[Pull request or nightly build] --> UT[Unit tests and schema migration tests]
    UT --> IR[Deterministic ingest replay]
    IR --> BENCH[Offline benchmark suite]
    BENCH --> LME[LongMemEval, LoCoMo, PerLTQA]
    BENCH --> RET[BEIR and MTEB]
    BENCH --> SYS[YCSB-style backend load]
    BENCH --> SEC[AgentPoison and MEXTRA]
    LME --> SCORE[Unified scorecard with bootstrap CIs]
    RET --> SCORE
    SYS --> SCORE
    SEC --> SCORE
    SCORE --> GATE{Regression gate passed}
    GATE -->|Yes| REL[Release candidate]
    GATE -->|No| BLOCK[Block merge and open triage issue]

The runbook should be executed in this exact order:

1. Freeze the record and edge schema. Until record identity, timestamps, provenance edges, supersession edges, and tombstones are explicit, every later benchmark can be invalidated by implementation drift.

2. Build the gold set before building the hierarchy. Use benchmark adapters plus a small hand-labeled KLBR-native set that includes exact episodic queries, vague pattern questions, stable trait identity questions, user updates, and abstentions. LongMemEval and LoCoMo make these distinctions concrete. citeturn1search0turn1search1

3. Train one flat baseline and never delete it. It should remain in CI forever as the control arm. Dense-retrieve-plus-rerank is the right baseline because that pattern is well grounded in DPR, monoBERT, and modern retrieve-rerank practice. citeturn11search0turn11search1turn11search2

4. Calibrate routing independently. Treat routing as its own supervised problem before feeding it into end-to-end retrieval. That is exactly how PerLTQA structures the problem. citeturn1search2

5. Calibrate L1 windows on temporal subsets only. Do not let non-temporal queries dominate the search. Measure retrieval recall and latency as separate outcomes.

6. Introduce backlinks only after you can score them. A backlink policy without provenance precision and support coverage metrics is just another uncontrolled retrieval path.

7. Benchmark backends on your actual access pattern. YCSB-style mixes are useful, but also replay your real benchmark trace because vector filters, namespace cardinality, and consolidation writes matter more than generic KV throughput. citeturn8search0turn8search6

8. Do not mark the system production-ready until delete and poisoning tests pass. AgentPoison and MEXTRA show why this is a first-order requirement for memory systems rather than an afterthought. citeturn2search3turn2search14

Roadmap and optional hardware exploration#

A sensible roadmap is 12 weeks for the semantic-memory core, with a stretch path to 16–20 weeks if you decide to include deeper backend scaling work, formal privacy hardening, or hardware-tier experiments.

The hardware and memory-system simulators you mentioned should stay out of scope until backend measurements prove that hardware tiers are the bottleneck. If that day comes, use gem5 for full-system architecture studies and coherence-aware experiments, DRAMsim3 or Ramulator 2.0 for DRAM-controller and memory-standard exploration, and NVMain for DRAM/NVM hybrid studies. These are excellent tools, but they are for the later question of where the software stack should live, not for the current question of whether the semantic architecture is correct. citeturn7search0turn7search8turn7search1turn7search2turn7search6turn7search3turn7search15

A good decision rule is simple: if your bottlenecks are still dominated by retrieval errors, reranker cost, consolidation policy, or delete propagation, do not start a hardware simulation track. If you later discover that memory-mapped vector indexes, NUMA effects, accelerator-side reranking, or cold-tier SSD/DAX placement dominate p95 latency or cost, then the hardware tools become worthwhile.

Risks, mitigations, and selected references#

The main risk is calibration drift. A layered memory system can look better simply because it hides mistakes in flattering summaries, while exact episodic failures become harder to notice. Mitigate that by keeping a permanent flat baseline, requiring provenance precision metrics for every backlink policy, and separately tracking update correctness and temporal consistency—the two long-memory failure modes that LongMemEval and LoCoMo make particularly visible. citeturn1search0turn1search1

The second risk is synthetic-routing bias. It is fine to bootstrap the classifier with synthetic queries, but do not freeze the routing model on synthetic data alone. PerLTQA’s decomposition strongly supports routing as a cheap first stage, but production routing needs real query traces, active-learning refreshes, and calibration checks. citeturn1search2

The third risk is governance debt. Append-only memory is excellent for provenance and debugging, but unsafe without namespace boundaries, tombstones, and derived-view invalidation. The closest product analogues—LangGraph memory namespaces and OpenAI memory controls—show why user-scoped storage and explicit deletion must be designed up front, not retrofitted later. citeturn10search0turn10search1turn10search13

The fourth risk is security under adversarial memory use. AgentPoison and MEXTRA show that long-term memory can be poisoned or extracted even when the surrounding application seems benign. The mitigation is to make trust metadata, source provenance, namespace isolation, redaction, and benchmarked security tests part of the mainline roadmap rather than the security backlog nobody reaches. citeturn2search3turn2search14

Selected primary references for the KLBR workstream are below.

• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. citeturn1search0
• LoCoMo: Evaluating Very Long-Term Conversational Memory of LLM Agents. citeturn1search1
• PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering. citeturn1search2
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. citeturn1search3
• MTEB: Massive Text Embedding Benchmark. citeturn2search0
• Dense Passage Retrieval for Open-Domain Question Answering. citeturn11search0
• Multi-Stage Document Ranking with BERT. citeturn11search1
• SQLite vec1 official documentation. citeturn0search0turn0search4
• sqlite-vec repository and release state. citeturn0search1turn0search13
• Qdrant indexing, filtering, storage, and distributed deployment docs. citeturn0search2turn6search1turn5search3
• Milvus architecture and component docs. citeturn6search2turn6search6turn5search10
• Faiss documentation and repository. citeturn5search4turn5search0turn6search11
• DiskANN paper and repository. citeturn5search5turn5search9
• LightMem and Letta sleep-time references. citeturn4search0turn4search2turn4search14
• AgentPoison and MEXTRA. citeturn2search3turn2search14
• YCSB and MLPerf Inference. citeturn8search0turn8search5turn8search8