Save this runbook if you use Claude + Codex locally.
On February 19, 2026, one measured pass cut zsh -i -c exit from 1.794s to 0.386s(-78.5%). [E1]
Everyone says "fix prompts"; in practice, this is a systems issue, and teams without baselines + retrieval gates ship false confidence.
This is the same repeatable 5-step sequence I now reuse: baselines, gates, memory tiers, retrieval benchmarks, and drift checks.
Evidence Snapshot (Audited February 20, 2026)
- launch agents loaded: 8/8 [E2]
- daily memory coverage: 108 files (2025-09-27 through 2026-02-20) [E3]
- extracted totals: 7596 prompts, 22858 tool calls [E3]
- benchmark A (2026-02-20T04:16:45Z): hit_rate=1.0000, mrr=0.7500,ndcg=0.8143, precision=0.1250, recall=1.0000, triage=0 [E4]
- benchmark B (2026-02-20T04:56:59Z): hit_rate=0.9500, mrr=0.7125,ndcg=0.7727, precision=0.1188, recall=0.9500, triage=2 [E5]
- latest benchmark (2026-02-20T06:58:19Z): dense model nomic-embed-text:latest, vector_dim=768, dense enabled, gates passing [E6]
5-Step Saveable Implementation
Step 1: Baseline First
Measure before any config edits.
- zsh -i -c exit: 1.794s -> 0.386s (-78.5%) [E1]
- codex --help: 0.210s -> 0.116s (-44.8%) [E1]
- claude --help: 0.464s -> 0.312s (-32.8%) [E1]
Step 2: Add Reliability Gates
Replace reminder-based quality with execution checks:
- preflight checks
- apply + verify loops
- browser validation for UI work
- explicit done criteria
Step 3: Split Memory into 3 Tiers
- Tier 1: raw session history
- Tier 2: curated working memory
- Tier 3: hybrid retrieval + graph layer
This prevents noisy logs from polluting high-signal memory.
Step 4: Benchmark Retrieval Like Infra
Track:
- Hit@K
- MRR@K
- NDCG@K
- Precision@K
- Recall@K
- triage issue count
Back-to-back snapshots caught quality drift (triage: 0 -> 2) while gates remained green. [E4][E5]
Step 5: Fix Drift in Ingestion Paths
Incident found:
- active drafts in ~/docs/content/drafts
- sync defaults still pointed to old root
Fix:
- set canonical drafts root
- keep legacy remap compatibility
- re-sync and rebuild index
Outcome: graph alignment became explicit and auditable. [E7]
Copy-Paste Run Sequence
~/control/automation/ai-dev-optimization/scripts/manage_knowledge_launchagents.sh status python3 ~/control/automation/ai-dev-optimization/scripts/run_snapshot.sh 24 python3 ~/control/automation/ai-dev-optimization/scripts/sync_article_memory.py python3 ~/control/automation/ai-dev-optimization/scripts/build_memory_rag_index.py --backend hybrid --allow-sparse-fallback --output ~/control/knowledge/knowledge-memory/data/rag/hybrid-index.json python3 ~/control/automation/ai-dev-optimization/scripts/benchmark_memory_retrieval.py
Weekly Cadence (Use This As a Checklist)
- Verify launch agents are loaded.
- Refresh extraction + sync.
- Rebuild retrieval index.
- Run benchmark and compare triage delta.
- Promote only high-signal memory updates.
Failure Patterns to Watch
- Health checks green, but capability degraded.
- “One memory file” turning retrieval into noise.
- Path drift between active drafts and sync roots.
Conclusion
The point of this setup is simple:
- less waiting
- less guesswork
- fewer silent regressions
Prompt quality still matters, but the durable gains came from instrumentation and operating discipline.
Use this as a working runbook before your next sprint, then iterate the thresholds and checks to match your stack.
Evidence key
Every [E] marker above is a first-party measurement from my own machine, not a
published benchmark. They are reproducible in method, not in absolute numbers:
your hardware, shell config, and corpus will move every figure. Treat the
deltas as the claim and the absolutes as context.
- [E1] — Shell and CLI startup timing, measured 2026-02-19. Before/after
pairs from
zsh -i -c exit,codex --help, andclaude --help, taken on the same machine in the same session, around one config pass. - [E2] — macOS launch-agent load check, 2026-02-20: 8 of 8 agents loaded.
- [E3] — Memory-tier extraction totals, 2026-02-20: 108 daily files spanning 2025-09-27 to 2026-02-20, yielding 7,596 prompts and 22,858 tool calls.
- [E4] — Retrieval benchmark A, run 2026-02-20T04:16:45Z.
- [E5] — Retrieval benchmark B, run 2026-02-20T04:56:59Z. Compared against [E4] to surface drift; the triage count moved 0 → 2 while gates stayed green.
- [E6] — Latest retrieval benchmark, run 2026-02-20T06:58:19Z, on
nomic-embed-text:latestatvector_dim=768with dense retrieval enabled. - [E7] — Graph-alignment pass following the same run, checked by inspection rather than by a scored metric.
Single-machine numbers. If you run this sequence and get different absolutes, that is the expected outcome — the point is that you now have baselines to compare against at all.



