Why Agent Memory Is Not a Vector Database: The Decision Ledger Playbook in 8 Rules

Ask what agent memory means and you will hear vector database. I went looking for one inside a production agent wired to Claude, and there is not a single embedding in it.

What it has instead is a ledger. Memory there is not an index but a decision record: what changed, who changed it, and whether they were allowed to. I did not write one for two years of agent work, and the drift stays silent until someone asks.

So what does a missing ledger actually cost you? Here is the exact 5-record runbook that agent runs on, pinned to commit 1718628, and why the approval gate comes first.

Tasks persisted as records. Approvals gated release. Audit logs captured every mutation. Retrieval ran on SQLite FTS5 and graph stats, and nothing else, so five record types were carrying the entire thing.

That is the part nobody sells you. Teams ask for memory and buy an embedding store. They still cannot answer the question that ends an argument in production: what did this agent decide last time, and was it allowed to?

An index cannot answer that. A ledger can.

Save this. It is the Decision Ledger Contract: the five record types, the two retrieval paths, and the approval gate that has to sit between them. It records decisions, it tracks status, and it preserves receipts.

It does not need a chatbot brain with a cosine novelty layer.

The schema already tells you what matters

Start with src/lib/schema.sql. The tasks table persists these columns:

title, description, status, priority, assigned_to, created_by,
created_at, updated_at, due_date, estimated_hours, actual_hours,
tags, metadata

Not a chat transcript. A state ledger.

What matters is that the row carries operational meaning. Each field is doing a job:

Then the migrations go further. src/lib/migrations.ts adds outcome, error_message, resolution, feedback_rating, feedback_notes, retry_count, and completed_at, which is the line where memory stops being a blob and becomes a ledger. A blob stores similarity; a ledger stores consequences.

If a task failed, the code wants to know why. If it was retried, the code wants to know how many times. If it finished, the code wants to know when and with what outcome. That matters because a production agent does not only need to recover context.

It needs to recover judgment.

The schema is already doing the harder job: it is preserving the decision surface.

And one more detail matters: assigned_to exists, and no separate public owner field exists. Ownership is operational, not decorative. The work is attached to an agent identity, not a vanity label.

Two retrieval paths beat one vector search

The other mistake is thinking retrieval equals embeddings. It does not. src/lib/memory-search.ts shows the actual memory surface: filesystem plus SQLite FTS5 plus graph stats, and that gives you two retrieval paths.

One path is text recall. FTS5 can find the file, the paragraph, the phrase, the note. The other path is structural recall, where graph stats and filesystem layout tell you what kind of thing you are looking at and where it lives in the operator stack.

That split matters. A vector database is good at approximate similarity, and it is useful when the question is fuzzy and the corpus is large. A decision ledger needs more than fuzzy similarity.

It needs exact precedent.

It needs the ability to say, "show me the last time we approved this kind of task," or "show me the file that owns this workflow," or "show me the note that explains why the retry count exists." That is why the public memory stack in this codebase is not a vector database story.

It is a retrieval story.

If I am looking for the decision, I want the original record. If I am looking for the reason, I want the trail. If I am looking for the precedent, I want the exact artifact that changed state. Vector search can sit on top of that; it should not replace it.

Where the tiered memory systems are right

This week the timeline is full of memory systems. Local-first stores that run without a cloud bill. Tiered designs that move facts from a hot working set down through warm layers into cold storage, with pruning and consolidation passes running in the background. Self-hosted stacks that promise an agent which remembers your work.

I want to be precise about this, because the easy move is to dunk on all of it, and that would be wrong. Those systems are solving a real problem, and they are solving it well.

If your agent re-reads the same file every session, a tiered store fixes that. If your context window is stuffed with material the model already saw, consolidation fixes that. If you are paying to send the same background twice, local-first fixes that. Those are genuine wins and I would take them.

The gap is what they are optimising for.

Every one of those designs optimises recall volume. How much can the agent get back, how cheaply, how fast. That is a retrieval problem, and retrieval problems have retrieval answers.

None of them answers the question that ends an argument in production: what did this agent decide last time, was it allowed to, and can I prove it.

A tier tells you a fact was stored. It does not tell you whether the fact was authorised, who acted on it, or what happened afterwards. Pruning makes that worse rather than better. A consolidation pass that compresses ten sessions into a summary has just destroyed the provenance of every decision inside them, and it did it silently, because compression is the feature.

So the two are not competitors. They sit at different layers:

If you only build one, build the ledger.

A missing retrieval layer makes the agent slow. A missing decision layer makes it unaccountable, and you will not notice until someone asks why the system did something.

Identity and scope are part of the record

A ledger is only useful if it knows who acted, and this code does. The runtime identity includes workspace-local agent name, global session key, working_memory, runtime_type, and Claude session IDs. That is not trivia. It is the binding between a decision and the actor that made it.

src/app/api/tasks/[id]/route.ts and the related enforcement helpers show RBAC and agent/task scope checks:

• requireRole gates the route.
• requireWorkspaceId scopes the workspace.
• requireAgentTaskAccess blocks access when the caller is not allowed to touch the assigned work.

That means memory is not free-floating. It is bounded, it is permissioned, and it is tied to identity and scope before it is tied to recall.

And the done path is not a vibe either. The route checks for Aegis approval before a task can move to done. The changelog is explicit: no generalized approval-policy engine yet. That matters. The code has execution approvals and allowlists, but it does not pretend to have a universal policy brain. It has specific gates for specific actions, which is the right shape.

A decision ledger should not guess who may mutate state. It should know, and it should say no when the actor is wrong, say no when the approval is missing, and say no when the scope is off.

That is how memory becomes enforceable.

Without that, memory becomes notes with a confidence problem.

Audit and receipts turn precedent into something you can trust

A ledger without audit becomes a diary, and this codebase goes further. src/lib/mcp-audit.ts records tool calls in mcp_call_log, and src/lib/receipt-signing.ts signs audit payloads with Ed25519 and verifies them later. That is the difference between "I think this happened" and "this is the record of what happened."

The receipt path is simple and strong. The payload is canonicalized, the hash is computed, the signature is created, and verification recomputes the hash and checks the signature again. If the payload changed, verification fails. That is not cosmetic.

It is evidence quality.

The same pattern shows up in the event surfaces. The public surface includes activities, audit_log, security_events, mcp_call_log, and SSE events. The code is not hiding behind a single black box. It is emitting history from multiple angles.

That is what a decision ledger needs: activity history for motion, audit history for authority, security events for trust, and tool-call receipts for provenance. If you are building agent memory and none of your records are signed, scoped, and replayable, you are leaving the hard part out.

You are building recall, not evidence.

Retries are not a vibe. They are state

One of the strongest signals that this system is a ledger is the way it handles retry and recovery. src/lib/task-dispatch.ts does more than send work once and forget it. It can reconcile deferred completions, extract completion text from dispatch payloads, route tasks through review and then into quality_review, recover after rejection, and record the failure path.

That matters because failure is not an exception to a ledger.

Failure is one of its main use cases. The migrations add retry_count and completed_at for a reason, the route logic can force quality_review before done, and the dispatch layer can promote or recover state.

The interesting question is not "did the agent answer?" The sharper question is "what state did the answer create?"

That is the core of a decision ledger.

A failed task with a clean error message is useful. A rejected task with a retry count is useful. A completed task with a timestamp and resolution is useful. A task that can be traced from assignment to approval to outcome is useful. That chain is better than an embedding that vaguely resembles a prior prompt.

The vector store can help you find the note. The ledger tells you whether the note should have changed the world.

The Decision Ledger Contract

This is the contract I would keep in my head when I design agent memory.

1. A record must say what changed.
2. A record must say who changed it.
3. A record must say when it changed.
4. A record must say why it changed.
5. A record must say what happened next.
6. Retrieval must find both the record and the precedent.
7. Scope and approval must be checked before mutation.
8. Receipts must be verifiable after the fact.

That is the Decision Ledger Contract.

It is simpler than a giant memory architecture and stricter than a loose note pile.

Here is the same contract in one block, ready to paste into a design doc:

Decision Ledger Contract
1. record what changed
2. record who changed it
3. record when it changed
4. record why it changed
5. record what happened next
6. retrieve the record and the precedent
7. check scope and approval before mutation
8. verify receipts after the fact

Every line above is checkable against the public code. These are the files it lives in:

Mission Control evidence set
- src/lib/schema.sql
- src/lib/migrations.ts
- src/lib/task-dispatch.ts
- src/lib/receipt-signing.ts
- src/lib/mcp-audit.ts
- src/app/api/tasks/[id]/route.ts
- src/lib/memory-search.ts
- app/CHANGELOG.md

It gives you five record types that matter most here:

• task rows for current state
• migration fields for outcome and recovery
• runtime identity for actor binding
• audit receipts for tool provenance
• event surfaces for operational history

And it gives you two retrieval paths that matter most:

• search the text and file system for precedent
• inspect the ledger and event stream for state

That is enough to keep an agent honest without pretending that memory is magic. It also scales better than the "store everything in embeddings" instinct, because not everything is meant to be remembered the same way. Some things are recall, some things are policy, some things are receipts, and some things are state. If you collapse them into one bucket, you make the agent clever and the system fragile.

How to add a ledger to an agent you already have

You almost certainly are not starting from an empty repository. So here is the order I would work in, on a system that is already running.

1. Find the mutation points first. Every place the agent changes state outside its own process: writes a file, calls an API, posts, pays, deploys. That list is usually shorter than expected, and it is the part I would give a ledger on day one.
2. Give each mutation an id before you give it a record. If the same intent can be submitted twice, nothing you write down afterwards will be trustworthy.
3. Write the five fields. What changed, who changed it, when, why, and what happened next. Put them in whatever store you already have. A table is fine. A JSONL file is fine. The schema matters more than the technology.
4. Record the refusals too. A ledger that only contains successes is a highlight reel. The entry where the agent was blocked is the one you will want during an incident.
5. Reconcile against the outside world on a schedule. Compare what the ledger says happened with what the external system says happened, and treat a mismatch as a stop condition rather than a log line.
6. Only then add retrieval. Text search over the ledger will answer most questions. Add embeddings when you have a concrete question that text search demonstrably failed to answer.
7. Put an expiry or a review date on anything that describes a moment rather than a rule, so the ledger does not quietly become an archive nobody trusts.

Steps 1 through 3 are usually an afternoon. Step 5 is the one teams skip, and it is the one that catches the failure you cannot reason about from inside the process.

When a vector database is genuinely the right call

I said the ledger comes first. That is not the same as saying embeddings are decoration.

A real set of jobs exists where vector search is the correct tool and a ledger would be the wrong one:

The failure mode is not using embeddings. It is using them for the one class of question they cannot answer, and then discovering that during an incident.

The tell is simple. If the answer needs to be defensible, it belongs in the ledger. If the answer only needs to be useful, retrieval is fine.

Limitations and what I am not claiming

I am not claiming vector search is useless. I am saying it is not the source of truth for this class of system.

I am not claiming Mission Control has a generalized approval-policy engine. The changelog marks the generalized approval-policy engine as future work.

I am not claiming the entire test suite is green. My local proof run was focused. This is the command I ran:

pnpm vitest run src/lib/__tests__/task-dispatch-reconciliation.test.ts

It passed 12 tests with 0 failures. A separate mixed test run hit a local harness cleanup issue, so I am not claiming the full suite is clean.

I am also not claiming this is the only valid architecture for agent memory. It is the one this codebase already points toward, and that is the key part. The public artifact already separates retrieval, authority, audit, and recovery, and the code already treats task state as a ledger.

The safe move is to lean into that shape instead of importing a fashionable database term and hoping it solves governance.

Bottom line

Agent memory is not a vector database. It is a decision ledger with retrieval on top.

If the system can tell you what happened, who did it, why it changed, and whether the record is trustworthy, you have memory that can survive production. If it can only tell you "this looks similar," you have a search box wearing confidence cosplay.

Build the ledger first. Add recall second. Keep the receipts.

Run proof, not folklore.

Get the next field note

I publish practical field notes for builders running agents in production - what shipped, what broke, and the system behind it.

Get the next one free: https://nyk.dev/#newsletter

Free. Unsubscribe anytime.

Join the private NYK alpha channel for early notes and updates: https://t.me/+GJ-FEpzcZrtmMTky

Follow @nykdotdev, then pick one: which ledger record would you instrument this week, decision or outcome?