We Gave Our Agent a Filesystem Instead of a Vector Database

Almost every AI product solves memory the same way. Take a document, chop it into chunks, turn each chunk into a vector, and when a question comes in, find the chunks whose vectors sit closest to it. That is retrieval-augmented generation, and it is the default for good reason. It works, it scales, and every framework ships with it.

We went the other way. Silvia’s agent gets a real filesystem and the same commands an engineer would use. It runs grep, find, and cat against markdown files. No embeddings, no chunking, no vector index.

This is why.

What breaks when you chop up financial data

The failure modes of chunking are well known in the abstract. In finance they are specific and expensive.

A cost basis table gets split across a chunk boundary. Retrieval returns the half with the header and none of the lots, or the lots without knowing which account they belong to. The model does not know it is holding half a table. It does the math anyway.

Then there is exact match. Ask about section 1202 and similarity search will happily hand you a passage about qualified small business stock that never states the section, while missing the one that does. That is the behavior embeddings are built for. They map things that mean the same thing to the same place. In finance, the things that must be exact are exactly the things that get smoothed away: a section number, a ticker, an account, a figure with a date attached.

And you only get one guess. Retrieval runs, some chunks come back, and the model answers with whatever it was handed. If the right passage scored eleventh and you asked for ten, nothing tells you it is missing. The answer arrives with the same confidence either way.

RAG solves a problem that mostly went away

Here is the part that gets missed.

RAG was designed for a world where the model was weak and the context window was small. You had to guess the exact right paragraphs up front, because that was all you could fit and you got one attempt. Every piece of machinery in a RAG stack, the chunking strategy, the reranker, the hybrid search, exists to make one guess as good as possible.

A modern agent does not work that way. It can run a search, read what came back, decide that was not it, and search again with a better term. Once the thing can look around, you do not need to guess right the first time. You need to give it a room to look in.

So we stopped building a better guesser and built a better filing cabinet.

How it works

Every user gets their own isolated filesystem. Part of it holds a shared library of platform knowledge, part of it is their personal memory, part of it holds documents they have uploaded.

The agent reaches it through a single tool that can search, write, and delete. Search runs a real command against real files. The command set is small and read-only by design: enough to find things, read them, and slice them, and nothing that can modify the system or reach the network. Commands run as an argument list rather than through a shell, so the usual tricks of chaining and redirection are not available at all.

Writes are narrower still. Small markdown files with required metadata, a bounded directory structure, and atomic replacement so a failed write never leaves a half-file the agent will later read as fact. Isolation between users is enforced at more than one layer rather than relying on a single check.

None of that is exotic. That is the point. It is a filesystem with the sharp edges taken off, and the agent uses it the way you would.

Why the filesystem is not inside the sandbox

There is a second choice buried in that description, and it matters more than it looks.

Agent harnesses are converging on a standard shape: give the agent a code execution sandbox, and put the filesystem inside it. That is clean, and it is how most of the tooling works. It also means the filesystem is attached to compute, so a persistent filesystem implies a persistent sandbox, and a sandbox per user implies paying for a container that sits idle almost all of the time. Anyone who has run this at scale ends up building elaborate machinery to hibernate and revive those containers, because the ratio of active to total is brutal.

We kept the two apart. The files live in storage that costs nothing at rest and scales without anyone thinking about it. The execution layer is a small stateless pool that assembles a user’s view on request and throws it away afterward. Nothing persists in the compute, so any worker can serve any user.

The consequence is that capacity scales with concurrent requests rather than with the number of users. A search takes milliseconds, so nothing has to stay warm for anyone, and a modest pool serves everybody.

The industry question has been how to make per-user sandboxes cheap. We think the better question is why the filesystem needs to be in the sandbox at all.

Reading before answering

The first rule in Silvia’s system prompt is not about tone or format. It is that the agent must search the user’s personal memory before responding to any message. Not when it seems relevant, not on complex questions. Every turn, including greetings, before anything else happens.

It reads a navigation file that maps what is stored where, then searches for terms from the message, then applies what it finds without announcing that it looked.

That rule is only affordable because retrieval is cheap. A search against a mounted filesystem costs milliseconds and no tokens. If every turn required an embedding call and a vector query, “check memory before every response” would be a latency budget nobody would approve.

What the agent writes down, and what it must not

Reading is half of it. The agent also writes.

Explicit instructions get saved immediately. Say “remember this” or “always do X” and it goes in a file. When the agent infers something instead, because you corrected how it formatted a report or mentioned a fact that will not change soon, it confirms once and then saves. One file per topic, stable names, no dated copies competing to be the source of truth.

The more important rule is about what memory must never hold: anything the database can produce on demand.

A note saying “IRA cash is $84,000” is wrong the moment the next trade settles, and worse, it will get trusted over the live feed later. So memory stores the durable things. Your target allocation. The fact that a withholding choice was deliberate. That you plan to retire at 63. Numbers come from the database every time, and when a file and the database disagree, the database wins, and the file is stale.

Getting that boundary right matters more than the retrieval mechanism. Memory that quietly becomes a second source of truth for numbers is worse than no memory at all.

Research that stays around

When a deep research session finishes, its outputs are written into the user’s filesystem alongside everything else. A report produced last month is one search away in today’s conversation.

This is where the choice compounds. There is no separate ingestion step, no re-indexing, no waiting for embeddings to land before the work becomes findable. The report is a file in a directory. The agent already knows how to read files in directories.

What we gave up

Exact search needs the right word. If a document says “qualified small business stock” and the agent searched for the section number, a plain text search will miss what a semantic one might have caught. That is a real cost, and we pay it.

What makes it survivable is that the agent gets more than one attempt. It can check the index, try a different term, list a directory, follow a lead. That is the same tradeoff a person makes going through a filing cabinet, and it is only available because the agent runs in a loop rather than answering from a single retrieval.

We should also be precise about scope. This is how the agent’s own memory works. Embeddings still exist elsewhere in the product, including conversation history recall. For a bounded, per-user knowledge base that one agent maintains and rereads constantly, a filesystem is the better tool. For other problems it is not.

The short version

An agent that can act in a loop does not need you to guess which paragraphs it will want. It needs a place to look and permission to look more than once.

We gave ours a filesystem, a handful of commands, and a rule that it has to read before it speaks. That turns out to be most of what memory is.

We built this layer for Silvia first. It is now available on its own as Trove, if you want to give your own agent a filesystem instead of a vector database.

Separate Answer Quality From Evidence and Control

A fluent financial answer can still be incomplete, outdated or wrong. Evaluate accuracy separately from grounding. Accuracy asks whether the conclusion is correct. Grounding asks whether the answer identifies current, relevant sources and whether those sources actually support the claim being made.

The system should also expose its limits. Check whether it can distinguish facts from assumptions, handle account-specific data without inventing missing details, and show the calculations behind an output. For consequential decisions, verify dates, jurisdiction and source material before relying on the result.

Data controls matter as much as answer quality. Review connection permissions, encryption, retention, export and deletion options, human access and what happens when an account is disconnected. Good financial software should make both the analysis and the handling of sensitive data understandable.

Questions to work through

• Ask for the source, effective date and assumptions behind important claims.
• Recalculate material figures independently before acting.
• Review permissions and data controls before connecting financial accounts.
• Use qualified professional judgment when tax, legal or fiduciary duties apply.

Put the decision in your complete financial context

Silvia can bring this analysis back to the household data that gives it meaning. Review your investment research alongside your Ask Silvia so the decision reflects current balances, ownership, liquidity, obligations and goals rather than one isolated number. Use the resulting view as a structured starting point, then confirm important tax, legal or investment decisions with the appropriate qualified professional.

Related Silvia guides

• The Claude Code of Finance: What a Personal Financial Agent Should Do