AI That Reads Your Own Documents: I Asked Mine 12 Questions and It Missed 3

If you want AI that reads your own documents, the setup is the easy part and almost every guide stops there. I have been running one since September 11, 2026. It holds 1,292 files and 3.2 million words, it answers in about 80 milliseconds, and keeping it current costs nothing. This morning I gave it twelve questions I already knew the answers to. It put the right note first on five of them and somewhere in eight results on nine of them. Three questions it missed completely, and not one of those three was the software's fault. Grading the thing and then fixing what the grade exposes is most of the job, and it is where my time goes when I do AI and data work for other small businesses.

Keeping three million words readable costs almost nothing

The upkeep on a document index is too small to budget for. A job runs on my laptop at 2:30 every morning, finds the files whose contents changed since last night, re-reads only those, and goes back to sleep. Across 27 logged runs the median took 12 seconds. Twenty-three of the 27 finished in under a minute, 14 finished in under 15 seconds, and the slowest one took 555 seconds. On a median night the job re-reads 0.96% of the notes, because that is how much of a knowledge base actually changes in a day.

The first build is the only expensive part, and expensive is the wrong word. Reading all 827 notes from scratch took 446 seconds. Since then the whole thing has grown to 1,292 files and 12,741 searchable passages, and it sits in a single database file of 83,808,256 bytes. That is 80 megabytes for the entire searchable memory of a business.

Money is not the constraint either. The model that turns text into numbers runs on hardware I already own, so the running cost is zero and no document leaves the building. I measured what the hosted version would cost out of curiosity. The index works out to about 5.59 million tokens, which I measured by running a sample through the model's own counter rather than estimating it, and OpenAI's published price for its small embedding model is $0.02 per million tokens. Reading all of it from scratch at that rate would run about 11 cents, or about 73 cents on the larger model. Eleven cents. If anyone quotes you a five-figure proof of concept, the quote is not about the reading.

So the cost question has a dull answer. The question worth asking is whether the thing finds what you asked for, and until this morning I had never measured that.

I asked it twelve questions I already knew the answers to

Nine of the twelve came back with the owning note somewhere in eight results, and on five of those nine it was ranked first. Three questions came back without the owning note at all. The test is simple enough that any owner can run it: write down twelve questions you genuinely ask the thing, decide in advance which document owns the answer to each one, then run them and check. Nothing about my index was changed for the test.

Being honest about the limits: the questions are mine, the grading is mine, and twelve is a small sample. One badly worded question moves the score by eight points. So treat 9 of 12 as a rough reading rather than a precise one. What it is good enough for is telling me that about one question in four comes back without the answer in it, and that is a number I did not have yesterday.

A miss rate around one in four also turns out to be normal, which I did not expect. A March 2026 study of chunking strategies across six knowledge domains found the best method put the right passage in the top five about 59% of the time, and ranked it first about 24% of the time. The worst method it tested ranked the right passage first 2 to 3% of the time. An independent evaluation across five corpora measured two things separately: how often the right text turned up at all, which ran between 85.4% and 91.9%, and how much of what came back was actually relevant, which ran between 1.4% and 8.0%. The second range is the one that matters. The right text is usually in there somewhere. Almost everything handed to the model next to it is noise.

The three misses had three different causes

None of the three failures was the search engine working poorly. Each one was a decision I made about what to put in front of it.

Somebody else's copy of my rule outranked my rule. I asked what the rule is about putting passwords in notes. The top three results all came from an open-source project I downloaded on July 8, 2026, which lives in a reference folder inside the same tree. One of them says "never write a password, key, or token value into a summary, setup doc, or note," which is close to what my own rule says. My rule first appears at rank 37. So the answer would have come back correct and sourced to a stranger, which is worse than it sounds. The day those two documents disagree, nothing tells me which one answered.

The wrong copies of the right number outranked the price list. I asked what my monthly retainer costs. Eight passages came back from seven different files, carrying 20 different dollar amounts between them: a prospect's hourly estimate, a client-specific scenario, hourly-rate math out of a decision memo, two quotes from a product line I no longer sell, a contract template. Exactly one of the eight held both of the numbers that are actually current, and that one was a working file out of my blog folder rather than the price list. The price list itself first appears at rank 13. A stale price is the expensive failure of the three, because nothing about it looks wrong. A confident answer built on a replaced document reads exactly like a confident answer built on the right one. Point that search at a customer-facing chatbot instead of at me and it quotes an old number in a clean sentence, and nobody catches it.

The right note sat one place outside the window. I asked what has to happen before I edit code on a live system. The file that answers it is ranked 9 and I ask for eight results. No lesson in that one beyond the obvious: a cutoff is a cutoff, and near misses are invisible from the outside.

Coursework is 27% of what my search reads. My business is 27.3%

University coursework takes up 27.0% of everything my search reads, 3,442 passages across 85 files, and my entire business takes up 27.3%, 3,472 passages across 335 files. I counted by folder, expecting the business to dominate, and found it tied with a certificate program. A third block, 1,560 passages across 53 files, is the saved daily output of an internal product I retired on September 23, 2026. That is 12.2% of the index describing something that no longer exists.

Add the coursework and the dead product together. 39.3% of what my own document search reads when I ask it a business question is schoolwork and an archive. Family notes are another 8.3%. The business is competing for attention against a stack of material that is either irrelevant or expired, and it loses about a quarter of the time.

Dead weight is not a software problem, and no model upgrade touches it. Deciding what belongs in there is a judgment call about your own business, which is probably why every guide that walks you through the upload skips it.

The search cannot tell you when it does not know

Every question gets eight confident-looking results, including questions with no answer anywhere in the files. I tested that directly. Three questions about things that appear nowhere in my documents (a truck warranty, a European capital, a dishwasher repair) scored 0.445, 0.479 and 0.439 on their best match. The twelve real questions scored 0.545 to 0.748, median 0.651.

So there is a signal. Real matches do score higher. But the gap at the margin is 0.066 between the worst genuine hit and the best invented one, and my search code uses no threshold at all. It sorts everything and hands back the top eight, every time, with no way to say "nothing in here answers that." A search that always returns something will sometimes hand over noise in exactly the posture it uses for a direct hit. A system that cannot say "I don't know" is the thing to understand before you let one answer a customer.

A bigger context window does not fix this

No, and the 2026 evidence on that is sharper than the 2025 evidence was. A context window is how much text a model can hold in view at once. The windows have gotten enormous, which makes "skip the search, paste everything in" sound like the simpler plan. Searching first and handing the model a short list is the approach with a name, retrieval. The alternative is handing it the whole filing cabinet and hoping.

A study published in May 2026 tested three current frontier models on spotting a subtly dangerous action inside a long transcript. All three missed those actions 2 to 30 times more often when the action sat after 800,000 tokens of ordinary activity than when it appeared on its own. In a June 2026 study of long-horizon search, models gave up or returned uncertain wrong answers well before exhausting the context window, and the rate of giving up early climbed as the context grew. The broadest look at the question, a benchmark of 2,326 test cases across eleven models, concluded there is no universal winner between retrieval and a long context window, because the right choice depends on the model, the task and the quality of what retrieval hands over.

The practical reading for a small business: a bigger window changes which problem you have, not whether you have one. Dumping everything in moves the filtering job from your search to the model's attention, and the model is worse at it than a short list of the right documents would be. The research on why AI gives generic answers about your business covers the other half of this, which is that no amount of context substitutes for the model having seen your actual work.

What to decide before you give AI your own documents to read

Five decisions, in the order they matter. Every one of them comes out of a number above.

1. Decide what is not allowed in. My deepest miss left my own rule sitting at rank 37, because a stranger's documentation was in the same folder tree saying nearly the same thing. Reference material you downloaded, vendor PDFs, and anything you did not write are the most likely things to outrank you.
2. Keep one copy of every number that matters. My price appears in prospect notes, proposals, contract templates and working files, and ten other files came back before the price list did. If you keep old versions for your records, keep them somewhere the search cannot reach.
3. Count what is in there before you trust it. One query against my own index told me 39% of it was dead weight. I had no idea, and I built the thing.
4. Give it a refresh job, not a refresh button. Mine runs at 2:30 AM and re-reads about 1% of the files in 12 seconds. A rebuild that depends on somebody remembering to press a button is a rebuild that happens once.
5. Grade it with questions you already know the answers to. Twelve questions and an afternoon gets you a hit rate, and without one you are trusting a number nobody has measured. The data your business already owns is the raw material. Whether the search can find the right piece of it on demand is a separate question, and it has an answer.

Giving AI your own documents to read is fast and nearly free to keep running. None of that is the hard part. The hard part is deciding what deserves to be in there, and then checking whether the thing actually finds it. I run that check on my own systems before I ask a client to trust one.