Cheaper, local LLMs change how AI searches your internal files: tree search vs vector search

Why did we run this test?

A common client request sounds simple: “We have thousands of internal PDFs and files. Let us ask questions and get correct answers with page references.” Most teams reach for one of three designs:

• Vector search. Cut the documents into passages, find the passages closest in meaning to the question, and give them to the AI.
• Search-then-read. Let the AI pick the relevant documents first, then search inside those.
• Agent search. Let the AI look at the table of contents, choose sections, read those pages, and keep going until it can answer.

Each design has a price and a failure mode. We wanted numbers for our own documents, not a general claim.

What did the test cover, and how was it scored?

• 83 questions written to resemble what a finance or compliance team asks: table lookups, calculations across several filings, comparisons between companies, chart readings, and questions the documents cannot answer, where the right response is to say so.
• 20 documents: 4 Hong Kong annual reports (K. Wah) and 16 US filings (12 quarterly reports and 4 annual reports from large US companies).
• Each question has a known correct answer and a known location in the document.

Scoring ran in two steps. A program first checked each answer against the expected figure or phrase. A second AI model then read the answers and corrected obvious grading mistakes, such as a correct number written in a different format; reviewed scores were up to 8.5 points higher than automatic ones. The reviewer was an AI, not a person, and a human re-check of a sample is still to be done.

The main methods were run three times. A single run is noisy by about 6 points, so we compare single runs only loosely. Costs use the vendor’s public price and have not yet been checked against the actual invoice.

Does the way you read the document matter more than the search method?

Yes. Every method starts from text, and there are two ways to get it: the PDF’s own text layer, which is fast and free but jumbles tables and often misses chart values, or an AI that reads each page and writes out the text, rebuilding tables cell by cell and writing the numbers printed on each chart.

For the same search method, switching to AI-written text raised accuracy by about 6 points (agent search with our own tree builder) to about 16 points (agent search with an off-the-shelf tree builder). Reading all 1,755 pages this way cost about $2.24, paid once.

A search method cannot find an answer the text never contained. A chart that reads “Revenue: 1,372” in the AI’s text can be searched; a chart with no text in the PDF layer cannot.

Does attaching the original chart image help?

No. We tried cropping only the detected figures and tables and attaching them to the question, and attaching whole page images for the pages that were picked. Neither improved results. Correct answers out of 83, across three runs:

The AI-written text already carries the chart values, so the image adds little and costs about 55% more when nothing is cached.

How well does plain vector search do?

It is a strong, cheap baseline. Vector search that keeps the 40 closest passages, packed into a budget of about 12,000 tokens (roughly 27 passages), scored 83.1% across three runs at about $0.003 per question. The 40 is the number of candidates, not the number of passages sent to the AI.

Search-then-read scored 82.7% at about the same cost. The two are statistically level: search-then-read picked the wrong document on some questions, and vector search missed the right passage on others.

Does tree search score higher than vector search?

Yes, by about 7 points. We call this tree search: the document becomes a table-of-contents tree, and the agent works the way a person would. It reads the table of contents, picks the sections most likely to hold the answer, reads those pages in full, and either answers or goes back and picks again.

With AI-written text it scored about 90% (90.4% in the original run; 76 and 75 correct out of 83 in two later runs) at $0.02 to $0.03 per question. Building the table of contents is a one-time cost across the 20 documents: about $2 with the off-the-shelf builder and about $4 with our own.

The gain is clear but not large at our noise level. The agent is also slower, about 12 seconds per question, and can run out of room on questions that need information from every document at once.

How do the search methods compare?

A fourth option is to skip search and put whole documents into the LLM’s prompt. We did not test it here, and published research suggests it is not a free alternative: in Lost in the Middle (Liu et al., TACL 2023), model accuracy fell when the relevant passage sat in the middle of a long context and as the context grew. Retrieval is still needed; the question is how much the LLM reads after it.

Vector search is the retrieval step in the original retrieval-augmented generation design (Lewis et al., 2020), built when passing more text to the model was costly. The comparison above tests whether that constraint should still decide the design.

What is the best use case for vector search, and for tree search?

Choose by what a wrong or missed answer costs, and by how many questions you ask. These are our recommendations from the test, not guarantees on every document set.

Many firms will use both: vector search to route a broad question to the right documents, then tree search to read them closely. We have not tested that combination in this study.

Do Chinese questions score lower than English ones?

About the same. We translated all 83 questions into Traditional Chinese, back-translated them to check the meaning, and fixed 13 that had drifted. On automatic scoring, vector search scored 60 to 62 out of 83 in Chinese and 63 to 66 in English; the other methods were level or slightly better in Chinese. The gap is mostly on questions that ask for a list. We did not run the AI review on the English control, so compare the automatic scores only.

Which ideas did not help?

We also tried designs that build summaries of document sections and search across those summaries. None beat plain search in our tests. These are our own simplified versions, not the published originals, so they are not a verdict on the original methods.

Why does a cheaper, local LLM change how you search?

Vector search was designed around a limit: LLM reading was slow and expensive, so you pre-select a few passages by similarity and hope the right one is among them. Its failures come from that guess. The right passage is not among the 40 closest, a table is split across passages, or the wrong fiscal year ranks first.

Tree search removes the guess. The LLM reads the table of contents, chooses sections and reads those pages in full. It scored about 7 points higher in our test, and it costs more tokens per question: roughly $0.02 to $0.03 against $0.003 on a hosted API.

That cost gap is why vector search stayed the default. It is also the part a local deployment changes. When the LLM runs on hardware you own, a question that reads more pages costs more compute time, not a larger per-token invoice, and your internal files never leave your network. For law, audit, tax, banking and family-office work, where a wrong or uncited answer is expensive and the documents are confidential, that is the trade we recommend looking at first.

What do we know about open-weights and private models so far?

Every test used Ultramarine 1 Flash from Paper Instruments. It is a post-trained version of Z.ai’s GLM-5.3 Flash. Paper Instruments announced it on 14 October 2026, and as of 27 October 2026 its Ultramarine page lists the model weights as “coming soon”. We have not checked the license. The vendor’s benchmark figures have not been independently reproduced, and we did not use them: every number on this page comes from our own 83 questions. The embedding model for vector search is OpenAI’s text-embedding-3-large, which is not open-source.

Every test ran through the vendor’s hosted API, not on our own hardware. The results show what the model can do on tree search, not what a private deployment would cost or how fast it would run. We plan to repeat the test on a self-hosted model, and until then we make no claim that it is cheaper or as accurate.

What are the limits of this test?

• AI review, not human review. A person should check a sample of the disputed answers.
• Some results come from a single run. Treat differences under about 6 points as unclear.
• Costs come from public prices and have not been reconciled with the actual invoice.
• One model, one document set. A different model or different documents could change the ranking.
• Our own versions of some methods. Where we built a simplified version of a published method, we say so and do not use the original name.
• Two early errors were fixed during the test. The cost estimate was about twice too high, and the 200,000-token limit we used was our own setting, not the model’s; we measured the model’s real limit as about 262,000 tokens.

What would we do with this?

• Start with clean document reading. Get tables and chart values into text before you spend on search.
• On a low budget, start with vector search. It is cheap, fast and accurate enough for many questions.
• For accuracy, add tree search and run the LLM on your own servers. Expect a setup cost per document set; on a hosted API, a higher cost per question.
• Check your own documents. Build a test set of 50 to 100 real questions and run these methods on it before you choose.

Want to try this on your own files?

We will run the same comparison on your documents and your questions: vector search against tree search, scored on a test set built with your team. If the results hold, we deploy the LLM and the search on your own servers, so confidential files never leave your network.

Book a call to scope a test, or read how we build private RAG on client infrastructure.