Letting business users ask questions of company data, without the AI making things up

Why does natural-language access to data go wrong?

Almost never because the model cannot reason. It goes wrong because the model is asked to do three jobs at once: understand the question, find the right data, and produce the exact query or fact. The first job is what language models are good at. The other two need to be right every time, and a model that is right 95% of the time is wrong in front of a client every week.

We have built this pattern over four very different kinds of data, and the failure always arrives the same way: a confident answer built on a column the model guessed, a table it read as prose, or a code it half-remembered.

What should the model do, and what should code do?

Draw the line before writing a prompt. The split we use:

On a deal-search build this meant the model never wrote SQL at all. It produced a filter plan such as {"primary_asset_type": "Industrial", "proposed_loan_to_value__lt": 70}, a schema rejected any field that does not exist, and version-controlled code turned the plan into a read-only query. Same question, same SQL, same rows, every time.

How do you close the vocabulary gap?

Before the system answers a single question, profile the data and describe it. For a database that means every column: type, null rate, cardinality, value ranges, top values. A model then writes a one-line description and a handful of synonyms per column at temperature zero, so “loan amount”, “LTV” and “southwest” resolve to real fields and real values.

Retrieval works from that profile, not from the raw schema: an entity index for exact names, vector search over the descriptions, and fuzzy matching for values (“york” finds New York). The output is a short summary of which tables, columns and values the question needs, and it is handed to the step that plans the query.

For EDI, the vocabulary gap is between teams: the business side says “invoice 4471”, the EDI side says “810”. Mapping business terms to transaction types is the first step, and a model fine-tuned on EDI vocabulary does the translating back into English.

How should documents with tables and charts be parsed?

By layout, before anything reaches a vector store. Detect the regions on each page: text, table, figure, caption. Crop each table and chart with its caption, read it with a vision model, and index the result alongside the image path, page number and file name. Text is chunked by meaning rather than by character count.

Two things follow. A question about a chart can be answered at all, and every answer can point to the page it came from. On a compliance library this was the whole point: an answer an auditor cannot cite is a liability, not an answer.

Retrieval then blends keyword and semantic search (we weight roughly 30/70), because names, codes and rule numbers are exactly what pure embeddings miss.

What if users ask vague questions?

They will, and retrieval suffers more than the model does. For an asset manager’s research assistant we benchmarked three rewriting strategies, each measured on what it retrieved against the unrewritten question:

No single strategy wins. Route by question type, and measure at retrieval rather than at the final answer.

How do you handle multi-part questions?

Real questions arrive as instructions: “summarise the risk updates, then put them in a table with Updated, Who is affected and Date.” A breakdown step splits these into sub-questions, and each sub-question is checked for relevance to the original before it runs, so the chain does not wander. An agent picks the tool for each part (retrieve, format as a table, search the wiki, calculate), and a final check asks whether the combined answer covers every part of what was asked.

For a table answer we score both the table and the plain retrieval answer for relevance and keep the better one, or refine the table from the retrieved text. The user sees one answer; the system has quietly picked between two.

How do you keep it trustworthy in production?

• Screen before retrieval. A safety check runs on every message before any data is touched.
• Label every source. Internal documents, the web and reference sources are tagged in the context and cited in the answer, never blended silently.
• Refuse well. When the data does not cover the question, say so. On an internal wiki bot, those refusals became the list of pages the team needed to write.
• Benchmark each component, not the whole. We test models side by side per step. On one build the larger model kept column descriptions, where quality drives matching, and a model about 90% cheaper took entity grouping, where it did as well or better.
• Log the plan, not just the answer. Parameters, the query that ran, the sources used. When an answer is questioned, you can show your working.