The most useful thing a company AI can say

A model with partial evidence answers anyway

Most company AI systems use retrieval-augmented generation, usually shortened to RAG. Before answering, the system searches your documents and hands the best passages to a language model. The model's job is to write fluent text, and it does that whether the passages are complete, partial or irrelevant. From the outside, a grounded answer and a fluent guess look identical. So the system has to catch missing evidence itself. The model won't do it.

Refusal needs a threshold

Retrieval returns a relevance score for every passage. Set a cutoff on it. Below the cutoff there is no answer: the system says the question isn't covered by approved documents and routes it to whoever owns the topic. Set the cutoff too high and it refuses questions it could have answered. Too low and the guessing comes back. Tune it on your own questions, not on a default.

Retrieval has to find the right page first

A refusal threshold is only as good as the search behind it. Two choices matter most.

First, search by meaning and by exact match at the same time. Meaning handles paraphrase, so "pump won't prime" finds a page about loss of suction. Exact match handles identifiers, because a part number or clause reference has no neighbor in meaning.

Second, chunk on the document's own headings. Each section becomes one chunk of roughly 500 tokens, a token being a small piece of a word. Tables and numbered procedures stay whole, since cutting them apart separates a value from the item it applies to. Stamp each chunk with its heading path so it carries its own context.

Labels and access are enforced before retrieval

Every chunk carries metadata: which document, which version (current or replaced), which product or site, and who may open it. The version label stops the system quoting last year's manual. The permission label does more. Access is checked in code before the search runs, so a restricted file is never retrieved and the model never sees it. Filtering after the model has read something is too late.

Every sentence cited, every number checked

The answer is written only from the passages that passed the threshold, and each sentence cites its page. A second pass, separate from the one that wrote the answer, checks every claim against its cited passage. Plain code then confirms that every number in the answer appears in the source text. If a check fails, the system retries once and then refuses.

Test it with questions it must refuse

An evaluation set is a list of real questions with the correct answer and the correct page. The part most teams skip is including questions the system should decline. Without them you measure how well it answers and learn nothing about how well it stays quiet.

Track four things: did it find the right page, is every claim supported, did it refuse when it should have, and did anything restricted leak. The last one has to be zero. Run the set on every change.

Then read the refusals each week. Every one marks a question people ask that no approved document answers, which makes the refusal log a list of procedures nobody wrote.

A version you can try today

No system can promise it is never wrong. What you can build is one that shows its evidence, so a person checks it in seconds.

The small version needs no engineers. Open a Project in Claude, upload a few of your own files, and add these standing instructions:

"Answer only from the files in this project. For every answer, quote the exact sentence and give the file name and page. If the answer isn't written in the files, say exactly: I couldn't find this in the files. Don't fill in from general knowledge. If two files contradict each other, show both and don't pick one."

Ask three questions you know the answer to and one you know isn't in the files. The last one is your refusal test. If Claude says it couldn't find it, the foundation is sound. If it invents an answer, tighten the instructions and ask again. Open the quoted page once or twice, because instructions improve behavior without guaranteeing it.