"Can we have a ChatGPT for our internal documents?" is one of the most common requests we hear. The technique behind it — retrieval-augmented generation, or RAG — is well understood: find the passages most relevant to a question, give them to a language model, and ask it to answer using only that material.
A first version can be running in an afternoon. Getting from there to something employees rely on every day takes considerably more care. The gap is almost never the model. It's the content, the retrieval, the permissions and the feedback loop around them.
Garbage in, confident garbage out
A knowledge assistant is only as good as the knowledge it can reach. Most companies have several versions of the same policy, wiki pages nobody has touched in years, and critical know-how that exists only in chat threads. Point a RAG system at all of that and it will happily quote an outdated document with total fluency.
So the first workstream is curation, not code. Decide which sources are authoritative for which topics. Archive or label what is stale. Identify the gaps where people currently ask a colleague because nothing is written down — those are often the highest-value documents to create.
Retrieval is where quality is won
When an answer is wrong, the cause is usually that the right passage was never retrieved. A few practices make a large difference:
- Chunk documents along their natural structure — headings, sections, list items — rather than by arbitrary length.
- Keep useful metadata with every chunk: source, owner, last updated date, product or region it applies to.
- Combine semantic search with keyword search, so exact terms like product codes and policy names aren't lost.
- Re-rank the candidates before they reach the model, and send fewer, better passages rather than many mediocre ones.
None of these are exotic. They simply require treating search as a product in its own right, with its own tests, rather than as plumbing.
Permissions are not optional
An internal assistant that can surface any document to anyone is a data-leak waiting to happen. The rule we follow is simple: the assistant should never retrieve something the person asking couldn't open themselves. In practice that means carrying access-control information into the index and filtering at query time, using the identity systems you already have.
This is also where trust is built or lost. The first time someone sees a salary spreadsheet in an answer, the project is over. Getting permissions right from day one is far cheaper than recovering from that moment.
Show your sources
An answer without a source asks for faith. An answer with a source invites verification — and verification is what turns scepticism into trust.
Every answer should cite the passages it relied on, with links people can click. This does three things at once. It lets users check the answer in seconds. It makes errors visible and attributable to a specific document, which tells you what to fix. And it nudges the system's design towards grounded answers, because ungrounded ones have nothing to cite.
Equally important is the ability to say "I don't know". An assistant that declines when retrieval comes back empty is more useful than one that improvises. Make refusal a tested behaviour, not an accident.
Measure, then keep measuring
Build an evaluation set early: a few dozen real questions, written by the people who will use the assistant, with the documents that should answer them. Score retrieval (did the right source come back?) separately from generation (was the answer faithful to it?). Run the set every time you change chunking, prompts or models, so improvements are real and regressions are caught.
After launch, add lightweight feedback — a thumbs up or down and an optional comment — and review it weekly. Patterns emerge quickly: a missing document, a confusing policy, a type of question the assistant should route to a human instead.
Ownership keeps it alive
Knowledge changes constantly, so an assistant without an owner decays. Assign responsibility for sources, give document owners visibility into how their content is used, and schedule regular reviews. The technology is the easy part. The habit of keeping knowledge current is what makes the assistant worth trusting a year from now.