Try asking a general-purpose AI assistant what your internal returns procedure is, or what the latest version of a tender specification says: it cannot know, because it has never seen those documents. At best it admits as much; at worst it makes up a plausible answer.
The technique that solves this is called RAG, short for retrieval-augmented generation. Before answering, the system searches your documents for the relevant passages and hands them to the language model (the LLM, the engine behind tools such as ChatGPT, Claude or Gemini) together with the question. The model does not need to “remember” anything: it reads the right pages at the moment it answers.
Here is how it works step by step, why it is usually a better bet than retraining a model, and what separates a prototype that impresses in a demo from a system people can rely on every day.
How it works, step by step
The work splits into two phases: preparing the documents, which is done once and then kept up to date, and answering, which happens with every question.
- Collection: PDFs, wiki pages, manuals, tickets, emails. The text is extracted and useful metadata is kept: title, date, version, who is allowed to read it.
- Splitting into pieces (chunking): each document is cut into sections a few paragraphs long, small enough to be precise and large enough to make sense on their own.
- Embeddings: a dedicated model turns each piece into a vector, a long sequence of numbers that represents its meaning. Texts about the same thing end up with vectors close to each other, even when they use different words.
- Storage: vectors, text and metadata go into a vector database such as Milvus, built to find the most similar items quickly, even among millions of entries.
- Retrieval: when a question comes in, it is turned into a vector too, and the database returns the closest pieces, which are the most relevant ones.
- Answer: the language model receives the question and the retrieved pieces, with the instruction to answer only on that basis. Because every piece carries its source document, the answer can cite its sources and readers can check them.
Why not “teach” the documents to the model?
The first alternative that comes to mind is fine-tuning, that is, retraining a model on your own documents. For company knowledge, RAG is the better choice in most cases:
- Updates: a new or corrected document can be in the index within minutes, whereas a fine-tuned model would have to be retrained.
- Sources: RAG shows where the answer comes from; a model that has “learned” something cannot tell you where it read it.
- Permissions: you can filter what is retrieved based on who is asking. Information absorbed into the model, on the other hand, cannot be hidden from a user.
- Cost and freedom: there is no training run to pay for, and you can switch language model without redoing the work on your documents.
Fine-tuning is still useful for other things: teaching a style, a response format or a very specific task. When it makes sense, the two techniques can be combined.
What determines answer quality
The model is the part that grabs attention, but quality is decided mostly beforehand, in preparation and retrieval. These are the factors to get right:
- Clean documents: outdated versions, duplicates and scanned PDFs with no text layer produce answers that are stale, contradictory or empty. Tidying up is often the first job.
- Sensible splitting: cutting a table in half or separating a heading from its paragraph makes retrieval worse. It pays to follow the document’s structure (sections, headings, lists), not just a character count.
- Well-tuned retrieval: combining search by meaning with keyword search (hybrid search) helps with codes, acronyms and proper names; a second pass that reorders the results (reranking) picks better pieces to pass to the model.
- Permissions enforced: the system must apply the same access rights as the original documents. The filter belongs in retrieval, not in instructions given to the model.
- An up-to-date index: when a document changes or is deleted, the index has to follow through an automated process, not by hand.
- Evaluation with real questions: you need a list of questions gathered from the people who will use the system, with the expected answers. You measure whether retrieval finds the right pieces and whether the answer is correct and faithful to the sources, and you repeat the test after every change.
Typical tools
A RAG system is built from a few building blocks, and for each one there are both open-source components and managed services:
- Orchestration: frameworks such as LangChain connect the steps (loading, splitting, retrieval, calling the model) and let you replace one component without rewriting the rest.
- Vector database: Milvus is open source and can be installed on your own servers or used as a managed service.
- Embedding models: Hugging Face hosts many of them, multilingual ones included, to run in-house; OpenAI and Google also offer them through their APIs.
- Language model: OpenAI, Anthropic’s Claude or Google’s Gemini through their APIs, or a model running on your own servers, for example with Ollama. The choice depends on confidentiality, quality and cost, and it is the subject of our article on running models in-house or in the cloud.
A practical note: if you change the embedding model, the vectors must be recalculated for every document. The language model, by contrast, can be swapped without touching the index.
Where to begin
A first project should have a small, clear scope: a set of documents with an owner, a group of users who currently waste time searching, and a list of test questions. With those three ingredients you build a pilot, measure it and decide whether to extend it. Indexing the entire company archive on day one, on the other hand, is the quickest way to get mediocre answers.
If you have manuals, procedures, contracts or tickets that people struggle to find their way through, we can start together with a pilot on a narrow scope and measure it against your real questions. Get in touch and let’s talk.
