Skip to content
Knowledge Hub/Article
Article

What Is RAG (Retrieval-Augmented Generation)? How It Works

We explain how RAG works, embeddings and vector databases, hybrid search and reranking, how it compares with fine-tuning, evaluation and access security.

Oğuzhan Gerçek··10 min read
What Is RAG (Retrieval-Augmented Generation)? How It Works

Short answer: RAG (retrieval-augmented generation) means that before a large language model answers, it fetches relevant passages from an external source, such as an organization's documents, and adds them to the prompt. The model can then answer with current, organization-specific information without being retrained, and it can show where the answer came from. Documents are split into chunks, each chunk is turned into a numeric vector by an embedding model and stored in a vector database, and the chunks closest to the question are found by similarity search. Fine-tuning changes a model's behavior and output format; RAG adds current information to the answer without touching the model. The ground rule for security is that no answer may draw on a document the user is not allowed to see.

What is RAG?

RAG stands for retrieval-augmented generation. The term comes from a paper by Patrick Lewis and colleagues published at NeurIPS 2020. The paper's abstract notes that language models store factual knowledge in their parameters, but their ability to access and precisely manipulate that knowledge is still limited. The authors' answer was to combine this parametric memory with a non-parametric one: a dense vector index accessed through a pre-trained neural retriever.

According to AWS, RAG makes the model reference an authoritative knowledge base outside its training data before it generates a response. Microsoft describes it as a pattern that grounds responses in your proprietary content. In other words, RAG is not a model; it is a search-and-context pipeline built in front of one.

Why is RAG needed?

RAG starts from the limits we covered in our LLM article:

  • Knowledge cut-off: According to AWS, training data is static and puts a cut-off date on what the model knows; RAG brings in new information at lower cost than retraining.
  • Hallucination: According to Google Cloud, providing facts to the model as part of the prompt can mitigate hallucinations.
  • Organizational data: OpenAI recommends optimizing the context when the knowledge a task needs was not in the training set, is out of date, or is proprietary.

How does RAG work?

The first phase is preparation, done before any question arrives:

  1. Collecting documents: According to Microsoft, enterprise content spans SharePoint, databases, blob storage and other platforms; it is pulled from there and converted to text (see our ETL article).
  2. Chunking: Documents are split into pieces that fit under the token limits of embedding and chat models. The Azure AI Search documentation recommends starting with 512-token chunks and a 25% overlap (128 tokens).
  3. Embedding: In OpenAI's definition, an embedding is a vector (list) of floating point numbers, and the distance between two vectors measures how related the texts are.
  4. Indexing: The vectors are written to a vector database together with the chunk's text and its source.

The second phase repeats for every question. According to AWS, the query is also converted to a vector and matched against the vector database, the retrieved passages are added to the prompt as context, and the model grounds its answer in that context; the chunk's source travels with it into the answer. AWS also recommends updating the embeddings as documents change. In the example below, embed, search and llm are placeholder functions for the embedding model, the vector database and the language model:

def answer(question, user, embed, search, llm):
    vector = embed(question)
    chunks = search(vector, k=5, groups=user.groups)
    context = "\n\n".join(c.text for c in chunks)
    prompt = (f"Answer only from these sources:\n{context}"
              f"\n\nQuestion: {question}")
    return llm(prompt), [c.source for c in chunks]

Embeddings and vector databases

According to Google Cloud, a vector database stores documents as embeddings in a high-dimensional space, allowing fast and accurate retrieval based on semantic similarity. AWS notes that these systems are typically built on k-NN indexes using algorithms such as HNSW and IVF, and that they allow filtering on metadata. The trade-off is recall: according to the pgvector documentation, exact search without an index gives perfect recall, while approximate indexes trade some recall for speed. The main options are:

  • A database extension: pgvector stores vectors alongside the rest of your data in PostgreSQL (see our PostgreSQL article), and AWS's managed PostgreSQL services support it.
  • A purpose-built vector database: Open-source examples include Milvus, which is distributed under the Apache 2.0 license and can be deployed on Kubernetes clusters, and Qdrant, which can combine semantic and lexical search.
  • A search engine or managed service: OpenSearch provides vector search inside a search engine, and Azure AI Search runs vector and keyword queries in a single request.

Hybrid search and reranking

Vector search finds conceptually similar content even when no keywords match. But according to Microsoft, queries over product codes, specialized jargon, dates and people's names perform better with keyword search, because it can identify exact matches. Hybrid search combines the two: Azure AI Search merges full-text results ranked with BM25 and vector results using Reciprocal Rank Fusion (RRF). With pgvector, the same job is done by pairing it with PostgreSQL full-text search.

According to Google Cloud, embeddings look only at the semantic similarity of a document and a query, whereas a reranking model scores the candidates returned by the first search on how well they answer the query, with a value between 0 and 1, and reorders them.

RAG or fine-tuning?

Microsoft defines fine-tuning as adapting a pretrained model to your task through additional training on task-specific data. The two methods solve different problems:

  • RAG: Adds missing, outdated or proprietary knowledge to the context at answer time. The model does not change; when a document is updated, the answer follows.
  • Fine-tuning: According to Microsoft, it helps responses follow a consistent tone, format or schema. But it adds training and hosting costs, and it does not replace retrieval for current information.

OpenAI's guide treats the two not as alternatives but as additive methods that stack. The datasets and model versions behind fine-tuning belong to the life cycle we describe in our MLOps article.

How is RAG evaluated?

A RAG system can fail in two places: it retrieves the wrong chunk, or it uses the right chunk badly. The Ragas paper therefore looks at three dimensions: whether the retrieval system identifies relevant and focused context passages, whether the model exploits those passages faithfully, and the quality of the generation itself. Its metrics do not require ground-truth human annotations. Microsoft's RAG evaluators make a similar split, measuring separately whether the response sticks to the given context without fabricating content (groundedness), how relevant the retrieved chunks are to the query (retrieval), and search quality against labeled ground truth (document retrieval). In practice, you keep a test set of real questions and rerun it whenever the chunking, the model or the prompt changes.

Security and access: document-level permissions

The ground rule is that no answer may draw on a document the user is not allowed to see. According to Microsoft, Azure AI Search can trim results to exclude matches that do not carry the user's or group's identity (security trimming); but permission changes in the source system are only reflected in search results after that metadata is synchronized to the index (see our IAM article).

OWASP's list for LLM applications gives this risk its own entry, "Vector and Embedding Weaknesses": LLM09 in the 2026 edition published on August 3, 2026, and LLM08 in the 2025 edition. The risks and mitigations in the 2025 entry include:

  • Unauthorized access: Inadequate access controls can expose embeddings that contain sensitive information; the mitigation is permission-aware vector and embedding stores and strict partitioning of datasets.
  • Embedding inversion: Attackers can invert embeddings and recover significant amounts of the source information, so vectors need the same protection as the source documents.
  • Poisoning: In OWASP's example, a résumé with hidden instructions in white text on a white background changes the decision of a RAG-based screening system; the mitigation is to validate every document before it enters the knowledge base (see our data poisoning article).
  • Logging: Retrieval activity is recorded in detailed, immutable logs (see our log article).

In the PoisonedRAG study, published at USENIX Security 2025, injecting five malicious texts per target question into a knowledge database with millions of texts achieved a 90% attack success rate, and the authors found existing defenses insufficient. If the documents contain personal data, we cover the obligations in our article on the KVKK (Türkiye's personal data protection law) and the question of where the data is processed in our cross-border transfer article; we cover data classification in our data security and DLP service.

Infrastructure and operations

If you run the embedding model and the language model on your own infrastructure, you need inference capacity (see our GPU article). Keeping the knowledge base current, rerunning the evaluation set and tracking model versions are MLOps work. In Microsoft's "agentic retrieval" approach, an LLM breaks a complex query into focused subqueries and runs them in parallel; we explain how AI agents connect to tools in our AI agent and MCP articles. We cover vector database operations in our database platform engineering service and the design of the whole layer in our AI stack design and engineering service.

Frequently asked questions

What does RAG mean? Retrieval-augmented generation: before answering, the language model retrieves relevant passages from an external source and bases its answer on them.

What is a vector database? A database that stores embedding vectors and quickly finds the records semantically closest to a query. It can be a separate product or an extension such as pgvector.

What is an embedding? A representation that turns text into a vector of numbers. The vectors of related texts sit close together, and similarity search measures that closeness.

What is fine-tuning, and how does it differ from RAG? Fine-tuning adapts a pretrained model to a task through additional training on task-specific data, which makes tone and format consistent. RAG instead adds current information at answer time without changing the model.

What is a RAG architecture? The pipeline formed by document collection and chunking, an embedding model, a vector database, a search layer and a language model. Access control, logging and evaluation are part of it too.

Sources