RAG Basics

Zapvol’s knowledge base does not use vector RAG — it uses the file-system paradigm + agentic RAG (grep_docs + a knowledge sub-agent, see Knowledge Organization and Retrieval). This page keeps RAG basics as a contrast: understanding the mainstream route’s mechanics and limits is what makes clear why we chose another.

The core technology for building a shared knowledge base is Retrieval-Augmented Generation (RAG). The central idea is to combine the thinking and generation capabilities of large language models with the breadth and timeliness of an external knowledge base—the model’s training data has a cutoff date, while the knowledge base can be updated at any time.

A typical RAG system consists of two parts: a retriever, which finds relevant fragments from the knowledge base, and a generator (usually an LLM), which uses these fragments as context to generate an answer. Let’s first get an intuitive feel for how RAG works through two examples, then delve into the technical details of the retriever.

Example 1: Wikipedia Knowledge Base. A user asks, “What is quantum entanglement?” The base model’s training data might not include the latest experimental results. The RAG process is as follows:

# 1. User query
query = "What is quantum entanglement? What are the latest experimental advances?"

# 2. Retrieval: Find the most relevant fragments from the Wikipedia knowledge base
results = retriever.search(query, top_k=3)
# results = [
# "Quantum entanglement is a quantum mechanical phenomenon where the quantum states of two particles are correlated...",
# "The 2022 Nobel Prize in Physics was awarded to three scientists for experiments with quantum entanglement...",
# "Bell's inequality experiments have demonstrated the non-locality of quantum entanglement..."
# ]

# 3. Generation: Use the retrieved results as context for the LLM to generate an answer
answer = llm.generate(
    system="Answer the user's question based on the following reference materials. If the materials are insufficient, state that clearly.",
    context=results,   # ← Retrieved knowledge fragments injected into the context
    question=query
)

Example 2: Company Knowledge Base. A user asks, “I bought something and want a refund. What’s the process?”:

query = "Refund process"
results = retriever.search(query, top_k=2)
# results = [
# "Refund Policy: Full refunds can be requested within 7 days of order receipt. An order number is required. Refunds will be processed within 3-5 business days...",
# "Refund Steps: 1. Go to 'My Orders' 2. Select the order to be refunded 3. Click 'Request Refund'..."
# ]
answer = llm.generate(system="You are a customer service assistant.", context=results, question=query)
# → "You can request a full refund within 7 days of receipt. Steps: Go to 'My Orders' → Select the order → Click 'Request Refund'..."

The pattern is identical in both examples: Retrieve relevant fragments → Inject into context → LLM generates answer based on context. The core value of RAG is enabling the LLM to use knowledge it hasn’t seen during training (the latest Wikipedia content, a company’s internal documents) without needing to retrain the model.

The quality of the retriever directly determines the effectiveness of RAG—if it can’t retrieve relevant fragments, even the strongest LLM has nothing to work with. This section starts with the first step of getting documents into the knowledge base—chunking—then turns to the two main retrieval approaches, dense embeddings (semantic understanding) and sparse embeddings (keyword matching), and how to combine them.

Document Chunking

Figure 3-5 shows the core flow of RAG during a query: retrieval, augmentation, and generation. However, before retrieval is possible, there is an indispensable offline preprocessing step—chunking: cutting long documents into fragments (chunks) suitable for independent retrieval. Chunking is necessary for two reasons. First, embedding models have limits on input length, and when an entire document is compressed into a single vector, multiple topics are mixed together, and the vector cannot accurately represent any single one—this is the same problem encountered with Enhanced Notes: the longer the paragraph, the harder it is for the embedding to capture the key points. Second, the goal of retrieval is to inject only the relevant part into the context. If the fragment is too large, it brings in a lot of irrelevant content, wasting the context window and diluting attention.

Common chunking strategies fall into three categories:

Fixed-size Chunking: The simplest method, cutting by a fixed number of tokens (e.g., 512), usually with some overlap between adjacent chunks (e.g., 50-100 tokens) to prevent key sentences from being cut off at the boundary. It is simple to implement and produces predictable results, but it completely ignores document structure—a paragraph, a piece of code, or a table can all be cut in half.

Recursive/Structure-Aware Chunking: This method recursively cuts along the document’s natural boundaries (chapter titles, paragraphs, sentences)—first trying to cut by larger boundaries, and if the chunk is still too long, falling back to smaller ones. This suits documents with explicit structure—Markdown, HTML—particularly well, and it is the most common default in production systems.

Semantic Chunking: Calculates the embedding similarity of adjacent sentences and cuts at semantic cliffs (where similarity drops sharply), ensuring each chunk has a single primary theme. Higher chunking quality comes at the cost of additional embedding computation.

The choice of chunk size and overlap is a classic trade-off: if chunks are too small, individual chunks lack complete information and become semantically ambiguous out of context (“The company’s revenue grew by 3%“—which company? which quarter?). If chunks are too large, a single chunk mixes multiple topics, the embedding vector is diluted, retrieval accuracy decreases, and a retrieval hit brings in more irrelevant content. A common starting point in practice is 256-1024 tokens per chunk with 10%-20% overlap between adjacent chunks, followed by tuning based on measured retrieval quality.

Finally, a thread we will pick up later in this chapter: whatever the strategy, chunking severs a fragment from its original context—who is “the company”? which report did this passage come from?—that information stays outside the chunk. This is chunking’s inherent flaw, and the “Contextual Retrieval” section later in this chapter tackles it head-on.

Dense Embeddings: From Lexical Association to Semantic Understanding

What is an Embedding? Computers can only process numbers; they cannot directly understand the meaning of “apple” and “orange.” The idea of embeddings is to convert each word or sentence into a string of numbers (called a “vector,” e.g., [0.2, -0.5, 0.8,…]), and to make vectors for semantically similar content close to one another. The mathematical space where these vectors reside is called the “vector space.” You can think of it as a high-dimensional map, where each word or sentence is a point, and semantically closer content is closer together, just as the positions of Beijing and Shanghai on a map reflect their geographical relationship. A classic example is: "king" - "man" + "woman" ≈ "queen", showing that vector operations can capture semantic relationships. “Dense” is relative to the “sparse embeddings” introduced later: dense vectors have values in every dimension, while sparse vectors have most dimensions equal to zero.

Dense embeddings use deep learning to map text into a vector space—semantically similar content has close vector distances. A common method for measuring how “close” two vectors are is cosine similarity: it calculates the cosine of the angle between two vectors. The closer the value is to 1, the more aligned the directions and the more semantically similar the content. Early approaches (Word2Vec) could only capture word co-occurrence relationships; context-aware models (BERT, BGE-M3) can understand context, giving the same word different vector representations in different contexts (note: BGE-M3 actually outputs dense, sparse, and multi-vector representations simultaneously; here we only use its dense output as an example).

Why use the angle instead of the distance? Because we care about whether the directions of two vectors are aligned (whether their semantics are similar), not their magnitudes (text length or frequency). Two documents with identical content but different lengths will have vectors of different magnitudes but the same direction; cosine similarity can correctly determine that they are semantically identical.

Intuitively, you can think of it this way: for two pieces of text with similar semantics, the corresponding vectors have a smaller angle and therefore higher similarity—two expressions related to cat ownership almost overlap in vector space (cosine value close to 1), while cat ownership and stock investment point in completely different directions (cosine value close to 0). Actual embedding models use 768-dimensional or even higher-dimensional vectors, but the principle for judging “similarity” is exactly the same.

Supplementary Note (optional manual calculation example; skipping it won’t affect subsequent reading): Assume in a simplified 3-dimensional vector space, the embedding vectors of three sentences are “How to raise a cat” → A = (0.9, 0.5, 0.1), “Cat care guide” → B = (0.8, 0.6, 0.1), “Stock investment strategy” → C = (0.1, 0.1, 0.9). The formula for cosine similarity is cos(θ) = (A·B) / (|A| × |B|), where A·B is the dot product (multiply corresponding dimensions and sum), and |A| is the magnitude of the vector (square root of the sum of squares of each dimension).

Similarity between A and B: dot product = 0.9×0.8 + 0.5×0.6 + 0.1×0.1 = 1.03, |A| ≈ 1.03, |B| ≈ 1.00, cos(θ) ≈ 0.99 (very similar). Similarity between A and C: dot product = 0.9×0.1 + 0.5×0.1 + 0.1×0.9 = 0.23, |C| ≈ 0.91, cos(θ) ≈ 0.25 (very different). 0.99 vs 0.25 clearly reflects the semantic distance.

From Word2Vec to Context-Awareness

In the early days of dense embeddings, techniques such as Word2Vec generated a fixed vector for each word by analyzing the co-occurrence relationships of words in massive amounts of text. These vectors could capture interesting linguistic patterns, such as the vector operation “king” - “man” + “woman” ≈ “queen” (the “king - man + woman ≈ queen” mentioned in the earlier introduction to embeddings comes from this discovery), showing that word vector spaces can encode complex semantic relationships in a linearly computable way.

However, static word vectors have a fundamental limitation: they cannot handle polysemy. The word “bank” has completely different meanings in “river bank” and “investment bank,” but Word2Vec assigns it the exact same vector. Modern embedding models (such as BERT, BGE-M3) can take the context of the entire sentence or even paragraph into account when generating a vector for a word. This is enabled by the self-attention mechanism—when the model calculates the vector for each word, it simultaneously references information from all other words in the sentence. Thus “apple” gets different vectors in “Apple releases a new product” and “I bought two pounds of apples”—the same word acquires a distinct, more precise representation in each context, a leap from “lexical-level” to “contextual-level” semantics. Furthermore, new-generation models like BGE-M3 also support multilingual and long-text inputs (earlier context-aware models like BERT have an input length limit of only 512 tokens, making them unsuitable for long texts).

Sparse Embeddings: Keyword-Based Exact-Match Retrieval

Unlike dense embeddings, which capture semantic similarity, sparse embeddings are rooted in traditional information retrieval: at their core is exact keyword matching. A sparse embedding represents a document as an extremely high-dimensional vector in which most dimensions are zero—only the dimensions corresponding to words that appear in the document are non-zero. The theoretical foundation is the classic Bag of Words (BoW) model, which treats a piece of text as a “bag of words,” caring only about which words appear and how often, ignoring word order entirely: “cat chases dog” and “dog chases cat” are identical in BoW. More sophisticated probabilistic ranking algorithms evolved from this foundation.

From TF-IDF to BM25

Let’s build intuition with a concrete example. Assume a knowledge base has 100 technical articles, and a user searches for “model distillation.” The word “model” appears in 60 articles (too common, low discriminative power), while “distillation” appears in only 3 articles (very rare, high discriminative power). A good retrieval algorithm should give higher weight to the word “distillation”—articles containing “distillation” are more likely to be what the user is actually looking for. This is the core idea behind TF-IDF and BM25.

TF-IDF is based on a simple intuition: the more frequently a word appears in a document (TF, Term Frequency), and the fewer documents in the collection contain it (lower Document Frequency, or DF, and thus higher Inverse Document Frequency, or IDF), the more important the word is. In the example above, “model” has df/N = 60%, so its IDF value is low; “distillation” has df/N = 3%, so its IDF value is high—therefore, “distillation” contributes much more to the ranking than “model.” However, TF-IDF does not account for document length (longer documents naturally have higher term frequencies), and term-frequency growth is linear: is a word that appears 10 times really twice as important as one that appears five times? BM25 introduces two key parameters to correct these issues. k1 controls the “saturation” of term frequency: intuitively, an article mentioning “distillation” 20 times is not really twice as relevant as one mentioning it 10 times. k1 causes the contribution of term frequency to gradually level off as it increases, preventing long documents from unfairly dominating due to term frequency accumulation. b controls document length normalization, allowing the algorithm to handle documents of different lengths more fairly. This makes BM25 a more robust and effective ranking function, and it remains an indispensable core component in major search engines today.

Learned Sparse Retrieval. This chapter uses classic BM25 as the representative of sparse retrieval because it requires no training, is transparent and reproducible, and is best suited for explaining the principles of sparse retrieval. That said, sparse retrieval itself has entered a “learned” stage: models such as SPLADE, along with the sparse output branch of BGE-M3, use neural networks to assign weights to each term—no longer just scoring based on term frequency and document frequency like BM25, but letting the model judge “how important this word is in this text,” and even assigning non-zero weights to terms that are semantically related but do not appear in the original text (term expansion). The result is still a sparse vector with most dimensions being zero, preserving lexical interpretability and exact matching while gaining some semantic generalization from the neural network. Think of it as a meeting point between the sparse and dense routes.

Hybrid Retrieval: The Art of Having the Best of Both Worlds

Both methods have blind spots: dense retrieval understands semantics but may miss keywords (searching for “HTTP-403” might return general discussions about “server error”), while sparse retrieval matches exactly but cannot understand synonyms (searching for “kitty” won’t find documents that only mention “cat”). The idea behind hybrid retrieval is simple—run both engines and merge the results—but the difficulty lies in how to integrate two sets of scores with vastly different distributions into a meaningful ranking.

A typical hybrid retrieval pipeline has three stages, each with its own job. The first is parallel retrieval: the system sends the query to the dense and sparse engines simultaneously, and each recalls a set of candidate documents.

The second is result fusion, which combines the two result sets into a unified candidate pool. The difficulty is that the scores from the two paths are not directly comparable: the similarity scores from dense retrieval (e.g., cosine similarity, theoretically ranging from −1 to 1, but normalized text embeddings in practice usually fall between 0 and 1) and the BM25 scores from sparse retrieval (which can be any value from 0 to tens) have completely different scales and distributions. Two common fusion methods are: first, normalizing the scores from each path separately and then performing a weighted sum; second, Reciprocal Rank Fusion (RRF)—completely discarding the original scores and only looking at the ranks. The combined score for each document is the sum of the smoothed reciprocals of its ranks in each result set, i.e., score = Σ 1/(k + rank), where k is a smoothing constant (often 60), used to reduce the score gap between the top-ranked positions. RRF is simple and robust, but it uses only rank information, discarding the rich relevance signal in the original scores (weighted normalized fusion keeps the scores, at the cost of scale alignment, which is genuinely hard to tune).

The third stage—neural reranking—does more than compensate for the information that RRF discards: whichever fusion method precedes it, reranking earns its place by switching to a stronger matching paradigm. A cross-encoder performs deep, interactive matching between query and document, far more accurately than the retrieval stage’s bi-encoder, which encodes each independently and compares them by vector arithmetic. Concretely, it scores the top N candidates (say, 50) from the fused pool one by one to produce the final ranking. Note that reranking does not replace fusion: fusion produces the unified candidate pool from the two result sets; reranking refines the ranking within that pool—without the former, the latter wouldn’t even know which documents to score.

An analogy: a recruiter skimming resumes for a first cut is the bi-encoder; an interviewer in deep conversation with each candidate is the cross-encoder. The former screens at scale on pre-extracted features; the latter lets the query and each candidate document meet “face-to-face” and be evaluated word by word. The reranker employs the “Cross-Encoder” architecture, in stark contrast to the “Bi-Encoder” used in the retrieval stage. A Bi-Encoder generates independent vectors for the query and document and calculates similarity through vector operations—very fast, but unable to capture deep matching relationships, suitable for initial screening from massive data. A Cross-Encoder concatenates the query and candidate document into a single piece of text and feeds it to the model, allowing the model to compare word by word and output a comprehensive relevance score—much slower, but more accurate in relevance judgments. Commonly used reranking models like BAAI/bge-reranker-v2-m3 adopt this architecture.

This “joint attention” mechanism allows the cross-encoder to capture subtle semantic associations that the bi-encoder cannot perceive, resulting in a final ranking that is far more accurate than any single retrieval method.

How to Measure Retrieval Quality? Tuning a multi-stage pipeline like this requires objective metrics. The three that matter most (all computed on a test query set with annotated answers):

Table 3-3 Three Core Metrics for Retrieval Quality

Metric	Intuitive Explanation
recall@k	The proportion of queries for which a document containing the correct answer appears in the top k retrieval results—answering “Were the right documents found?” It is the metric most closely aligned with RAG’s core requirement: as long as the relevant document enters the context, the LLM has a chance to use it.
MRR (Mean Reciprocal Rank)	For each query, take the reciprocal of the rank of the first relevant document, then average across all queries—answering “How high up was the first hit?” Rank 1 gives a score of 1, rank 10 gives only 0.1.
nDCG (normalized Discounted Cumulative Gain)	Considers both the rank and relevance of all relevant documents; the score discount for relevant documents increases the further down the ranking they appear—answering “What is the overall quality of the sorted list?”

Industry reports also commonly mention “retrieval failure rate.” For example, in the Anthropic data cited later in this chapter, the retrieval failure rate refers to the proportion of queries where the correct information does not appear in the top-20 retrieval results—essentially 1 − recall@20. When you encounter such numbers, pin down which metric they map to and what k is before comparing across sources.

So far everything we have retrieved has been plain text. Real-world knowledge lives in far more forms than that.

Multimodal Information Extraction: Beyond the Boundaries of Text

In the knowledge base pipeline, multimodal information extraction sits at the very front—the ingestion and indexing stage. It determines the form in which non-textual content enters the knowledge base, and therefore how much information later chunking, embedding, and retrieval can use. Knowledge does not live only in text: charts, PDF layouts, and speech all need handling too. Architecturally there are three paths, and the core trade-off is fidelity versus cost.

Native Multimodal Processing: A Unified Semantic Space

The core technological breakthrough of native multimodal processing is the mapping of different data types into a unified, high-dimensional semantic space via specialized encoders. For images, multimodal models with publicly documented architectures (such as Qwen-VL and LLaVA) typically integrate a visual encoder based on the Vision Transformer (ViT)—simply put, “it cuts an image into small patches and treats them as ‘visual words’, then processes them with a Transformer” (the specific architectures of closed-source models like GPT-4o and Gemini are not public, but they are generally believed to follow a similar approach). Specifically, ViT divides an image into fixed-size patches and serializes each into a vector, the way words in a sentence are processed, so the patches sit alongside text word vectors in a shared multimodal embedding space. The Transformer’s self-attention mechanism can treat text and image tokens equally, computing arbitrary cross-modal correlations. This end-to-end joint processing provides unparalleled contextual fidelity—when the model directly “sees” the page layout, charts, and text of a PDF, it can understand the spatial and semantic relationships between text and images, making it particularly suitable for documents with complex layouts and high information density.

Extract to Text: A Low-Cost Approach

Extract to Text is a two-stage process: first, specialized tools (like OCR services, audio transcription services) convert non-textual content into plain text, which is then input into a language model. This reflects a design philosophy of modularity and cost-effectiveness: any multimodal task becomes a plain-text task, compatible with every language model, and the extracted text can be cached and reused. The cost is lost context—all layout, chart, and image information is thrown away during extraction.

Tool-Based Analysis: On-Demand Deep Dive

Treating multimodal analysis as a tool is a hybrid approach. It starts with text extraction, providing the Agent with an initial text summary, while also equipping the Agent with tools for in-depth analysis of the original file (e.g., analyze_image, analyze_pdf). This “on-demand deep dive” strategy balances the low cost of initial processing with the high fidelity of deep analysis.

Knowledge Organization and Retrieval — the route Zapvol actually uses, and why it is more controllable here
User Memory System — recall that likewise skips vectors for files + keyword match