The DDS Sovereign RAG Masterclass

Why This Class Exists

Most RAG tutorials reach for a cloud embedding API and a hosted vector database, then stop at a demo that falls over the moment you point it at real documents. This class builds the opposite: a complete pipeline that runs entirely on hardware you own, with every default that bites you called out and fixed, and every line of code runnable.

What Is Retrieval-Augmented Generation?

A language model knows only what it learned in training. Ask it about your own documents — your product catalog, your internal notes, last quarter's report — and it cannot answer, because that text was never in its training data. Retrieval-augmented generation fixes this without retraining anything. At query time, the system searches your documents for the passages most relevant to the question and hands them to the model as context. The model then answers from that context.

The key reframing: the model never learns your data. It stays frozen. Your data lives in a searchable store, and you inject the relevant slice into the prompt for each question. This is why RAG is cheap, updatable, and auditable — you can change a document and the next query reflects it instantly, and you can see exactly which passages produced an answer.

The Five Stages

Every RAG pipeline, local or cloud, is the same five stages. The first three happen once, at ingest. The last two happen on every query.

Stages 1 through 3 are your indexing pipeline — run them whenever the corpus changes. Stages 4 and 5 are your query pipeline — they run in milliseconds-to-seconds per question. The separation matters: indexing is the expensive, occasional work; querying is the cheap, frequent work. Keep them as separate code paths.

Why Fully Local

You can run any of these five stages in the cloud. The decision to run all of them locally buys four specific things.

The cost of local is that you run and tune the components yourself, and your throughput is bounded by your hardware rather than an elastic cloud. For a sovereignty-first studio that already runs local models, that is a cost you have mostly already paid. The privacy and zero-marginal-cost properties are the payoff.

The Stack

Three components, plus Python to wire them together. Each is open-source and runs locally.

The 8 Architectural Decisions

Decide these eight before you write the pipeline. Each has a default that is right most of the time and a failure mode that quietly degrades answers if you choose wrong. Two of them — distance metric and context window — fail with no error at all.

Decision 1 — Distance metric: cosine or L2

The vector store ranks chunks by a distance metric. Cosine measures angle between vectors; L2 measures straight-line distance.

Tradeoff: for normalized text embeddings, cosine is the standard and gives intuitive similarity scores. L2 can rank differently and is sensitive to vector magnitude.

Failure mode: ChromaDB defaults to L2. Leaving the default while expecting cosine produces subtly wrong rankings with no error. The space cannot be changed after the index is created, so you would have to rebuild.

Recommendation: set cosine explicitly at collection creation. This is the single most common ChromaDB mistake; fix it before you ingest a single document.

Decision 2 — Embedding context window

The embedder converts a chunk to a vector, but only up to its context limit.

Tradeoff: a larger window lets you embed bigger chunks whole; a smaller window forces finer chunking.

Failure mode: nomic-embed-text advertises 8192 tokens, but the default Ollama context may process only the first 2048, silently dropping the rest of a long chunk. The vector then represents only the chunk's opening, and retrieval quietly misses content that was actually there.

Recommendation: keep chunks comfortably under the effective window, or raise num_ctx so the whole chunk is embedded. Never assume the advertised limit is the active one.

Decision 3 — Chunk size

How big is each passage you embed?

Tradeoff: large chunks carry more context but blur multiple topics into one vector, hurting precision. Small chunks are precise but can lose the surrounding context that makes a passage meaningful.

Failure mode: embedding whole documents as single chunks. The vector averages everything, so retrieval returns the document for almost any query and the model drowns in irrelevant text.

Recommendation: start at a few hundred tokens per chunk. Tune from there against your actual corpus and queries. This is a knob you will return to.

Decision 4 — Chunk overlap

Do adjacent chunks share a little text at their boundaries?

Tradeoff: overlap preserves ideas that straddle a boundary, at the cost of some duplicate storage and the chance of retrieving two near-identical chunks.

Failure mode: zero overlap splitting a key sentence across two chunks, so neither chunk embeds the complete idea and neither retrieves well for it.

Recommendation: a small overlap — roughly 10 to 20 percent of chunk size — catches boundary-straddling ideas without much waste.

Decision 5 — Retrieval depth (top-k)

How many chunks do you pass to the model per query?

Tradeoff: low k risks missing the chunk that holds the answer; high k floods the prompt with noise and burns context window the generation model needs.

Failure mode: setting k to one and missing the answer whenever it lives in the second-ranked chunk; or setting k to twenty and burying the relevant passage in irrelevant ones.

Recommendation: start at three to five. Raise it only if you observe answers missing content that you know is in the corpus.

Decision 6 — Embedding model choice

Which model turns text into vectors?

Tradeoff: larger embedders can be marginally more accurate but slower and heavier; smaller ones are fast and cheap to run.

Failure mode: changing the embedding model after you have indexed. Vectors from different models are not comparable, so a query embedded with a new model cannot match chunks embedded with the old one. You must re-embed the whole corpus.

Recommendation: nomic-embed-text is the strong local default. Pick it once and re-index fully if you ever change it. Treat the embedder as part of the index's identity.

Decision 7 — Chunk identity and updates

How do you handle a document changing after it is indexed?

Tradeoff: rebuilding the whole index on any change is simple but slow at scale; incremental updates are fast but need stable IDs.

Failure mode: random or sequential IDs that change on every re-ingest, so you cannot tell which chunks already exist and you accumulate duplicates.

Recommendation: derive each chunk's ID deterministically from its source path and position, then upsert. Changed chunks overwrite; unchanged chunks are skipped. Re-indexing becomes incremental.

Decision 8 — Grounding and refusal

How hard do you constrain the model to the retrieved context?

Tradeoff: a loose prompt lets the model blend retrieved context with its training knowledge, which sometimes helps and sometimes hallucinates. A strict prompt confines it to the context and tells it to refuse when the context is insufficient.

Failure mode: a prompt that just pastes context and asks the question. The model answers confidently even when the retrieved chunks do not contain the answer, inventing details.

Recommendation: instruct the model explicitly to answer only from the provided context, to cite the source chunks, and to say it does not know when the context is insufficient. This is your strongest anti-hallucination lever.

Setup and Models

Two Python dependencies and two Ollama models. Nothing else. The reference build reads a folder of .txt and .md files and answers questions about them.

Project layout

sovereign-rag/
├── requirements.txt
├── corpus/            # drop your .txt and .md files here
│   ├── doc-a.md
│   └── doc-b.txt
├── chroma_store/      # created automatically; the persistent vector store
├── ingest.py          # stages 1-3: ingest, chunk, embed, store
└── query.py           # stages 4-5: retrieve, generate

requirements.txt

ollama>=0.4.0
chromadb>=0.5.0

Pull the models and install

# 1. Install Ollama from https://ollama.com, then pull the two models.
#    The embedder (small, fast):
ollama pull nomic-embed-text

#    A generation model sized to your VRAM. llama3.1 (8B) is a solid
#    default for a 12GB card. Swap for any instruct model you prefer.
ollama pull llama3.1

# 2. Confirm Ollama is serving (default port 11434):
ollama list

# 3. Install the two Python deps (a venv is recommended):
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Ingest and Chunk

Stage 1 loads files; stage 2 splits them into overlapping chunks. The chunker here is deliberately simple and dependency-free: it splits on character count with overlap. Token-aware splitting is more precise, but character chunking is transparent and good enough to learn on — and it has no extra dependency.

import os
import hashlib

CHUNK_SIZE = 1200       # characters, not tokens — roughly 300 tokens
CHUNK_OVERLAP = 200     # ~15% overlap to catch boundary-straddling ideas


def load_corpus(corpus_dir: str) -> list[dict]:
    """Stage 1 — load every .txt and .md file in the corpus directory."""
    docs = []
    for root, _dirs, files in os.walk(corpus_dir):
        for fname in files:
            if not fname.lower().endswith((".txt", ".md")):
                continue
            path = os.path.join(root, fname)
            with open(path, "r", encoding="utf-8") as fh:
                text = fh.read()
            docs.append({"source": os.path.relpath(path, corpus_dir), "text": text})
    return docs


def chunk_text(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
    """Stage 2 — split one document into overlapping character windows."""
    if size <= overlap:
        raise ValueError("CHUNK_SIZE must be greater than CHUNK_OVERLAP")
    chunks = []
    start = 0
    text = text.strip()
    while start < len(text):
        end = start + size
        chunk = text[start:end].strip()
        if chunk:
            chunks.append(chunk)
        # advance by size minus overlap so windows overlap
        start += size - overlap
    return chunks


def chunk_id(source: str, index: int) -> str:
    """Deterministic ID — same source + position always yields the same ID,
    so re-ingest upserts in place instead of duplicating (Decision 7)."""
    raw = f"{source}::{index}"
    return hashlib.sha1(raw.encode("utf-8")).hexdigest()[:16]

Note the deterministic chunk_id. Because the ID is a hash of source path plus position, re-ingesting an unchanged file produces the same IDs, so ChromaDB's upsert overwrites in place rather than accumulating duplicates. Change a file and only its chunks change.

Embed and Store

Stage 3 embeds each chunk with nomic-embed-text via the Ollama client and stores the vector, the text, and the source metadata in ChromaDB. This is where the two silent-killer decisions get fixed in code: cosine distance is set at collection creation, and chunks are sized under the effective embedding window.

import chromadb
import ollama

CHROMA_PATH = "./chroma_store"
COLLECTION = "sovereign_corpus"
EMBED_MODEL = "nomic-embed-text"


def get_collection():
    """Open the persistent store and get-or-create the collection with
    cosine distance set explicitly (Decision 1 — the L2 default footgun).
    The space cannot be changed after creation, so it is set here once."""
    client = chromadb.PersistentClient(path=CHROMA_PATH)
    return client.get_or_create_collection(
        name=COLLECTION,
        configuration={"hnsw": {"space": "cosine"}},
    )


def embed_one(text: str) -> list[float]:
    """Stage 3 — embed a single chunk with nomic-embed-text on Ollama.
    Returns a 768-dimension vector."""
    resp = ollama.embed(model=EMBED_MODEL, input=text)
    # ollama.embed returns {"embeddings": [[...]]} — one row per input
    return resp["embeddings"][0]


def index_corpus(corpus_dir: str) -> int:
    """Run stages 1-3 end to end and upsert into ChromaDB.
    Returns the number of chunks indexed."""
    collection = get_collection()
    docs = load_corpus(corpus_dir)

    ids, embeddings, documents, metadatas = [], [], [], []
    for doc in docs:
        chunks = chunk_text(doc["text"])
        for i, chunk in enumerate(chunks):
            ids.append(chunk_id(doc["source"], i))
            embeddings.append(embed_one(chunk))
            documents.append(chunk)
            metadatas.append({"source": doc["source"], "chunk": i})

    if not ids:
        print("No chunks found. Add .txt or .md files to the corpus directory.")
        return 0

    # upsert so re-runs overwrite by deterministic ID instead of duplicating
    collection.upsert(
        ids=ids,
        embeddings=embeddings,
        documents=documents,
        metadatas=metadatas,
    )
    return len(ids)

Retrieve and Generate

Stages 4 and 5 run on every query. Stage 4 embeds the question with the same model used at ingest, then asks ChromaDB for the nearest chunks. Stage 5 builds a grounded prompt from those chunks and has the local LLM answer.

GEN_MODEL = "llama3.1"
TOP_K = 4               # Decision 5 — retrieval depth


def retrieve(question: str, k: int = TOP_K) -> list[dict]:
    """Stage 4 — embed the question the SAME way as ingest, then fetch
    the k nearest chunks by cosine distance."""
    collection = get_collection()
    q_vec = embed_one(question)          # same embedder, same client
    res = collection.query(
        query_embeddings=[q_vec],
        n_results=k,
        include=["documents", "metadatas", "distances"],
    )
    # ChromaDB returns parallel lists wrapped one level deep (per query)
    hits = []
    for doc, meta, dist in zip(
        res["documents"][0], res["metadatas"][0], res["distances"][0]
    ):
        hits.append({"text": doc, "source": meta["source"],
                     "chunk": meta["chunk"], "distance": dist})
    return hits


def build_prompt(question: str, hits: list[dict]) -> str:
    """Stage 5a — assemble a grounded prompt. Each chunk is labeled with a
    source tag so the model can cite it (Decision 8 — grounding + refusal)."""
    context_blocks = []
    for n, h in enumerate(hits, start=1):
        tag = f"[{n}] {h['source']} (chunk {h['chunk']})"
        context_blocks.append(f"{tag}\n{h['text']}")
    context = "\n\n".join(context_blocks)

    return (
        "You are a careful assistant. Answer the question using ONLY the "
        "context below. Cite the sources you used by their bracket number, "
        "e.g. [1]. If the context does not contain the answer, say you do "
        "not know rather than guessing.\n\n"
        f"CONTEXT:\n{context}\n\n"
        f"QUESTION: {question}\n\n"
        "ANSWER (with citations):"
    )


def answer(question: str, k: int = TOP_K) -> dict:
    """Stages 4-5 — retrieve, ground, and generate locally. Returns the
    answer text plus the sources it was grounded in."""
    hits = retrieve(question, k)
    if not hits:
        return {"answer": "The knowledge base is empty. Run ingest first.",
                "sources": []}
    prompt = build_prompt(question, hits)
    resp = ollama.chat(
        model=GEN_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return {
        "answer": resp["message"]["content"],
        "sources": [{"n": i + 1, "source": h["source"], "chunk": h["chunk"]}
                    for i, h in enumerate(hits)],
    }

Three details earn their place here. The query is embedded with embed_one — the exact same function ingest used — so the vectors are comparable. The prompt names each chunk with a bracket tag, giving the model something concrete to cite. And the instruction tells the model to refuse when the context is insufficient, which is the difference between a grounded answer and a confident hallucination.

The Complete Scripts

Assembled into the two runnable files. Everything above, in order, with a command-line entry point on each. Copy these verbatim into ingest.py and query.py.

ingest.py

import os
import sys
import hashlib
import chromadb
import ollama

# ---- shared config (must match query.py) -------------------------------
CHROMA_PATH = "./chroma_store"
COLLECTION = "sovereign_corpus"
EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 1200       # characters (~300 tokens)
CHUNK_OVERLAP = 200     # ~15% overlap


def load_corpus(corpus_dir: str) -> list[dict]:
    docs = []
    for root, _dirs, files in os.walk(corpus_dir):
        for fname in files:
            if not fname.lower().endswith((".txt", ".md")):
                continue
            path = os.path.join(root, fname)
            with open(path, "r", encoding="utf-8") as fh:
                text = fh.read()
            docs.append({"source": os.path.relpath(path, corpus_dir), "text": text})
    return docs


def chunk_text(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
    if size <= overlap:
        raise ValueError("CHUNK_SIZE must be greater than CHUNK_OVERLAP")
    chunks, start, text = [], 0, text.strip()
    while start < len(text):
        chunk = text[start:start + size].strip()
        if chunk:
            chunks.append(chunk)
        start += size - overlap
    return chunks


def chunk_id(source: str, index: int) -> str:
    return hashlib.sha1(f"{source}::{index}".encode("utf-8")).hexdigest()[:16]


def get_collection():
    client = chromadb.PersistentClient(path=CHROMA_PATH)
    return client.get_or_create_collection(
        name=COLLECTION,
        configuration={"hnsw": {"space": "cosine"}},
    )


def embed_one(text: str) -> list[float]:
    return ollama.embed(model=EMBED_MODEL, input=text)["embeddings"][0]


def index_corpus(corpus_dir: str) -> int:
    collection = get_collection()
    ids, embeddings, documents, metadatas = [], [], [], []
    for doc in load_corpus(corpus_dir):
        for i, chunk in enumerate(chunk_text(doc["text"])):
            ids.append(chunk_id(doc["source"], i))
            embeddings.append(embed_one(chunk))
            documents.append(chunk)
            metadatas.append({"source": doc["source"], "chunk": i})
    if not ids:
        print("No chunks found. Add .txt or .md files to the corpus directory.")
        return 0
    collection.upsert(ids=ids, embeddings=embeddings,
                      documents=documents, metadatas=metadatas)
    return len(ids)


if __name__ == "__main__":
    corpus_dir = sys.argv[1] if len(sys.argv) > 1 else "./corpus"
    n = index_corpus(corpus_dir)
    print(f"Indexed {n} chunks from {corpus_dir} into {COLLECTION}.")

query.py

import sys
import chromadb
import ollama

# ---- shared config (must match ingest.py) ------------------------------
CHROMA_PATH = "./chroma_store"
COLLECTION = "sovereign_corpus"
EMBED_MODEL = "nomic-embed-text"
GEN_MODEL = "llama3.1"
TOP_K = 4


def get_collection():
    client = chromadb.PersistentClient(path=CHROMA_PATH)
    return client.get_or_create_collection(
        name=COLLECTION,
        configuration={"hnsw": {"space": "cosine"}},
    )


def embed_one(text: str) -> list[float]:
    return ollama.embed(model=EMBED_MODEL, input=text)["embeddings"][0]


def retrieve(question: str, k: int = TOP_K) -> list[dict]:
    collection = get_collection()
    res = collection.query(
        query_embeddings=[embed_one(question)],
        n_results=k,
        include=["documents", "metadatas", "distances"],
    )
    hits = []
    for doc, meta, dist in zip(
        res["documents"][0], res["metadatas"][0], res["distances"][0]
    ):
        hits.append({"text": doc, "source": meta["source"],
                     "chunk": meta["chunk"], "distance": dist})
    return hits


def build_prompt(question: str, hits: list[dict]) -> str:
    blocks = []
    for n, h in enumerate(hits, start=1):
        blocks.append(f"[{n}] {h['source']} (chunk {h['chunk']})\n{h['text']}")
    context = "\n\n".join(blocks)
    return (
        "You are a careful assistant. Answer the question using ONLY the "
        "context below. Cite the sources you used by their bracket number, "
        "e.g. [1]. If the context does not contain the answer, say you do "
        "not know rather than guessing.\n\n"
        f"CONTEXT:\n{context}\n\n"
        f"QUESTION: {question}\n\n"
        "ANSWER (with citations):"
    )


def answer(question: str, k: int = TOP_K) -> dict:
    hits = retrieve(question, k)
    if not hits:
        return {"answer": "The knowledge base is empty. Run ingest.py first.",
                "sources": []}
    resp = ollama.chat(
        model=GEN_MODEL,
        messages=[{"role": "user", "content": build_prompt(question, hits)}],
    )
    return {
        "answer": resp["message"]["content"],
        "sources": [{"n": i + 1, "source": h["source"], "chunk": h["chunk"]}
                    for i, h in enumerate(hits)],
    }


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print('Usage: python query.py "your question here"')
        sys.exit(1)
    result = answer(" ".join(sys.argv[1:]))
    print(result["answer"])
    print("\nSources:")
    for s in result["sources"]:
        print(f"  [{s['n']}] {s['source']} (chunk {s['chunk']})")

Run It

Three commands from a cold start to a grounded, cited answer.

# 1. Index your corpus (stages 1-3). Drop files in ./corpus first.
python ingest.py ./corpus
#  → "Indexed 142 chunks from ./corpus into sovereign_corpus."

# 2. Ask a question (stages 4-5).
python query.py "What is our return policy for sale items?"
#  → a grounded answer, followed by the source chunks it used.

# 3. Re-run ingest any time the corpus changes. Deterministic IDs mean
#    unchanged chunks are overwritten in place, not duplicated.
python ingest.py ./corpus

The first ingest is the slow step — every chunk is embedded once. Queries are fast: one embedding call for the question, a millisecond vector search, and one generation call. Nothing in this loop touches the network.

Grounding and Citations

The single biggest reliability lever in RAG is the prompt's grounding instruction. The build above already does three things that most first attempts miss.

It tags every chunk. Each retrieved chunk enters the prompt with a bracket number and its source path. The model is asked to cite by bracket number, so the answer carries provenance you can verify. A claim cited [2] can be traced straight back to a specific chunk of a specific file.

It constrains to the context. The instruction says to answer using only the provided context. This stops the model from blending in half-remembered training knowledge, which is where most RAG hallucinations originate — not from the retrieved text, but from the model reaching past it.

It permits refusal. The model is told to say it does not know when the context is insufficient. Counterintuitively, a system that sometimes says "I don't know" is more trustworthy than one that always answers, because its answers mean something. Refusal is a feature.

Tuning and Gotchas

Once the pipeline runs, answer quality lives in a small number of knobs. Turn these, in this order.

Fix the two silent killers first

Before tuning anything, confirm both invisible defaults are handled. Cosine distance must be set at collection creation — the build does this in get_collection, and because the space cannot change after creation, a store built on the L2 default must be deleted and rebuilt. Embedding truncation: with a 1200-character chunk you are well under the 2048-token effective window, so nomic-embed-text sees the whole chunk. If you raise CHUNK_SIZE substantially, either keep it under that window or raise the model's context so chunks are not silently clipped.

Then tune retrieval

Measuring retrieval, not just vibes

Tune against evidence. Assemble a small set of questions whose answers you know live in specific chunks. After each change, run them and check whether the right chunk appears in the retrieved set and whether the answer cites it. This turns tuning from guesswork into a measurable loop — exactly the proof-over-vibes discipline a sovereign system rewards, because you are the only one who can validate it.

Failure Modes

The Sovereignty Case

It is worth being explicit about why this architecture is worth the tuning effort, because the cloud alternative is genuinely easier to stand up.

A hosted RAG service gives you managed embeddings, a managed vector database, and elastic scale in an afternoon. In exchange, your documents are processed on someone else's hardware, every embedding and every query is billed, and the system stops working when the network does. For a great many projects that trade is fine.

For a sovereignty-first studio, it is not. When the corpus is your own internal knowledge, your clients' material, or anything you are not willing to send to a third party, local is not a preference — it is the constraint. The same decision that protects the data also removes the per-token meter and the connectivity dependency. You run capable models on hardware you already own, index once, and query forever at the cost of electricity.

Closing Notes

You now have a complete, runnable, fully local RAG system and the judgment to tune it. The compressed playbook for your next knowledge base:

1. Set cosine before you ingest. ChromaDB defaults to L2, the space cannot change after creation, and the failure is silent. This is the first line of get_collection for a reason.
2. Keep chunks under the embedding window. nomic-embed-text may truncate long chunks with no error. Size chunks deliberately or raise the context.
3. Share one config across ingest and query. Same embedding model, same path, same collection. If the embedder diverges, vectors stop being comparable and retrieval quietly fails.
4. Use deterministic chunk IDs and upsert. Re-indexing becomes incremental instead of a duplicate-accumulating full rebuild.
5. Tune chunk size and top-k against a known-answer test set. These two knobs move quality more than the model does. Measure, do not guess.
6. Ground the prompt and allow refusal. Answer only from context, cite by bracket number, say "I don't know" when the context is thin. This is your anti-hallucination lever.
7. Prove it offline. Disconnect the network and run it. If it still answers, you have genuine data sovereignty — not a cloud dependency in disguise.

Frequently Asked Questions

Eighteen questions builders ask most. These mirror the FAQPage schema at the top of the page, which surfaces them in AI overviews and rich results.