Why Build RAG at the Edge?
Here’s the thing about cloud-based RAG: it works great until you need privacy, offline access, or predictable costs. Every query sends your documents to a third-party API, every embedding call racks up a bill, and the moment your internet goes down, your knowledge base goes with it.
Running RAG at the edge — on your own machine, fully offline, with local models — flips that equation. Your documents never leave your hard drive. Your queries cost zero per token. And your whole pipeline works on a laptop, a Raspberry Pi, or even a mobile phone. I’ve written before about why enterprises are moving away from cloud AI APIs — this is the same philosophy, applied to your personal knowledge base.
That’s where Google’s Gemma 3 and EmbeddingGemma come in. Gemma 3 handles the chat and reasoning side. EmbeddingGemma — a tiny 300-million parameter model at just 622MB — handles the embedding side. Both run locally through Ollama, no GPU required for the smaller variants.
In this guide, I’ll walk through building a complete RAG system that runs entirely on your machine, uploads PDFs and documents, and answers questions based on their content. All offline. All private. No API keys needed.
The RAG Pipeline, Simplified
Before we write code, let’s map out the pipeline. RAG has two distinct phases:
Phase 1: Ingestion (happens once per document)
- Load — Read the document (PDF, TXT, DOCX, Markdown)
- Chunk — Split it into manageable pieces (500-1000 tokens each)
- Embed — Convert each chunk into a vector using EmbeddingGemma
- Store — Save those vectors in a local vector database (FAISS or ChromaDB)
Phase 2: Query (happens every time you ask a question)
- Embed the question — Convert the user’s query into a vector using the same embedding model
- Retrieve — Find the top-K most similar chunks from the vector database by cosine similarity
- Augment — Stuff the retrieved chunks into a prompt as context
- Generate — Pass the augmented prompt to Gemma 3, which answers grounded in the retrieved documents
The beauty of this design? You pay the embedding cost once during ingestion and amortize it across every future query. A document collection of 100 PDFs might take a couple of minutes to index, but after that, every question returns in seconds.
What You’ll Need
- Ollama — Download from ollama.com (works on macOS, Linux, Windows)
- Python 3.10+ — With a virtual environment
- A few hundred MB of disk space — The models are surprisingly small
Pull the models:
ollama pull gemma3
ollama pull embeddinggemmaThat’s it. gemma3 defaults to the 4B parameter variant, which runs comfortably on 8GB of RAM. embeddinggemma is only 622MB — it’s designed for on-device use.
Setting Up the Project
Create your folder structure:
local-rag/
├── documents/ # Drop your PDFs and docs here
├── rag_core.py # The main RAG engine
├── app.py # A simple CLI or Streamlit interface
└── requirements.txtpip install ollama chromadb pypdf sentence-transformers fastapi uvicornLet’s start with the core engine.
Building the RAG Core Engine
Step 1: Document Loader with PDF Support
We need to handle multiple file types. PDFs are the trickiest, so let’s tackle those head-on with pypdf:
import os
from pypdf import PdfReader
def load_document(filepath):
ext = os.path.splitext(filepath)[1].lower()
if ext == ".pdf":
reader = PdfReader(filepath)
text = "\n".join([page.extract_text() for page in reader.pages])
elif ext == ".txt":
with open(filepath, "r", encoding="utf-8") as f:
text = f.read()
elif ext == ".md":
with open(filepath, "r", encoding="utf-8") as f:
text = f.read()
else:
raise ValueError(f"Unsupported file type: {ext}")
return textStep 2: Smart Chunking
Chunking is where most RAG systems either shine or fall apart. Too small, and you lose context. Too large, and retrieval becomes fuzzy. I use overlapping chunks of roughly 500 characters:
def chunk_text(text, chunk_size=500, overlap=50):
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start = end - overlap
return chunksThe overlap ensures that a concept split across two chunks doesn't get lost. If a sentence starts at position 480 and ends at 520, both chunks contain it.
Step 3: Embedding with EmbeddingGemma
Here's where the magic happens. We use Ollama's Python library to call EmbeddingGemma locally:
import ollama
def embed_texts(texts):
"""Batch embed a list of text chunks using EmbeddingGemma."""
response = ollama.embed(
model="embeddinggemma",
input=texts
)
return response["embeddings"]EmbeddingGemma returns L2-normalized vectors by default, which means cosine similarity reduces to a simple dot product — faster and numerically stable.
Edge design note: EmbeddingGemma is specifically built for this. At 300M parameters, it runs on hardware as constrained as a mobile phone. Google designed it for on-device RAG pipelines, and that shows in both the memory footprint (622MB quantized) and the 100+ language support.
Step 4: Vector Storage with ChromaDB
ChromaDB is the friendliest local vector database you'll meet. It persists to disk, requires zero configuration, and has a clean Python API:
import chromadb
from chromadb.config import Settings
def create_vector_store(collection_name="my_docs"):
client = chromadb.PersistentClient(
path="./chroma_db",
settings=Settings(anonymized_telemetry=False)
)
collection = client.get_or_create_collection(name=collection_name)
return collection
def index_documents(collection, chunks, embeddings, filepath):
ids = [f"{os.path.basename(filepath)}_{i}" for i in range(len(chunks))]
metadatas = [{"source": filepath, "chunk": i} for i in range(len(chunks))]
collection.add(
ids=ids,
embeddings=embeddings,
documents=chunks,
metadatas=metadatas
)
print(f"Indexed {len(chunks)} chunks from {filepath}")Step 5: Retrieval
When a user asks a question, we embed their query, search the vector store, and pull the most relevant chunks:
def retrieve(collection, query, k=3):
query_embedding = ollama.embed(
model="embeddinggemma",
input=[query]
)["embeddings"][0]
results = collection.query(
query_embeddings=[query_embedding],
n_results=k
)
return results["documents"][0], results["metadatas"][0]The k parameter controls how many chunks to retrieve. Three is a good starting point — enough to provide solid context without overwhelming the LLM's context window.
Step 6: Generation with Gemma 3
This is where the pipeline culminates. We take the retrieved chunks, build a prompt that instructs Gemma 3 to answer only from the provided context, and let it generate:
def generate_answer(query, context_chunks):
context = "\n\n".join(context_chunks)
prompt = f"""You are a helpful assistant. Answer the question based ONLY on the provided context.
If the context does not contain enough information, say "I don't have enough information to answer that."
CONTEXT:
{context}
QUESTION:
{query}
ANSWER:"""
response = ollama.chat(
model="gemma3",
messages=[{"role": "user", "content": prompt}]
)
return response["message"]["content"]The prompt engineering here matters. Explicitly telling the model to say "I don't know" when context is insufficient is your guardrail against hallucination. Gemma 3 respects this well — it's trained to follow instructions reliably.
Putting It All Together
Here's the complete pipeline wired as a simple CLI interface:
import os
import argparse
from rag_core import load_document, chunk_text, embed_texts
from rag_core import create_vector_store, index_documents
from rag_core import retrieve, generate_answer
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--ingest", help="Path to document or folder to index")
parser.add_argument("--query", help="Ask a question")
args = parser.parse_args()
collection = create_vector_store()
if args.ingest:
path = args.ingest
if os.path.isfile(path):
files = [path]
else:
files = [os.path.join(path, f) for f in os.listdir(path)
if f.endswith((".pdf", ".txt", ".md"))]
for filepath in files:
print(f"Loading {filepath}...")
text = load_document(filepath)
chunks = chunk_text(text)
embeddings = embed_texts(chunks)
index_documents(collection, chunks, embeddings, filepath)
print("Ingestion complete.")
if args.query:
results, metadata = retrieve(collection, args.query)
answer = generate_answer(args.query, results)
print(f"\nQuestion: {args.query}\n")
print(f"Answer: {answer}\n")
print("Sources:")
for m in metadata:
print(f" - {m['source']} (chunk {m['chunk']})")
if __name__ == "__main__":
main()Run the ingestion:
python main.py --ingest ./documents/my-report.pdfThen query:
python main.py --query "What were the key findings in the report?"The model answers with citations pointing back to the source document and chunk, so you can verify everything it says. If you want a more turnkey approach, I also covered how to build a local AI-powered document search engine with DocuBrowser — it's a different take on the same problem with a pre-built UI.
Why Edge Design Matters for RAG
Building RAG at the edge isn't just about being cool — it solves real problems that cloud RAG can't touch. I've been on a self-hosting kick lately — from running my own AI meeting assistant to this RAG system — and the common thread is that local AI is finally practical enough for day-to-day use.
Privacy by Design
Your documents never leave your machine. For sensitive content — legal contracts, medical records, internal company docs — sending them through a third-party API is a non-starter. Edge RAG keeps everything local by default.
Zero Recurring Cost
Cloud embedding APIs charge per token. For a document collection of 100 PDFs at 50 pages each, you're looking at thousands of tokens just to index. Every query after that costs more. Edge RAG with Ollama costs exactly zero after setup — your hardware, your electricity, your models.
Offline Operation
No internet? No problem. Edge RAG works on a plane, in a remote area, or in an air-gapped environment. If you're working with classified or sensitive materials, this isn't a nice-to-have — it's the requirement.
Low Latency
Every cloud API call adds 200-800ms of network latency. Edge RAG eliminates that entirely. Embedding with EmbeddingGemma on a laptop takes milliseconds. Generation with Gemma 3 4B runs at 30-40 tokens per second on a modern CPU — fast enough for interactive use.
The EmbeddingGemma Advantage
This is what makes the edge approach viable. Before EmbeddingGemma, you had two options: use a large embedding model that barely runs on a laptop, or use a tiny one that sacrifices accuracy. EmbeddingGemma at 300M parameters is the sweet spot — small enough for on-device deployment, accurate enough to beat models 10x its size on benchmark retrieval tasks. Google designed it specifically for this use case, and it shows.
Going Further: Production Edge Design Patterns
Once the basic pipeline works, here are edge-focused improvements worth considering:
Streamlit UI for Drag-and-Drop Uploads
Wrap the RAG engine in a Streamlit interface for non-technical users. Upload PDFs through a browser, ask questions in a chat box, get answers instantly. No terminal needed.
FastAPI for Local API Server
Expose the RAG pipeline as a REST API running on localhost. Other apps on your machine — note-taking tools, document managers, automation scripts — can query your knowledge base without running Python directly.
Persistent Session History
Store conversation history so follow-up questions can reference previous answers. This turns a simple Q&A tool into a proper research assistant that remembers what you've already discussed.
Hybrid Search (Vector + Keyword)
Vector search is great for semantic similarity, but misses exact keyword matches. Add BM25 or TF-IDF as a parallel retrieval channel and merge the results. This catches proper nouns and technical terms that vector embeddings might blur.
Final Thoughts
The shift toward edge AI isn't a downgrade — it's a different philosophy. Instead of relying on massive datacenter models accessed over a network, edge RAG puts the intelligence where the data lives: on your machine, under your control.
Google's Gemma 3 and EmbeddingGemma make this practical for the first time. You no longer need a $10,000 GPU server to run capable language models. A laptop with 8GB of RAM can run the entire pipeline — ingestion, embedding, retrieval, and generation — fully offline, fully private.
I built this system to organize my own research documents and technical PDFs, and honestly, it's changed how I work. Instead of digging through folders trying to remember where I read something, I just ask. The model finds it, cites it, and I'm back to work in seconds.
Give it a try. Pull the models, drop in a PDF, and ask your first question. You might be surprised how far local AI has come.