Building a RAG System with LangChain, FAISS, and DeepSeek LLM

Large language models are powerful, but they hallucinate when asked about documents they haven't seen. Retrieval-Augmented Generation (RAG) solves this by grounding the model's response in real, retrieved context. In this article I'll walk through the exact pipeline I built for my RAG-LLM project — from PDF ingestion to final answer generation.

What is RAG and Why Does It Matter?

A standard LLM like GPT or DeepSeek is trained on a fixed corpus. Ask it about your company's internal policy document and it'll either refuse or make something up. RAG fixes this by retrieving the most relevant chunks of your document at query time and injecting them into the prompt as context. The model then reasons over real text rather than hallucinating.

The core loop is: ingest → embed → store → retrieve → generate. Each step has meaningful design decisions, and getting them wrong tanks quality dramatically.

Step 1 — PDF Ingestion and Chunking

I used PyPDF to extract raw text from uploaded PDFs. The tricky part is chunking — splitting text into pieces that are small enough to be semantically precise, but large enough to carry enough context for the LLM to use.

I settled on 512-token chunks with a 64-token overlap using LangChain's RecursiveCharacterTextSplitter. The overlap ensures sentences that span chunk boundaries don't lose their meaning. Too small and chunks lose context; too large and retrieval returns noisy results.

Step 2 — Embedding with sentence-transformers

Each chunk is converted to a dense vector using sentence-transformers/all-MiniLM-L6-v2 — a fast, lightweight model that produces 384-dimensional embeddings with strong semantic quality. I chose it over OpenAI embeddings to keep the system entirely local and free.

The embedding step happens once at ingest time. At query time, only the user's question is embedded — this single vector is compared against all stored chunk vectors to find the closest matches.

Step 3 — FAISS for Fast Similarity Search

FAISS (Facebook AI Similarity Search) stores all chunk embeddings in an index and retrieves the top-k nearest neighbors in milliseconds, even for thousands of chunks. I used IndexFlatL2 — exact nearest-neighbor search using L2 (Euclidean) distance. For documents under ~50 pages, this is fast enough and perfectly accurate.

LangChain wraps FAISS cleanly — you call FAISS.from_documents(chunks, embeddings) and get a retriever back. At query time, retriever.get_relevant_documents(question) returns the top 4 chunks.

Step 4 — Prompt Engineering and DeepSeek LLM

The retrieved chunks are injected into a structured prompt before being sent to the LLM. My prompt template looks like this:

You are an assistant that answers questions based ONLY on the provided context.
If the answer is not in the context, say "I don't know based on the document."

Context:
{context}

Question: {question}
Answer:

The explicit instruction to say "I don't know" is critical — it prevents the model from reverting to its training data when the document doesn't cover the question. I used DeepSeek as the LLM via Hugging Face's inference API, which gave strong reasoning quality at no cost.

Step 5 — ChromaDB as Persistent Storage

FAISS indexes are in-memory — restarting the app means re-embedding everything. For persistence, I integrated ChromaDB as an alternative backend. ChromaDB persists embeddings to disk and supports metadata filtering, making it more suitable for production deployments where documents accumulate over time.

Lessons Learned

• Chunk size is the most important hyperparameter. I ran ablations at 256, 512, and 1024 tokens — 512 gave the best precision/recall balance.
• Overlap matters more than you think. Without overlap, questions at chunk boundaries returned wrong answers.
• Retrieval quality limits LLM quality. Even a perfect LLM can't answer from bad retrieved context. Debug retrieval first.
• Prompt framing changes outputs dramatically. Adding "based ONLY on the provided context" reduced hallucinations by roughly 80% in manual testing.

Result

The final system handles PDFs of any length, returns accurate answers with source context, and runs entirely locally (except the DeepSeek API call). Latency from question to answer is typically under 3 seconds for documents under 100 pages.