How SourceMapR Solves RAG Observability: A Practical Guide
A practical guide to implementing evidence observability in your RAG pipeline with SourceMapR.
If you've read Evidence Observability for RAG: Why Debugging RAG Pipelines Still Sucks, you understand the problem: traditional agent and LLM observability tools show you retrieved chunks, but they don't show you where those chunks came from - the original documents, the parsed data, or how documents were processed. This article shows you how SourceMapR solves this by providing complete pipeline visibility from raw documents through all intermediate steps to the final answer.
What Is SourceMapR?
SourceMapR is an open-source RAG debugging tool that provides evidence observability for RAG pipelines. Unlike traditional observability tools that only show retrieved chunks, SourceMapR maps answers back to original documents, parsed data, and all intermediate steps in your RAG pipeline. It's designed specifically for developers who need to see the complete journey from raw documents to final answers.
SourceMapR is local-first (runs on your machine, your data stays local), zero-config (two lines of code to get started), framework-agnostic (works with LangChain, LlamaIndex, and OpenAI), and MIT-licensed (no vendor lock-in, completely open source). It automatically instruments your RAG pipeline without requiring code changes, tracing everything from document loading and parsing through chunking, embedding, retrieval, and LLM calls - giving you complete visibility into how raw documents become answers.
The Problem-Solution Matrix
Here's how SourceMapR solves common RAG debugging problems:
| Problem | SourceMapR Solution |
|---|---|
| "Which chunks did the retriever return?" | See every retrieved chunk with similarity scores, plus where they came from in the original documents |
| "Where did these chunks come from?" | Map chunks back to original documents and parsed data - see the complete document processing pipeline |
| "What prompt was sent to the LLM?" | Full LLM tracing captures prompts, responses, and token counts |
| "Why did the model hallucinate?" | Click any chunk to view it in the original PDF, see how documents were parsed, and verify grounding |
| "How were my documents processed?" | See the complete pipeline: raw documents → parsed data → chunks → retrieval → answer |
| "Is my chunking strategy working?" | Compare experiments side by side, see how parsed data became chunks |
| "How do I trace LLM answers to sources?" | Complete evidence lineage from answer → chunks → parsed data → original documents |
| "What are the similarity scores?" | RAG similarity score viewer shows retrieval quality and ranking |
All of this is available through a real-time dashboard that updates as your RAG pipeline executes.
What Gets Traced: Complete RAG Evidence Tracing
SourceMapR provides comprehensive RAG observability by tracing every stage of your pipeline:
| Stage | Data Captured |
|---|---|
| Original Documents | Raw documents (PDFs, text files) with file paths and metadata |
| Document Loading | Filename, absolute path, page count, text length, full text |
| Parsing & Processing | Full extracted text with page breaks - see exactly what was parsed from each document |
| Chunking | Chunk ID, text, index, page number, char positions, metadata |
| Embeddings | Model name, vector dimensions, duration |
| Retrieval | Query, top-k results, similarity scores, source file paths |
| LLM Calls | Model, messages/prompt, response, tokens, latency, tool calls |
Key Features: RAG Evidence Tracing and Observability
1. Answer to Evidence Mapping: Trace LLM Answers to Original Documents
Every answer in your RAG system can be traced back to the exact retrieved chunks that produced it, and more importantly, back to the original documents and parsed data those chunks came from. Unlike traditional observability tools that only show retrieved chunks, SourceMapR shows you the complete path: answer → chunks → parsed data → original documents. The RAG evidence viewer shows similarity scores for each retrieved chunk, complete metadata including page numbers and file paths, retrieval ordering and ranking, and chunk text with context. Most importantly, you can click any chunk to see it in the original document and understand how the document was parsed and processed. This is essential for grounding verification - ensuring your model's answers are actually supported by the retrieved evidence and understanding where that evidence originated.
2. RAG Evidence Viewer: Map Chunks to Original Documents and Parsed Data
The RAG evidence viewer lets you click any retrieved chunk to see it highlighted in the original document (PDF, text file, etc.). This is the key differentiator from traditional observability tools - you don't just see the chunk text, you see where it came from in the original document and how the document was parsed. This provides crucial context for understanding why specific chunks were retrieved (context around the chunk), whether chunk boundaries make sense, if important context was cut off during chunking, how the document was processed, and how the chunk relates to surrounding content. You can see the complete journey: original document → parsed data → chunk → retrieval → answer. This is especially valuable for debugging RAG hallucinations - you can immediately see if the retrieved context actually supports the model's answer and trace it back to the source document.
Note: The document viewer is optimized for PDF files. HTML document support is experimental - files may render but chunk highlighting and navigation may not work as expected.
3. Full LLM Tracing: Prompt and Response Capture
Complete LLM tracing captures everything about your model calls: the exact prompt sent to the model (including all retrieved context), full response text with finish reasons, token usage (prompt tokens, completion tokens, and total tokens), latency (request duration in milliseconds), model info (model name, temperature, max_tokens, and other parameters), and tool calls (function calls and tool usage when applicable). This LLM tracing is essential for understanding token costs, debugging prompt construction, and optimizing your RAG pipeline's performance.
4. Retrieval and Chunking Strategy Debugging
SourceMapR provides complete visibility into your RAG pipeline's retrieval and chunking process, enabling similarity score debugging (see exact similarity scores for each retrieved chunk to understand retrieval quality), chunking strategy debugging (view how chunks were created with character positions, page numbers, and metadata), retrieval evaluation (compare different retrievers, embedding models, and chunk sizes), vector search debugging (understand why certain chunks ranked higher than others), and chunk boundary inspection (see if important context was lost during chunking). This is critical for optimizing your RAG pipeline. You can answer questions like "Is my chunk size too small?" or "Are my similarity scores meaningful?"
5. Experiment Tracking for RAG Pipeline Optimization
SourceMapR's experiment tracking lets you organize runs and compare configurations side-by-side. This is essential for testing different retrievers (compare vector search vs hybrid search vs reranking), optimizing chunk sizes (compare chunk-size-256 vs chunk-size-512 vs chunk-size-1024), evaluating embedding models (test different embedding models and see their impact on retrieval), and A/B testing configurations (systematically compare different pipeline configurations). This experiment tracking makes it easy to answer questions like "Which chunking strategy works best for my documents?" or "Does reranking improve my results?"
6. Real-time RAG Trace Viewer
Watch new traces appear as your RAG pipeline executes. This real-time RAG trace viewer gives you immediate feedback on evidence lineage and retrieval transparency.
7. LangChain and LlamaIndex Observability Support
SourceMapR provides comprehensive LangChain observability and LlamaIndex observability. These frameworks are fully supported:
| Framework | Documents | Chunks | Retrieval | LLM Calls |
|---|---|---|---|---|
| LlamaIndex | ✅ | ✅ | ✅ | ✅ |
| LangChain | ✅ | ✅ | ✅ | ✅ |
| OpenAI | - | - | - | ✅ |
And all without modifying your existing pipeline.
Quick Start: Using SourceMapR for RAG Observability
Getting started with SourceMapR takes less than 60 seconds. Here's how to add RAG observability to your pipeline:
Installation
git clone https://github.com/kamathhrishi/sourcemapr.git
cd sourcemapr && pip install -e .Start the Dashboard
The SourceMapR dashboard runs locally on your machine. Run sourcemapr server to start the dashboard at http://localhost:5000. You can also run it in the background with sourcemapr server -b or on a custom port with sourcemapr server -p 8080.
Instrument Your Pipeline
Add RAG observability to your existing code with just two lines:
For LlamaIndex:
from sourcemapr import init_tracing, stop_tracing
init_tracing(endpoint="http://localhost:5000")
# Your existing LlamaIndex code - unchanged
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
documents = SimpleDirectoryReader("./papers").load_data()
index = VectorStoreIndex.from_documents(documents)
response = index.as_query_engine().query("What is attention?")
stop_tracing()For LangChain:
from sourcemapr import init_tracing, stop_tracing, get_langchain_handler
init_tracing(endpoint="http://localhost:5000")
handler = get_langchain_handler()
# Your existing LangChain code - unchanged
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import ChatOpenAI
loader = PyPDFLoader("./papers/attention.pdf")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=512)
chunks = splitter.split_documents(documents)
# Pass handler to trace LLM calls
llm = ChatOpenAI(model="gpt-4o-mini")
response = llm.invoke("What is attention?", config={"callbacks": [handler]})
stop_tracing()That's it. Your code stays the same - SourceMapR automatically instruments document loading, chunking, retrieval, and LLM calls. Open http://localhost:5000 to see traces appear in real-time.
Real-World RAG Debugging Scenarios: What SourceMapR Actually Helps With
Here are concrete examples of how SourceMapR's RAG observability solves real debugging problems:
Scenario 1: "Which chunks did the retriever return, and where did they come from?"
Problem: Traditional observability tools show you retrieved chunks, but not where they came from. You can't see the original documents, the parsed data, or how documents were processed. Solution: SourceMapR's RAG evidence viewer shows every retrieved chunk with similarity scores, plus maps each chunk back to its original document and parsed data. You can see exactly which chunks were returned, their ranking, their scores, and most importantly, where they originated in the original documents and how those documents were processed.
Scenario 2: "What prompt did my LLM actually receive?"
Problem: You're not sure if your prompt construction is working correctly. You can't see the final prompt that was sent to the model. Solution: Full LLM tracing captures the exact prompt sent to the model, including all retrieved context. You can see token counts, latency, and the complete prompt/response cycle.
Scenario 3: "Why is my RAG model hallucinating?"
Problem: Your model is producing answers that aren't supported by the retrieved context. Traditional observability tools show you the chunks, but you can't see the original documents or how they were parsed. Solution: SourceMapR's evidence lineage lets you trace model answers back to original documents through all intermediate steps. Click any chunk to see it in the original document, view how the document was parsed, and see the complete pipeline from raw document → parsed data → chunk → retrieval → answer. This enables complete grounding verification - you can immediately see if the retrieved context actually supports the model's answer and understand how the original document was processed.
Scenario 4: "How to debug RAG hallucinations?"
Problem: You need a systematic way to debug hallucinations and understand where your pipeline is failing. Solution: SourceMapR provides a complete audit trail. You can see if the problem is in retrieval (wrong chunks), chunking (lost context), or prompt construction (poor context formatting). This makes it easy to identify when chunking strategy debugging is needed.
Scenario 5: "Is my chunking strategy broken?"
Problem: You're not sure if your chunk size is optimal or if important context is being lost during chunking. Solution: SourceMapR's chunking strategy debugging shows exactly how chunks were created, with character positions and page numbers. You can compare experiments with different chunk sizes to see which works best for your documents.
Scenario 6: "Is my retrieval too vague or too strict?"
Problem: You can't tell if your similarity scores are meaningful or if your retrieval is working correctly. Solution: The RAG similarity score viewer shows retrieval quality and ranking. You can inspect chunk boundaries and similarity scoring to optimize your vector search. This retrieval evaluation helps you understand if you need to adjust your embedding model or retrieval parameters.
Scenario 7: "How do I trace LLM answers to original documents?"
Problem: You need to verify that your model's answers are actually grounded in the retrieved documents, but traditional observability tools only show chunks, not the original documents or parsed data. Solution: SourceMapR provides complete evidence lineage from answer → chunks → parsed data → original documents. You can trace every answer back to its source document, see how that document was parsed and processed, and understand all intermediate steps in the pipeline. This enables explainable RAG with complete transparency from raw documents to final answers.
All of these RAG debugging capabilities are impossible with normal logs. That's why evidence observability for RAG is essential for building reliable RAG systems.
Who Should Use SourceMapR for RAG Observability?
SourceMapR is the RAG debugging tool for developers who are building with LangChain or LlamaIndex and need LangChain observability or LlamaIndex observability, want visibility into RAG internals and evidence tracing, need grounding verification to ensure model answers are properly sourced, are experimenting with chunking strategies and retrievers and need chunking strategy debugging, want an open-source, MIT-licensed RAG pipeline debugger without vendor lock-in, and are tired of print-debugging RAG systems and need a proper RAG trace viewer.
When SourceMapR Shines
SourceMapR is designed for local development (debug your RAG pipeline during development), experiment evaluation (compare different configurations and strategies), pipeline optimization (understand what's working and what's not), grounding verification (ensure your model's answers are properly sourced), and educational purposes (learn how RAG pipelines actually work).
Limitations
SourceMapR is not designed for massive enterprise monitoring at scale, production monitoring with thousands of requests per second, or multi-tenant SaaS deployments. For local development, debugging, and experimentation, SourceMapR provides the RAG observability and evidence tracing you need to build reliable RAG systems.
Conclusion: Getting Started with SourceMapR
RAG systems are complex. They involve multiple stages - document loading, parsing, chunking, embedding, retrieval, and generation - and failures can occur at any point. Traditional observability tools show you retrieved chunks, but they don't show you the original documents, the parsed data, or the intermediate processing steps. Evidence observability gives you the complete picture from raw documents to final answers.
SourceMapR provides the RAG debugging tools you need to trace LLM answers back to original documents and parsed data, see all intermediate steps in your pipeline, debug RAG hallucinations with complete grounding verification, optimize chunking strategies and retrieval, understand similarity scores and retrieval quality, compare experiments and configurations, and build explainable RAG systems with full transparency from raw documents to final answers.
If you're building RAG systems with LangChain or LlamaIndex, evidence observability isn't optional - it's essential. SourceMapR makes it easy to add RAG observability to your pipeline with just two lines of code. Get started today: https://github.com/kamathhrishi/sourcemapr
To understand why evidence observability matters, read Evidence Observability for RAG: Why Debugging RAG Pipelines Still Sucks.