Langfuse for RAG: Observability, Tracing, and Evaluation

Building a RAG pipeline is the easy part. Knowing whether it works reliably in production is not. Retrieval can return irrelevant chunks, the model can ignore the context it received, and quality can regress silently across prompt or embedding changes. Without tracing and evaluation, these failures are invisible until a user reports a bad answer.

Langfuse is an open-source LLM engineering platform that provides the observability layer for RAG systems. Combined with RAGAS for evaluation, it creates a complete quality loop: tracing makes the pipeline visible, evaluation makes it measurable, and experiments make improvement systematic. 

Let’s see how to set it up, what to measure, and how to use the data to improve your RAG system.

What Is Langfuse and Why it Matters for RAG

Langfuse is an open-source platform built specifically for the LLM application lifecycle. It provides tracing, scoring, prompt management, datasets, and cost analytics with native understanding of LLM-specific constructs: traces, spans, prompts, token usage, and evaluation scores.

For RAG systems, this matters because generic logging tools capture request and response data without understanding the internal pipeline. 

Langfuse traces each stage of the pipeline (retrieval, augmentation, generation) as separate spans within a single trace, making it possible to identify exactly where a quality failure occurred.

The Observability Gap in RAG Systems

RAG systems have three silent failure modes that only tracing can surface.

Irrelevant chunk retrieval. The vector search returns documents that match semantically but do not contain the information needed to answer the query. The model generates a plausible answer from irrelevant context, and the output looks correct but is not grounded in the right source material.

Model ignoring retrieved context. The model has the correct context in its prompt but generates an answer from its training data instead. This produces hallucinations that are difficult to detect without comparing the output against the retrieved chunks.

Quality regressions across versions. A change to the embedding model, chunk size, system prompt, or retrieval parameters degrades answer quality. Without continuous evaluation, these regressions go unnoticed until they accumulate into a visible quality drop.

Untraced RAG is unmanaged RAG. If you cannot see what the retrieval stage returned and what the model did with it, you cannot diagnose or fix quality issues.

Langfuse in the LLM Engineering Stack

Langfuse sits on top of orchestration frameworks (LangChain, LlamaIndex) and vector stores (Milvus, Pinecone, Weaviate). It does not replace them. It provides the observability and feedback layer that makes the pipeline debuggable and measurable.

Its core feature areas include tracing (capturing the full execution of each request as a tree of spans), scoring (attaching quality metrics to traces), prompt management (versioning and tracking prompt changes), datasets (curated test sets for controlled experiments), and cost analytics (tracking token usage and LLM costs per trace).

Understanding the RAG Pipeline: Key Components

Langfuse traces map directly to the stages of the RAG pipeline. Understanding the architecture clarifies what each span represents.

Retrieval Stage

Retrieval is the highest-leverage point in the pipeline and the most common source of quality failures. The query is embedded and matched against the vector store to find relevant document chunks. 

Tracing this stage reveals which chunks were fetched, how they ranked, and how they related to the original query. A low-quality answer almost always traces back to a retrieval problem.

Augmentation and Generation Stage

The retrieved chunks are combined with the query into a prompt and passed to the LLM. This is where hallucinations occur when the model drifts from the source material. 

Tracing captures the exact prompt sent to the model, the full output, and the token usage. Comparing the output against the retrieved context is how you detect whether the model grounded its answer in the provided documents.

Setting Up Langfuse for RAG Tracing

Setup requires installing the Langfuse SDK, configuring API keys, and adding tracing decorators to your pipeline functions. The @observe() decorator wraps a function and creates a trace for each invocation, capturing inputs, outputs, and timing. Nested spans are created using start_as_current_observation() to trace sub-steps like retrieval and generation as children of the root trace.

Full tracing requires only a few lines of code added to existing pipeline functions.

Integrating with LlamaIndex and Milvus

Langfuse provides a LlamaIndexCallbackHandler that auto-instruments indexing and query operations. Set the handler globally, and every LlamaIndex operation (including queries against a Milvus Lite vector store) produces a detailed trace with retrieval and generation spans. Call flush() after operations to ensure immediate trace visibility in the Langfuse UI.

Integrating with LangChain

Langfuse's CallbackHandler attaches to any LangChain chain via the config parameter. Pass the handler when invoking a chain, and Langfuse captures the full execution trace. For more granular analysis, add a retriever-typed span to isolate document fetching from generation, making it possible to evaluate each stage independently.

Tracing Your RAG Pipeline with Langfuse

A complete RAG trace consists of a root span representing the full interaction, with nested child spans for retrieval and generation. The root span captures the user query and final answer. The retrieval span captures the query, retrieved chunks, and similarity scores. The generation span captures the full prompt, model output, and token usage.

Enrich traces with metadata to enable filtering and segmentation in the Langfuse UI. Attach user_id to track quality per user. Add session_id to group multi-turn conversations. Use tags to categorize traces by feature, environment, or experiment. This metadata is what makes traces analyzable at scale rather than individually browsable.

Evaluating RAG Quality with RAGAS and Langfuse

RAGAS is an open-source evaluation framework designed specifically for RAG pipelines. Its key advantage for production use is reference-free evaluation: you do not need ground-truth labels to run it on live traces. This means you can evaluate production data continuously without maintaining a labeled dataset.

Core RAGAS Metrics

Four metrics cover the primary quality dimensions of a RAG system.

Faithfulness measures whether the answer is factually consistent with the retrieved context. A low score means the model is generating claims that are not supported by the source documents.

Answer relevancy measures whether the answer addresses the question that was asked. A low score means the model is producing responses that are off-topic or only partially relevant.

Context precision measures whether the relevant chunks are ranked higher in the retrieved set. A low score means the retrieval stage is returning relevant documents but burying them below irrelevant ones.

Context recall measures whether the retrieval stage found all the relevant information. A low score means important source material is missing from the retrieved context entirely.

Scoring Individual Traces vs. Batch Scoring

Per-trace scoring runs RAGAS metrics on every request. This gives high-resolution quality data but costs more because each evaluation makes additional LLM calls. Batch scoring samples a subset of traces periodically and scores them as a group. This reduces cost and gives an aggregate quality view but may miss individual failures.

A hybrid approach works well in practice: run per-trace scoring on a sampled percentage of production traffic for continuous monitoring, and run batch scoring on the full trace set periodically (daily or weekly) for trend analysis.

Pushing Scores Back to Langfuse

Use langfuse.create_score() to link a RAGAS metric value to a specific trace_id. This makes scores visible in Langfuse dashboards, filterable in the UI, and available for trend analysis. Scoring is a blocking operation, so run it after the answer is returned to the user or in a background thread to avoid adding latency to the response.

Iterating and Improving Your RAG Pipeline

Langfuse traces and RAGAS scores together create a feedback loop. Low scores on specific metrics point to specific improvement actions.

Chunking Strategy Experiments

Chunk size and overlap are high-leverage variables that affect both retrieval precision and generation quality. Smaller chunks increase precision but may miss context. Larger chunks provide more context but reduce relevance.

Langfuse Datasets let you isolate the retrieval component and test chunking strategies without paying full LLM generation costs on every iteration. Create a dataset of representative queries, run retrieval with different chunk configurations, and compare context precision and recall scores to find the configuration that produces the best retrieval results.

Prompt and Retrieval Optimization

Langfuse's prompt versioning links each prompt change to its downstream evaluation scores. When you modify the system prompt, the traces generated under the new version carry scores that can be compared directly against the previous version.

Map metric signals to specific fixes. Low faithfulness scores indicate the model is not grounding its answers in the retrieved context. The fix is usually stronger grounding instructions in the system prompt. Low answer relevancy scores indicate the model is not connecting its response to the original question. The fix is usually better query-context framing in the prompt template.

Monitoring RAG in Production

Langfuse enables four production monitoring mechanisms.

Cost tracking via token usage per trace. Monitor how much each RAG request costs and identify queries that consume disproportionate tokens due to large context windows or long outputs.

Latency tracking per span. Measure how long retrieval and generation take independently. Identify whether slow responses are caused by vector search latency or LLM inference time.

User feedback scores written back via the SDK. Capture explicit user feedback (thumbs up/down, ratings) and link it to traces for correlation with automated evaluation scores.

LLM-as-a-judge evaluators for near-real-time quality checks. Configure Langfuse to run an LLM evaluator on sampled production traces, providing automated quality assessment without waiting for batch evaluation runs.

Common Mistakes When Implementing Langfuse in RAG Pipelines

Three mistakes consistently undermine the value of Langfuse implementations.

Tracing Without a Scoring Strategy

Collecting traces without attaching evaluation scores creates a large dataset with no quality signal. Traces alone tell you what happened. Scores tell you whether it was good. Without scores, traces are only useful for individual debugging, not for systematic quality monitoring or trend analysis. 

Define a scoring strategy (which RAGAS metrics, at what frequency, per-trace or batch) before or alongside tracing implementation.

Ignoring Retrieval Spans and Over-focusing on Generation

Teams commonly default to optimizing the generation prompt when the real quality problem is in retrieval. If the retrieval stage returns irrelevant chunks, no prompt change will fix the output. 

Langfuse's span-level visibility makes it possible to isolate where quality is lost. Always diagnose the retrieval stage before modifying the generation prompt.

Skipping Metadata Enrichment on Traces

Traces without user_id, session_id, or tags are difficult to segment and analyze at scale. A trace that says "this answer scored 0.4 on faithfulness" is less actionable than one that says "this answer scored 0.4 on faithfulness, came from the billing FAQ use case, and was from user X in session Y." Treat metadata enrichment as a first-class concern from day one.

Langfuse RAG vs. Other Observability Approaches

Langfuse occupies a specific position in the observability landscape for LLM applications.

Langfuse vs. Generic APM Tools

Tools like Datadog and New Relic track infrastructure metrics: latency, error rates, throughput. They do not natively understand LLM-specific constructs like prompts, retrieved context, token-level costs, or evaluation scores. 

Debugging a RAG quality issue in a generic APM tool requires correlating infrastructure logs with application logs manually. Langfuse captures the full LLM interaction (prompt, context, output, score) in a single trace, making RAG debugging faster and more precise.

Langfuse vs. Other LLMOps Platforms

Alternatives in the LLMOps space include LangSmith and Arize Phoenix. Langfuse differentiates on three factors: it is fully open source, it can be self-hosted for teams with data residency requirements, and it has native RAGAS integration for reference-free evaluation. 

For teams that need full control over their observability stack or operate in regulated environments where data cannot leave their infrastructure, Langfuse's self-hosting capability is a meaningful advantage.

Final Thoughts

The Langfuse and RAGAS stack provides a complete quality loop for RAG systems. Tracing makes the pipeline visible. Evaluation makes it measurable. Datasets and experiments make improvement systematic and controlled. Without this loop, RAG development relies on manual testing and user complaints to surface quality issues.

For teams building RAG applications that need to work reliably in production, this observability layer is foundational infrastructure. It is what separates a demo from a system you can confidently deploy.

If you are building RAG systems and need engineering support for tracing, evaluation, or pipeline optimization, connect with our team at Leanware.

Frequently Asked Questions

Can Langfuse be self-hosted?

Yes. Langfuse is fully open source and can be self-hosted using Docker. This is relevant for teams with data residency requirements or those operating in regulated industries where trace data (which contains user queries and LLM outputs) cannot be sent to external cloud services. Langfuse also offers a managed cloud option for teams that prefer not to manage infrastructure.

Does RAGAS require ground-truth labels to evaluate RAG quality?

No. RAGAS performs reference-free evaluation, meaning it assesses quality using the query, retrieved context, and generated answer without requiring pre-labeled ground-truth answers. This is what makes it practical for evaluating production traces where ground-truth data does not exist.

Does Langfuse work with vector stores other than Milvus?

Yes. Langfuse is vector-store agnostic. It traces the retrieval operation regardless of which vector store is used. Langfuse has documented integrations with Milvus, Pinecone, Weaviate, Chroma, and others. The tracing captures the retrieval inputs and outputs at the application level, not the vector store level.

Does tracing with Langfuse add latency to RAG responses?

Langfuse's SDK sends trace data asynchronously, so it does not add meaningful latency to the response path. Trace data is batched and flushed in the background. RAGAS evaluation, however, involves additional LLM calls and should run after the response is returned or in a background process to avoid affecting user-facing latency.

Should I use per-trace or batch scoring for RAGAS evaluation?

Use a hybrid approach. Run per-trace scoring on a sampled percentage (5-10%) of production traffic for continuous quality monitoring. Run batch scoring on larger trace sets periodically (daily or weekly) for aggregate trend analysis. Per-trace scoring provides high resolution but costs more due to additional LLM calls. Batch scoring is cost-efficient and gives a broader quality view.