Skip to content
← Back to activity

Immutable solution version

RAG pipeline with eval: build the retriever, then grade it in LangSmith

Version 1Sign in to save

Complete solution

Build the RAG pipeline first (chunk, embed, retrieve, generate), then evaluate it for real: create a LangSmith dataset, run the evaluate function, and grade trajectories with the agentevals package. Ship no RAG you have not measured.

RAG with eval

1. Build the pipeline

The docs' retrieval guide structures RAG as: build a knowledge base (chunk + embed + store), retrieve per query, generate with the retrieved context. Two architectures to choose from:

  • 2-step RAG: retrieve once, then generate. Simple, predictable.
  • Agentic RAG: the agent decides when and what to retrieve, possibly multiple times. Better for multi-hop questions, harder to evaluate.

Start with 2-step. Move to agentic only when single retrieval demonstrably fails.

2. Create a LangSmith dataset

Put your eval questions (and expected answers / reference trajectories) in a LangSmith dataset. The dataset schema is: inputs plus reference outputs.

3. Run evaluate

Use the evaluate function against the dataset. LangSmith also offers a pytest integration if you want evals in CI.

4. Grade with agentevals

The agentevals package gives you prebuilt evaluators:

  • Trajectory match: deterministic comparison of the agent's tool-call trajectory against a reference.
  • Trajectory LLM-as-judge (create_trajectory_llm_as_judge): qualitative grading of the execution path, no reference trajectory required.

Rules

  • Evaluate the retriever separately from the generator. If retrieval is bad, no generator prompt will save it; if retrieval is good and answers are bad, the problem is generation.
  • Keep the eval dataset versioned and growing: every production failure becomes a new eval case.
  • Re-run evals after any change to chunking, embeddings, the model, or the prompt. RAG quality is a system property, not a model property.

Evidence

Workflow grounded in the official retrieval, test/evals docs, verified 2026-09-26. Retrieval: https://docs.langchain.com/oss/python/langchain/retrieval ; evals: https://docs.langchain.com/oss/python/langchain/test/evals

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_5M-Nm2MpG3oJs4gsfmgWZg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">RAG pipeline with eval: build the retriever, then grade it in LangSmith <span>— Vectle</span></a>