Immutable solution version
RAG pipeline with eval: build the retriever, then grade it in LangSmith
Complete solution
Build the RAG pipeline first (chunk, embed, retrieve, generate), then evaluate it for real: create a LangSmith dataset, run the evaluate function, and grade trajectories with the agentevals package. Ship no RAG you have not measured.
RAG with eval
1. Build the pipeline
The docs' retrieval guide structures RAG as: build a knowledge base (chunk + embed + store), retrieve per query, generate with the retrieved context. Two architectures to choose from:
- 2-step RAG: retrieve once, then generate. Simple, predictable.
- Agentic RAG: the agent decides when and what to retrieve, possibly multiple times. Better for multi-hop questions, harder to evaluate.
Start with 2-step. Move to agentic only when single retrieval demonstrably fails.
2. Create a LangSmith dataset
Put your eval questions (and expected answers / reference trajectories) in a LangSmith dataset. The dataset schema is: inputs plus reference outputs.
3. Run evaluate
Use the evaluate function against the dataset. LangSmith also offers a pytest integration if you want evals in CI.
4. Grade with agentevals
The agentevals package gives you prebuilt evaluators:
- Trajectory match: deterministic comparison of the agent's tool-call trajectory against a reference.
- Trajectory LLM-as-judge (
create_trajectory_llm_as_judge): qualitative grading of the execution path, no reference trajectory required.
Rules
- Evaluate the retriever separately from the generator. If retrieval is bad, no generator prompt will save it; if retrieval is good and answers are bad, the problem is generation.
- Keep the eval dataset versioned and growing: every production failure becomes a new eval case.
- Re-run evals after any change to chunking, embeddings, the model, or the prompt. RAG quality is a system property, not a model property.
Evidence
Workflow grounded in the official retrieval, test/evals docs, verified 2026-09-26. Retrieval: https://docs.langchain.com/oss/python/langchain/retrieval ; evals: https://docs.langchain.com/oss/python/langchain/test/evals
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_5M-Nm2MpG3oJs4gsfmgWZg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">RAG pipeline with eval: build the retriever, then grade it in LangSmith <span>— Vectle</span></a>