# RAG with eval
## 1. Build the pipeline
The docs' retrieval guide structures RAG as: build a knowledge base (chunk + embed + store), retrieve per query, generate with the retrieved context. Two architectures to choose from:
- 2-step RAG: retrieve once, then generate. Simple, predictable.
- Agentic RAG: the agent decides when and what to retrieve, possibly multiple times. Better for multi-hop questions, harder to evaluate.
Start with 2-step. Move to agentic only when single retrieval demonstrably fails.
## 2. Create a LangSmith dataset
Put your eval questions (and expected answers / reference trajectories) in a LangSmith dataset. The dataset schema is: inputs plus reference outputs.
## 3. Run evaluate
Use the `evaluate` function against the dataset. LangSmith also offers a pytest integration if you want evals in CI.
## 4. Grade with agentevals
The `agentevals` package gives you prebuilt evaluators:
- Trajectory match: deterministic comparison of the agent's tool-call trajectory against a reference.
- Trajectory LLM-as-judge (`create_trajectory_llm_as_judge`): qualitative grading of the execution path, no reference trajectory required.
## Rules
- Evaluate the retriever separately from the generator. If retrieval is bad, no generator prompt will save it; if retrieval is good and answers are bad, the problem is generation.
- Keep the eval dataset versioned and growing: every production failure becomes a new eval case.
- Re-run evals after any change to chunking, embeddings, the model, or the prompt. RAG quality is a system property, not a model property.