# RAG with eval

## 1. Build the pipeline

The docs' retrieval guide structures RAG as: build a knowledge base (chunk + embed + store), retrieve per query, generate with the retrieved context. Two architectures to choose from:

- 2-step RAG: retrieve once, then generate. Simple, predictable.
- Agentic RAG: the agent decides when and what to retrieve, possibly multiple times. Better for multi-hop questions, harder to evaluate.

Start with 2-step. Move to agentic only when single retrieval demonstrably fails.

## 2. Create a LangSmith dataset

Put your eval questions (and expected answers / reference trajectories) in a LangSmith dataset. The dataset schema is: inputs plus reference outputs.

## 3. Run evaluate

Use the `evaluate` function against the dataset. LangSmith also offers a pytest integration if you want evals in CI.

## 4. Grade with agentevals

The `agentevals` package gives you prebuilt evaluators:

- Trajectory match: deterministic comparison of the agent's tool-call trajectory against a reference.
- Trajectory LLM-as-judge (`create_trajectory_llm_as_judge`): qualitative grading of the execution path, no reference trajectory required.

## Rules

- Evaluate the retriever separately from the generator. If retrieval is bad, no generator prompt will save it; if retrieval is good and answers are bad, the problem is generation.
- Keep the eval dataset versioned and growing: every production failure becomes a new eval case.
- Re-run evals after any change to chunking, embeddings, the model, or the prompt. RAG quality is a system property, not a model property.