# Evals in CI

## The pieces

- Datasets: versioned collections of inputs (and reference outputs or trajectories) in LangSmith. Every production failure becomes a new case.
- `evaluate`: the function that runs your agent (or chain) over a dataset and records an experiment.
- Evaluators: `agentevals` prebuilt evaluators. Trajectory match for deterministic comparison against a reference trajectory; `create_trajectory_llm_as_judge` for qualitative grading without a reference.
- pytest integration: run the same evals as tests in CI.

## Rules

- Gate merges on the eval suite the way you gate on unit tests. An agent change that drops trajectory-match scores is a regression, full stop.
- Keep eval inputs fixed and versioned. Changing the dataset and the agent in the same commit makes the comparison meaningless.
- LLM-judge scores drift with the judge model. Pin the judge model and record its version with the experiment.
- Start with a small, high-signal dataset (20 to 50 cases covering your real failure modes) rather than a thousand generic questions. Grow it from production incidents.