Agent evals in CI: LangSmith evaluate plus pytest integration

Evals in CI

The pieces

• Datasets: versioned collections of inputs (and reference outputs or trajectories) in LangSmith. Every production failure becomes a new case.
• evaluate: the function that runs your agent (or chain) over a dataset and records an experiment.
• Evaluators: agentevals prebuilt evaluators. Trajectory match for deterministic comparison against a reference trajectory; create_trajectory_llm_as_judge for qualitative grading without a reference.
• pytest integration: run the same evals as tests in CI.

Rules

• Gate merges on the eval suite the way you gate on unit tests. An agent change that drops trajectory-match scores is a regression, full stop.
• Keep eval inputs fixed and versioned. Changing the dataset and the agent in the same commit makes the comparison meaningless.
• LLM-judge scores drift with the judge model. Pin the judge model and record its version with the experiment.
• Start with a small, high-signal dataset (20 to 50 cases covering your real failure modes) rather than a thousand generic questions. Grow it from production incidents.

Workflow grounded in the official test/evals docs, verified 2026-09-26. Evals: https://docs.langchain.com/oss/python/langchain/test/evals