> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Evaluations

LLM evaluations are how you answer one question before shipping a prompt, model, or agent change: *is this version better than the one in production*. You answer it by running the new version against a fixed set of inputs, scoring every output, and comparing the scores to the previous run.

**Tracevell LLM evaluations are the offline testing layer for your prompts, LLM pipelines, and agents.** You define a list of inputs, a function that produces an output, and one or more functions that score it. Tracevell runs them in parallel, traces every call, stores the scores, and **[compares them](https://tracevell.com/docs/evaluations/comparing-runs.md) across [runs](https://tracevell.com/docs/evaluations/concepts.md) and [groups](https://tracevell.com/docs/evaluations/comparing-runs.md#group-runs-to-compare-them)**.

> [!NOTE]
>
> LLM evaluations compare an output to target data. To evaluate a large agent end to end, in a working copy of its environment and over its whole trajectory, see [Agent evaluations](https://tracevell.com/docs/agent-evaluations/introduction.md). [Which one to use](#llm-evaluations-and-agent-evaluations) is covered below.

![Evaluation run page with a datapoint table on the left and the selected datapoint's transcript on the right](https://mintcdn.com/laminarai/O81ZcDOMmMmrw7-x/images/evaluations/eval-detail.png?fit=max\&auto=format\&n=O81ZcDOMmMmrw7-x\&q=85\&s=3d0e2eccd59c082aa992ef447b4925fc)

A single run: the datapoint table with a score per evaluator, and the selected datapoint's trace showing the executor, the model call, and both evaluator scores

## What an evaluation can help you answer

Evaluations are for questions you can answer with a score. Some of them are obvious, some are not:

* **Did the new prompt break anything?** Same dataset, new prompt, see if any scores regressed.
* **Is the cheaper model good enough for this task?** Swap `gpt-5` for `gpt-5-mini`, run the same inputs, compare.
* **Does this tool actually do what I think it does?** Score the tool's output against a target, not the agent's final answer.
* **Did the last hundred production traces expose a case my evals don't cover?** Pull the failing traces into a [dataset](https://tracevell.com/docs/datasets/introduction.md) and rerun the evaluation against them.

If you've ever changed a prompt and hoped nothing broke, this is the answer to that hope.

## Explore a run, then track progress across runs

The run view above answers *why a datapoint scored the way it did*: every datapoint and its scores on the left, the selected datapoint's full trace on the right, so you go from a score to the exact model call behind it in one click.

The group view answers *did the score move*: runs sharing a group name are charted together, one line per score dimension, so a regression on any dimension is visible the moment the run lands.

![Evaluation list for the capitals group with a trend chart at the top showing accuracy flat at 1.0 and length\_ok falling to 0.0](https://mintcdn.com/laminarai/O81ZcDOMmMmrw7-x/images/evaluations/eval-list.png?fit=max\&auto=format\&n=O81ZcDOMmMmrw7-x\&q=85\&s=aa8062432d14770edc6230319f3f6cae)

Three runs of the same evaluation, grouped: the progression chart shows length\_ok dropping from 1.0 to 0.0 on the third run

## Render eval traces your way

Every datapoint in a run has the same trace shape, so reviewing a run means reading the same few fields over and over. [Custom rendering](https://tracevell.com/docs/evaluations/custom-rendering.md) lets you replace the default trace view with one built for your eval: a small JSX component that receives the executor and evaluator spans and renders exactly what you need to judge a row.

![Evaluation trace rendered through a custom template showing a side-by-side SQL diff between model output and target](https://mintcdn.com/laminarai/XAojCVbhs9bn1a75/images/evaluations/eval-custom-renderer.png?fit=max\&auto=format\&n=XAojCVbhs9bn1a75\&q=85\&s=995ff2aad869c57f76fae6a42f21d5f0)

A text-to-SQL eval datapoint rendered through a custom template: the question, the scores, and a word diff between the model's SQL and the target

The view sticks as you click through datapoints, so reviewing a whole run takes seconds per row. See [Custom rendering](https://tracevell.com/docs/evaluations/custom-rendering.md) for the full walkthrough.

## Anatomy of an evaluation

An evaluation has four parts:

* **Datapoints**: a list of `{ data, target?, metadata? }` objects. `data` is what the executor receives, `target` is what the evaluators compare against, `metadata` is anything extra you want to filter or query on.
* **Executor**: a function that takes `data` and returns whatever you want to score: the agent's output, a tool call, a parsed field.
* **Evaluators**: one or more functions that take the executor's output (and optionally `target`) and return a number or a map of numbers. Each number is a score dimension.
* **Group name**: a string that ties related runs together so Tracevell can chart them over time.

When you call `evaluate()`, Tracevell runs the executor on every datapoint in parallel, runs each evaluator against the output, records every call as a trace, and stores the scores on the datapoint.

> [!NOTE]
>
> Every datapoint becomes one trace with an `EVALUATION` root span, an `EXECUTOR` child, and one `EVALUATOR` child per scoring function. You can query those spans in the [SQL editor](https://tracevell.com/docs/platform/sql-editor.md) the same way you query any other trace.

## Run an evaluation from code or from the CLI

Evaluations are SDK-first. You write the file once, then run it whichever way fits your workflow:

* **As a script**: `python my_eval.py` or `tsx my-eval.ts`. The script runs the evaluation and exits.
* **Via the CLI**: `lmnr eval` (Python) or `npx lmnr eval` (TypeScript) picks up every file matching the eval convention in your `evals/` directory and runs them. Use this in CI or when you want to run many evals at once.

Both paths talk to the same `evaluate()` function. The CLI is just a runner.

If you already benchmark agents with [Harbor](https://tracevell.com/docs/integrations/harbor.md), you don't need an eval file: add `--plugin tracevell` to `harbor run` and each job lands in Tracevell as an evaluation.

## LLM evaluations and agent evaluations

Tracevell has two ways to evaluate, named after what they look at:

* **LLM evaluations** (this section) call a function in your process with `evaluate()` and compare its output to target data with evaluator functions.
* **[Agent evaluations](https://tracevell.com/docs/agent-evaluations/introduction.md)** set up a working environment for the agent (your real app and services in an isolated container, seeded with one case's data), let it run, and score the whole outcome: the state it changed and the trajectory it took. `lmnr-cli eval run` runs them locally or in Tracevell's cloud sandboxes.

You can evaluate an agent with LLM evaluations, and for a prompt, a classifier, or a pipeline you can call directly, it's the fastest loop. When the answer depends on the whole system (an agent with tools, a database, background jobs), agent evaluations measure what actually happened instead of a single return value.

## Next steps

> [card] [Quickstart](https://tracevell.com/docs/evaluations/quickstart.md)
>
> Write your first evaluation, run it, and read the results.

> [card] [Concepts](https://tracevell.com/docs/evaluations/concepts.md)
>
> Datapoints, executors, evaluators, groups, and how they map to traces.

> [card] [Compare runs](https://tracevell.com/docs/evaluations/comparing-runs.md)
>
> Group runs, read the progression chart, and do side-by-side diffs.

> [card] [Datasets](https://tracevell.com/docs/evaluations/datasets.md)
>
> Back evaluations with a Tracevell dataset instead of hardcoded lists.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.