> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Evaluations in Real Environments with Tracevell

Agent evaluations test your agent end to end, the way it runs in production. Every eval case gets its own isolated container: your real application with its real services (database, queue, cache) and the data that reproduces one specific scenario. A driver calls the app through an entry point it already has, and a verifier scores the artefacts of the run. You run the evals with [`lmnr-cli eval run`](https://tracevell.com/docs/agent-evaluations/running-evals), locally on Docker or in Tracevell's cloud sandboxes, and every trial lands in Tracevell with its score next to the [full trace](https://tracevell.com/docs/platform/viewing-traces.md) of the agent run.

> [!NOTE]
>
> Agent evaluations are in beta. The `lmnr-cli eval run` command, its flags, and the folder layout may still change.

## Agent evaluations and LLM evaluations

Tracevell has two ways to evaluate, based on what the evaluations look at:

* **Agent evaluations** (this section) set up a working environment for the agent, let it run, and score the whole outcome, including the state changes, and the agent trajectory.
* **[LLM evaluations](https://tracevell.com/docs/evaluations/introduction.md)** call a function in your process with `evaluate()`, then compare its output to target data with evaluator functions.

LLM evaluations work for agents too, and for a prompt, a classifier, or a short pipeline they are often the fastest option. Agent evaluations fit once the agent uses tools and acts on real systems.

| | LLM evaluations | Agent evaluations |
| - | - | - |
| Unit under test | A function in your process | Your app and its services in a container |
| Case data | `data` / `target` on a datapoint | Per-case files seeded into the environment |
| Scoring | Evaluator functions on the output | A verifier script over the resulting state, full trajectory (trace) and artifacts |
| Runs on | Your machine or CI | Local Docker, or Tracevell sandboxes with `--remote` |
| Best for | Prompts, single LLM calls, pipelines you can call directly | Agents with tools, multi-service apps, replaying production failures |

The two share the same UI views, [run comparison](https://tracevell.com/docs/evaluations/comparing-runs.md), and SQL access. Both can also coexist in one project.

## Why evaluate the agent in its environment

Unit tests are a poor fit for agents that act on the world:

* **The outcome is not in the final message.** Agents may write database rows and change the state of the world around them. Scoring just the final message misses whether the right thing actually happened.
* **The bugs are in the integration.** Downstream services may experience transient issues; agents may or may not self-heal. Fixtures hide these kinds of issues.
* **Production failures need their exact context.** Reproducing a failure means the same conversation, records, and account state.

Agent evaluations answer all three by giving every case a fresh, complete copy of the stack: your real code and services, seeded with the case's data.

* **Isolation.** Each case boots its own stack with its own data, so cases don't interfere with each other and can run in parallel, in any order.
* **Local and cloud from one folder.** `lmnr-cli eval run` runs trials on your Docker. Add `--remote` to run the same eval in Tracevell sandboxes, one per trial.

## How a run works

You keep an `evals/` folder next to your code. Each case in it is a Harbor task: image definition, the data for this scenario, and a verifier. When you run `lmnr-cli eval run`, Tracevell:

1. Builds your image from the live source tree.
2. Starts one container per trial, uploads the case's data, and runs the driver against your app.
3. Runs the verifier over what the app produced and records its reward as the score.
4. Attaches the app's LLM and tool calls to the trial's trace, so you can investigate failing scores.

Every run of a suite is charted in one evaluation group. [Concepts](https://tracevell.com/docs/agent-evaluations/concepts.md) shows what goes into a trial's container and what comes out, and maps each piece to what you see in Tracevell.

## Next steps

> [card] [Quickstart](https://tracevell.com/docs/agent-evaluations/quickstart.md)
>
> Have your coding agent set it up, or build a two-case suite by hand.

> [card] [Concepts](https://tracevell.com/docs/agent-evaluations/concepts.md)
>
> Tasks, environments, agents, verifiers, and trials, and how they map to Tracevell.

> [card] [Build a production harness](https://tracevell.com/docs/agent-evaluations/production-harness.md)
>
> Run the real services, seed realistic data, and keep the eval honest.

> [card] [Track failures with Signals](https://tracevell.com/docs/signals/introduction.md)
>
> Find the production traces worth turning into cases.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.