> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Evaluation Concepts - Tasks, Trials, and Verifiers

An [agent evaluation](https://tracevell.com/docs/agent-evaluations/introduction.md) suite is a folder of [Harbor](https://www.harborframework.com) tasks with a small amount of Tracevell configuration on top. This page defines each piece and shows where it appears in Tracevell once you [run the suite](https://tracevell.com/docs/agent-evaluations/running-evals.md).

![One agent evaluation trial. The task folder's task.toml picks the image, and its environment/ files are copied into a fresh container that holds your app and its services. The driver calls the app, and the app writes /logs/agent/result.json and /logs/agent/trajectory.json. The task's tests/ folder arrives only when the verifier runs; /tests/test.sh reads result.json and writes /logs/verifier/reward.json. Tracevell records the trajectory as the trial's trace and the reward as its scores.](https://mintcdn.com/laminarai/YtBM8mQsu02Qfp9Z/images/agent-evaluations/trial.svg?fit=max\&auto=format\&n=YtBM8mQsu02Qfp9Z\&q=85\&s=0aea78d27a4de83b2277cb06e75847cc)

Each trial gets a fresh container from your image. The case data goes in when the container starts, the verifier's files only when it runs. The trajectory and the reward come back to Tracevell.

## What a suite is made of

Harbor's vocabulary maps onto evals of your own product like this:

* **Task (case)**: one scenario. A directory with a `task.toml` (which image, resources, timeouts), an `instruction.md`, the case's data in `environment/`, and a verifier in `tests/`. In production evals, tasks are variations of the same scenario: same code and services, different data.
* **Environment**: the container where a trial runs, started from one image that holds your app and its services. The task's `environment/` files are copied into it at the start.
* **Agent**: the code that drives the trial. In Harbor's original use this is a general-purpose agent like Claude Code. In agent evaluations of your product, it's a thin class that starts your stack, seeds the case, and calls your app's entry point.
* **Verifier**: `tests/test.sh`, run after the agent finishes. It grades the result with a check, a rubric, or an LLM judge, and writes a reward, such as `1` or `0`.
* **Trial**: one attempt at one task: start the environment, run the agent, run the verifier. Run a task several times with `--n-attempts` to measure variance.
* **Suite**: a directory in `evals/` with a `harbor.yaml` (a Harbor job config) and its tasks. Most projects have one suite.
* **Trajectory**: the app's LLM calls and tool calls during the trial, shown in the trial's trace.

## Where each piece shows up in Tracevell

Each concept maps to something you can open in Tracevell:

| Harbor | Tracevell |
| - | - |
| Suite | Evaluation [group](https://tracevell.com/docs/evaluations/comparing-runs.md#group-runs-to-compare-them). Every run of the suite is charted together, local and remote. |
| One `lmnr-cli eval run` of a suite | Evaluation |
| Trial | Datapoint, with its own trace |
| Verifier rewards | Scores, one per reward key |
| Trajectory | `LLM` and `TOOL` spans under the trial's `agent` span |

The trace shape and how retries, crashes, and cancellations are counted are the same as for the [Harbor integration](https://tracevell.com/docs/integrations/harbor.md#what-you-see-in-tracevell).

## Next steps

> [card] [Quickstart](https://tracevell.com/docs/agent-evaluations/quickstart.md)
>
> Have your coding agent set up a suite, or build one by hand.

> [card] [The evals folder](https://tracevell.com/docs/agent-evaluations/evals-folder.md)
>
> Reference for `lmnr.yaml`, `harbor.yaml`, `task.toml`, and secrets.

> [card] [Compare runs](https://tracevell.com/docs/evaluations/comparing-runs.md)
>
> Every run of a suite is charted in its group. Diff two runs side by side to see which cases regressed.

> [card] [Harbor integration](https://tracevell.com/docs/integrations/harbor.md)
>
> Already running `harbor run` on a benchmark? Add the Tracevell plugin directly.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.