> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Agent Evaluations with lmnr-cli eval run

`lmnr-cli eval run` runs every [agent evaluation](https://tracevell.com/docs/agent-evaluations/introduction.md) suite in your `evals/` folder, locally on Docker or in Tracevell's cloud sandboxes with `--remote`, and records each trial in Tracevell with its trace.

Run it from the repo root, since that is where it looks for `.env` and `.env.local`. Use **local** mode while you iterate, after every change. Use **`--remote`** to run many more trials at once. `lmnr-cli` uploads the images, starts the run in Tracevell, and returns. Results appear in the project as trials finish.

| | Local | `--remote` |
| - | - | - |
| Harbor runs | On your machine | In a Tracevell sandbox |
| Trials run | On your Docker | In Tracevell sandboxes, one per trial |
| You need | Docker, uv, `LMNR_PROJECT_API_KEY` | Docker, `lmnr-cli login` (or `LMNR_PROJECT_API_KEY` in CI) |
| Harbor's job output | `evals/<suite>/jobs/` | Stays in the sandbox: debug from the trial's trace |

> [!NOTE]
>
> Both modes need Docker on the machine where you run `lmnr-cli`, because that is where the images are built. In remote mode the built images are then uploaded.

```bash theme={null}
lmnr-cli eval run                                    # every suite in ./evals
lmnr-cli eval run --suite support --task "duplicate-*" --n-attempts 3
lmnr-cli eval run --task duplicate-charge        # one case
lmnr-cli eval run --remote --n-concurrent 20
lmnr-cli eval run --remote --model anthropic/claude-sonnet-5
```

| Flag | What it does |
| - | - |
| `[dir]` | The evals folder. Defaults to `evals`. |
| `--suite <name>` | Run only this suite. Repeatable. |
| `--task <glob>` | Run only tasks matching the glob. Repeatable. |
| `--n-attempts <n>` | Trials per task. Overrides `harbor.yaml`. |
| `--n-concurrent <n>` | Concurrent trials. Overrides `harbor.yaml`. |
| `--model <name>` | Agent model. Overrides `harbor.yaml`. |
| `--env KEY[=VALUE]` | Extra agent env var. Repeatable. A bare `KEY` is read from your environment. |
| `--json` | Print a JSON summary instead of logs. |

### In CI

Use `--json` and the exit code. A local run prints `{mode, suites, failed, url}`; a `--remote` run prints `{mode: "remote", uploadId, sandboxId, suites, url}` once the run has started. Set `LMNR_PROJECT_API_KEY` and the secrets your `harbor.yaml` references as CI secrets.

| Exit code | Meaning |
| - | - |
| `16` | Invalid evals folder, or the packed folder is over the upload limit |
| `17` | `uv` or Docker isn't installed |
| `18` | A suite's `harbor run` failed. The other suites still run. |
| `19` | A `${VAR}` placeholder or `LMNR_PROJECT_API_KEY` is unset |
| `20` | A Docker build, pull, or save failed, or an image isn't `linux/amd64` |

### Remote limits and image requirements

* **One container per trial.** docker-compose tasks don't run remotely. `cpus` and `memory_mb` above the sandbox limit are clamped, and a run that exceeds the time limit is stopped.
* **The folder upload is capped at 1 GiB.** Symlinks, `.git`, `node_modules`, virtualenvs, caches, and `jobs/` directories are skipped, and `.env*` files are never uploaded. Keep large fixtures out of `evals/`.
* **Images need `tar`, `gzip`, and `bash`**, plus `su` if a task runs as a non-root user. They're built for `linux/amd64` (emulated on Apple silicon), so architecture-specific downloads should use `ARG TARGETARCH`.
* **Concurrency is capped per project** (50 sandboxes by default). A higher `--n-concurrent` queues rather than running more at once.
* **Large images** may need a higher `build_timeout_sec` under `[environment]` in `task.toml` (default 600 seconds), which bounds image preparation plus sandbox start.
* **Self-hosted Tracevell:** set `LMNR_GRPC_PORT` in your shell or `.env` for `--remote` runs, or the trial traces don't arrive.

## Next steps

> [card] [The evals folder](https://tracevell.com/docs/agent-evaluations/evals-folder.md)
>
> Reference for `lmnr.yaml`, `harbor.yaml`, `task.toml`, and secrets.

> [card] [Compare runs](https://tracevell.com/docs/evaluations/comparing-runs.md)
>
> Every run of a suite is charted in its group. Diff two runs side by side to see which cases regressed.

> [card] [Read a failing trial](https://tracevell.com/docs/platform/viewing-traces.md)
>
> Open a low-scoring datapoint and read the app's run as a transcript.

> [card] [Troubleshooting](https://tracevell.com/docs/agent-evaluations/troubleshooting.md)
>
> Fixes for stale images, missing env vars, and empty traces.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.