> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Harbor agent evaluations

> Add one flag to harbor run and read every trial as a scored datapoint with a full agent transcript, for Claude Code, Codex, or your own agent.

Tracevell has a plugin for [Harbor](https://www.harborframework.com), so the runs you already launch with `harbor run` show up in Tracevell as [evaluations](https://tracevell.com/docs/evaluations/introduction.md). Every trial gets its own trace with the agent's LLM calls and tool calls, the verifier's rewards become scores, and runs on the same dataset are charted together so you can see whether a new agent or model moved the score.

Harbor is a framework for evaluating agents in sandboxed environments: it runs an agent such as Claude Code, Codex, or your own on a dataset like Terminal-Bench, then runs each task's verifier to score the result. The Tracevell plugin hooks into Harbor's job and trial events. You don't change your agent or your tasks.

> [!TIP]
>
> To evaluate your own product (the real app and its services in a container, each case seeded with its own data), see [Agent evaluations](https://tracevell.com/docs/agent-evaluations/introduction.md). `lmnr-cli eval run` builds the images, runs Harbor with this plugin, and can run trials in Tracevell's cloud sandboxes.

## Getting started

**1. Install the Tracevell SDK next to Harbor**

Install `lmnr` in the same Python environment as `harbor`:

```bash theme={null}
pip install lmnr
```

If you installed Harbor with `uv tool install harbor`, add `lmnr` to that tool environment instead:

```bash theme={null}
uv tool install harbor --with lmnr
```

Check that Harbor picked up the plugin:

```bash theme={null}
harbor plugins list
```

The output lists `tracevell` with the import path `lmnr.integrations.harbor:LaminarPlugin`.

**2. Set your project API key**

```bash theme={null}
export LMNR_PROJECT_API_KEY=your-project-api-key
```

Get the key from **Project Settings → Project API keys** in Tracevell.

**3. Run Harbor with the plugin**

Add `--plugin tracevell` to your usual command:

```bash theme={null}
harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-sonnet-5 \
  --plugin tracevell
```

The plugin logs a link to the evaluation when the job starts. Trials appear in Tracevell as they finish.

## Agents written in TypeScript or any other language

Harbor loads plugins from Python, so the Tracevell plugin ships in the Python SDK (`lmnr`), not in `@lmnr-ai/lmnr`. That doesn't limit which agents you can evaluate. The plugin reads Harbor's trial results and the agent's [ATIF](https://docs.harborframework.com/agents/atif) trajectory, not the agent process, so it reports any agent Harbor can run:

* Node.js and TypeScript agents such as Claude Code, Gemini CLI, OpenCode, and Harbor's `langgraph` agent on its Node runtime.
* Python agents, and agents in any other language Harbor can run in its sandboxes.

For a TypeScript agent, you still install `lmnr` with `pip` next to `harbor`. Nothing gets installed inside the sandbox.

## What you see in Tracevell

Each Harbor job becomes one evaluation, and each trial becomes one datapoint in it.

| Harbor | Tracevell |
| - | - |
| Job | Evaluation, named after the job. Job id, job name, and agents are stored in its metadata. |
| Dataset, e.g. `terminal-bench@2.0` | Evaluation [group](https://tracevell.com/docs/evaluations/comparing-runs.md#group-runs-to-compare-them), so runs on the same dataset are charted together. |
| Trial | Datapoint. `data` holds the task name and instruction. |
| Verifier rewards | Scores, one per reward key (for example `reward`). |
| Agent's final message | Executor output. |
| Agent, model, trial URI, exception type | Datapoint metadata. |

Every trial is its own trace. The root `EVALUATION` span holds the trial's rewards and, if the trial failed, the exception. Under it are four spans for the trial's phases, with Harbor's own timings:

* `environment_setup`
* `agent_setup`
* `agent`, an `EXECUTOR` span with the agent's total tokens and cost.
* `verifier`, an `EVALUATOR` span whose output is the rewards.

When the agent writes an ATIF trajectory (`agent/trajectory.json` in the trial directory), the plugin replays it under the `agent` span. Every model turn becomes an LLM span with the model, provider, messages, tokens, and cost. Every tool call becomes a TOOL span with its arguments and result. Subagent trajectories are nested under the tool call that started them. Open a datapoint and you read the agent's run as a [transcript](https://tracevell.com/docs/platform/viewing-traces.md), not a list of span names.

![Harbor evaluation in Tracevell with a failed trial's trace and verifier output open](https://mintcdn.com/laminarai/9KcFgYof2j-_DF4Q/images/integrations/harbor-evaluation.png?fit=max\&auto=format\&n=9KcFgYof2j-_DF4Q\&q=85\&s=451f01fa76492a6893002cbbd30cd126)

A failed Harbor trial in Tracevell: the agent's model calls and commands, and the verifier output explaining the failure.

A few details on how results are counted:

* **Retries** overwrite the same datapoint, so a trial Harbor retried shows the final attempt.
* **Trials without rewards** (for example, the agent crashed before the verifier ran) score `0`. Harbor's mean counts them the same way, so the averages in Tracevell match Harbor's.
* **Cancelled trials** get no scores and are marked `cancelled` in metadata.
* **Errors talking to Tracevell** are logged and don't stop the Harbor run, unless you set `fail_fast`.

## Options

Pass options with `--pk key=value` on `harbor run`. Most options can also be set with an environment variable.

| Option | Environment variable | Default |
| - | - | - |
| `project_api_key` | `LMNR_PROJECT_API_KEY` | None. Required. |
| `evaluation_name` | `HARBOR_LAMINAR_EVALUATION` | Harbor job name |
| `group_name` | `HARBOR_LAMINAR_GROUP` | Dataset name, or `harbor` if the job has none |
| `trajectory_spans` | | `true`. Set to `false` to skip replaying ATIF trajectories. |
| `fail_fast` | `HARBOR_LAMINAR_FAIL_FAST` | `false`. Set to `true` to fail the Harbor job when Tracevell returns an error. |
| `base_url` | `LMNR_BASE_URL` | Tracevell Cloud |
| `http_port`, `grpc_port` | | SDK defaults |

For example, to compare two agents in one group:

```bash theme={null}
harbor run --dataset terminal-bench@2.0 --agent codex --model openai/gpt-5.6-sol \
  --plugin tracevell \
  --pk evaluation_name=codex-gpt-5.6-sol \
  --pk group_name=terminal-bench-agents
```

If you pass more than one `--plugin`, target the Tracevell plugin with `--pk tracevell.group_name=terminal-bench-agents`.

### Self-hosted Tracevell

Point the plugin at your instance with `base_url`, `http_port`, and `grpc_port`, the same settings as [self-hosted evaluations](https://tracevell.com/docs/evaluations/self-hosted.md):

```bash theme={null}
harbor run --dataset terminal-bench@2.0 --agent claude-code \
  --model anthropic/claude-sonnet-5 \
  --plugin tracevell \
  --pk base_url=http://localhost \
  --pk http_port=8000 \
  --pk grpc_port=8001
```

## Next steps

> [card] [Read a failing trial](https://tracevell.com/docs/platform/viewing-traces.md)
>
> Open a low-scoring datapoint and read the agent's run in the transcript view to see where it went wrong.

> [card] [Compare runs](https://tracevell.com/docs/evaluations/comparing-runs.md)
>
> Runs in the same group share a progression chart, and you can diff two runs side by side.

> [card] [Track failure patterns](https://tracevell.com/docs/signals/introduction.md)
>
> Define a Signal such as "the agent gave up before running the tests" and count it across every trial.

> [card] [Query across trials](https://tracevell.com/docs/platform/sql-editor.md)
>
> Ask which tasks fail for every agent from the SQL editor, the SQL API, or the MCP server.

## Related integrations

> [card] [Claude Code](https://tracevell.com/docs/integrations/claude-code.md)
>
> Trace Claude Code sessions with a one-command plugin.

> [card] [Codex](https://tracevell.com/docs/integrations/codex.md)
>
> Trace Codex CLI sessions with a one-command plugin.

> [card] [OpenCode](https://tracevell.com/docs/integrations/opencode.md)
>
> Trace OpenCode coding agent sessions.

> [card] [Claude Agent SDK](https://tracevell.com/docs/integrations/claude-agent-sdk.md)
>
> Trace Claude Agent SDK runs and their subagents.

> [card] [LangChain / LangGraph](https://tracevell.com/docs/integrations/langchain.md)
>
> Trace LangChain chains and LangGraph graphs.

> [card] [All integrations](https://tracevell.com/docs/integrations.md)
>
> Browse every provider, framework, coding agent, and browser integration Tracevell supports.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.