> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Evaluations Quickstart with lmnr-cli

This quickstart gets a first [agent evaluation](https://tracevell.com/docs/agent-evaluations/introduction.md) suite running with `lmnr-cli eval run`. The fastest path is to let your coding agent build it with the Tracevell skill. If you would rather see every file, follow [Get started manually](#get-started-manually) below.

## Set up with your coding agent

You need [Docker](https://docs.docker.com/get-docker/), [uv](https://docs.astral.sh/uv/), and Node.js. From the repo root, link the directory to a Tracevell project:

```bash theme={null}
npx lmnr-cli@latest setup
```

`setup` signs you in, writes `LMNR_PROJECT_API_KEY` to `.env`, and installs the Tracevell agent skill, which knows the evals folder layout.

Then prompt your coding agent. For example:

> *Set up agent evaluations for the refund flow with `lmnr-cli eval run`. Use our real database and seed each case from a production trace.*

If you don't have production traces yet, describe in plain words what you want to evaluate.

The skill follows the same guidance as these docs: it finds the entry point, builds one image, seeds per-case data, writes the verifier, and runs the suite locally. Review what it wrote against [Build a production harness](https://tracevell.com/docs/agent-evaluations/production-harness.md), then open the run in Tracevell.

## Get started manually

This walkthrough evaluates a support agent that handles billing tickets. It looks up a customer's charges in a SQLite database and can refund one. Each case starts with its own copy of the database, and the verifier checks two things: whether the right refund is in the database, and whether the reply to the customer is any good. An LLM judge grades the reply.

The complete project is [`examples/agent-evals`](https://github.com/lmnr-ai/lmnr-python/tree/main/examples/agent-evals) in the `lmnr-python` repo. Clone it to follow along, or copy the files into your own repo as you go.

The agent is a single module, [`app/support_agent.py`](https://github.com/lmnr-ai/lmnr-python/blob/main/examples/agent-evals/app/support_agent.py). It runs an Anthropic tool loop over two tools, `list_charges` and `refund_charge`. It calls `Laminar.initialize()` and wraps the tools in `observe`, so each trial's trace shows the model calls and the tool calls. Trimmed to the parts the harness relies on:

```python app/support_agent.py theme={null}
import os

from lmnr import Tracevell, observe

Laminar.initialize()

DB_PATH = os.environ.get("APP_DB", "/srv/app/app.db")
MODEL = os.environ.get("AGENT_MODEL", "claude-sonnet-5")


@observe(span_type="TOOL")
def list_charges(email: str) -> list[dict]: ...


@observe(span_type="TOOL")
def refund_charge(charge_id: str) -> dict: ...


@observe()
def handle_ticket(ticket: str) -> str:
    ...  # call the model, run the tools it asks for, return its reply
```

The driver imports `DB_PATH` and `handle_ticket`, and the Harbor agent sets `AGENT_MODEL`. The app's dependencies, `anthropic` and `lmnr`, are in the project's `pyproject.toml` and `uv.lock`, which the image installs.

**1. Install the prerequisites and link the project**

You need [Docker](https://docs.docker.com/get-docker/), [uv](https://docs.astral.sh/uv/), and Node.js for `lmnr-cli`. `lmnr-cli eval run` runs a pinned version of Harbor with the Tracevell plugin through `uvx`, so you don't install Harbor yourself.

From the repo root, link the directory to a Tracevell project. Skip this if you already ran it above:

```bash theme={null}
npx lmnr-cli@latest setup
```

`setup` signs you in and writes `LMNR_PROJECT_API_KEY` to `.env`. See [Tracevell CLI](https://tracevell.com/docs/platform/cli.md) for details.

**2. Lay out the evals folder**

Create this structure next to your code:

```
evals/
  lmnr.yaml                     # the image tasks use, and how to build it
  image/
    Dockerfile
    shared/
      run_case.py               # seeds the database, calls the agent, writes result.json
  support/                      # a suite
    harbor.yaml
    agent/
      __init__.py               # empty
      agent.py
    tasks/
      duplicate-charge/         # a case
        task.toml
        instruction.md
        environment/
          case.json             # the ticket
          seed.json             # the database rows for this case
        tests/
          test.sh               # the verifier
          verify.py
          expected.json         # the reference answer
```

**3. Describe the image**

`lmnr.yaml` names the image and says how to build it. Paths are relative to `evals/`, and `context: ..` makes the repo root the build context, so the image is built from your live source:

```yaml evals/lmnr.yaml theme={null}
images:
  refund-agent-evals:
    dockerfile: image/Dockerfile
    context: ..
```

The Dockerfile starts from [uv's official image](https://docs.astral.sh/uv/guides/integration/docker/), which already has uv and Python 3.14 on Debian trixie. It installs the locked dependencies first, so they stay cached when only the app code changes:

```dockerfile evals/image/Dockerfile theme={null}
FROM ghcr.io/astral-sh/uv:python3.14-trixie-slim

ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
WORKDIR /srv/app

# Dependencies first, so they stay cached when only the app code changes.
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-install-project
ENV PATH="/srv/app/.venv/bin:$PATH"

COPY evals/image/shared/ /opt/harness/
COPY app/ ./app/

WORKDIR /case
```

The driver creates this case's database from `seed.json`, imports the production function unchanged, calls it with the ticket, and writes the outcome to `/logs/agent/result.json`:

```python evals/image/shared/run_case.py theme={null}
import json
import sqlite3
import sys
import time

sys.path.insert(0, "/srv/app")
from app.support_agent import DB_PATH, handle_ticket  # the production code path

case = json.load(open("/case/case.json"))

# Create this case's database from its seed data.
conn = sqlite3.connect(DB_PATH)
conn.executescript(
    """
    CREATE TABLE charges (
        id TEXT PRIMARY KEY, email TEXT, description TEXT, amount REAL, created_at TEXT
    );
    CREATE TABLE refunds (charge_id TEXT, amount REAL);
    """
)
for c in json.load(open("/case/seed.json"))["charges"]:
    conn.execute(
        "INSERT INTO charges VALUES (:id, :email, :description, :amount, :created_at)", c
    )
conn.commit()
conn.close()

started = time.time()
try:
    result = {"outcome": "completed", "reply": handle_ticket(case["ticket"])}
except Exception as e:
    result = {"outcome": "failed", "error": repr(e)}
result["duration_s"] = round(time.time() - started, 2)

json.dump(result, open("/logs/agent/result.json", "w"), indent=2)
print(json.dumps(result))
```

**4. Write the suite config and the agent**

`harbor.yaml` is a Harbor job config. It refers to the API key as a `${VAR}` placeholder that `lmnr-cli` fills in when the run starts, so the key itself does not end up in a file you commit:

```yaml evals/support/harbor.yaml theme={null}
agents:
  - import_path: agent.agent:SupportAgent
    model_name: anthropic/claude-sonnet-5
    env:
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
datasets:
  - path: tasks
n_concurrent_trials: 4
```

The Harbor agent runs the driver inside the container. Everything it needs is already in the image, so `install()` has nothing to do. It passes `self.model_name` on to the app, which lets `--model` override the model for a run:

```python evals/support/agent/agent.py theme={null}
from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext


class SupportAgent(BaseInstalledAgent):
    @staticmethod
    def name() -> str:
        return "support"

    async def install(self, environment: BaseEnvironment) -> None:
        pass  # everything the agent needs is already in the image

    @with_prompt_template
    async def run(
        self, instruction: str, environment: BaseEnvironment, context: AgentContext
    ) -> None:
        # "anthropic/claude-sonnet-5" -> "claude-sonnet-5"
        model = (self.model_name or "").split("/", 1)[-1]
        await self.exec_as_agent(
            environment,
            command="python /opt/harness/run_case.py",
            env={"AGENT_MODEL": model},
            timeout_sec=240,
        )
```

**5. Add a case**

The task points at the image by name. Harbor copies everything in the task's `environment/` folder into `workdir` (here `/case`) when the container starts:

```toml evals/support/tasks/duplicate-charge/task.toml theme={null}
[environment]
docker_image = "refund-agent-evals"
workdir = "/case"

[agent]
timeout_sec = 300

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
```

The `[verifier.env]` block gives the verifier the key it needs to call the LLM judge.

```markdown evals/support/tasks/duplicate-charge/instruction.md theme={null}
Handle the support ticket in case.json.
```

```json evals/support/tasks/duplicate-charge/environment/case.json theme={null}
{"ticket": "Hi, this is maria.lopez@example.com. I was charged twice for my October invoice, two identical charges of $49 on Oct 3. Can you refund one of them?"}
```

> [accordion] environment/seed.json: the database rows for this case
>
> ```json evals/support/tasks/duplicate-charge/environment/seed.json theme={null}
> {
>   "charges": [
>     {"id": "ch_8f3a1c", "email": "maria.lopez@example.com", "description": "Team plan, September", "amount": 49.0, "created_at": "2026-09-03T09:12:41Z"},
>     {"id": "ch_2b7d90", "email": "maria.lopez@example.com", "description": "Team plan, October", "amount": 49.0, "created_at": "2026-10-03T09:12:44Z"},
>     {"id": "ch_c41e66", "email": "maria.lopez@example.com", "description": "Team plan, October", "amount": 49.0, "created_at": "2026-10-03T09:12:45Z"}
>   ]
> }
> ```

> [!WARNING]
>
> The agent can read everything in `environment/`, so the reference answer must not go there. Put it in `tests/`, which Harbor only uploads when the verifier runs.

```json evals/support/tasks/duplicate-charge/tests/expected.json theme={null}
{
  "refund_count": 1,
  "refund_amount": 49.0,
  "reply_should": "Confirm that one of the two October charges was refunded, and not promise anything beyond that."
}
```

The verifier first checks the database, which is the hard evidence of what the agent did. It then asks an LLM judge whether the reply is acceptable. It writes one reward per check, and each becomes its own score in Tracevell. If the agent run itself failed, both rewards are `0` and the judge is skipped:

```bash evals/support/tasks/duplicate-charge/tests/test.sh theme={null}
#!/bin/bash
python /tests/verify.py
```

> [accordion] tests/verify.py: the database check and the LLM judge
>
> ```python evals/support/tasks/duplicate-charge/tests/verify.py theme={null}
> import json
> import sqlite3
>
> from anthropic import Anthropic
>
> expected = json.load(open("/tests/expected.json"))
> result = json.load(open("/logs/agent/result.json"))
>
> if result["outcome"] != "completed":
>     print(f"agent run did not complete: {result}")
>     json.dump({"refund": 0, "reply": 0}, open("/logs/verifier/reward.json", "w"))
>     raise SystemExit(0)
>
> reply = result["reply"]
>
> # 1. Hard check: what actually happened in the database.
> conn = sqlite3.connect("/srv/app/app.db")
> count, total = conn.execute("SELECT COUNT(*), COALESCE(SUM(amount), 0) FROM refunds").fetchone()
> refund_ok = count == expected["refund_count"] and total == expected["refund_amount"]
> print(f"refunds: {count} totalling {total}")
>
> # 2. LLM judge: is the reply to the customer any good?
> verdict = Anthropic().messages.create(
>     model="claude-haiku-4-5-20251001",
>     max_tokens=200,
>     # Structured output, so the verdict is always valid JSON.
>     output_config={
>         "format": {
>             "type": "json_schema",
>             "schema": {
>                 "type": "object",
>                 "properties": {"pass": {"type": "boolean"}, "reason": {"type": "string"}},
>                 "required": ["pass", "reason"],
>                 "additionalProperties": False,
>             },
>         }
>     },
>     messages=[
>         {
>             "role": "user",
>             "content": (
>                 f"A support agent wrote this reply to a customer:\n\n{reply}\n\n"
>                 f"The reply should: {expected['reply_should']}"
>             ),
>         }
>     ],
> )
> judge = json.loads(verdict.content[0].text)
> print(f"judge: {judge}")
>
> json.dump(
>     {"refund": int(refund_ok), "reply": int(judge["pass"])},
>     open("/logs/verifier/reward.json", "w"),
> )
> ```

Make `test.sh` executable with `chmod +x`.

To add another case, copy the `duplicate-charge` directory, then change `case.json` and `seed.json` to describe a different ticket and account, and change `expected.json` to match. For example, a customer asking for a refund on a charge that should be declined expects `"refund_count": 0`.

**6. Add secrets**

Put `ANTHROPIC_API_KEY` in `.env` at the repo root, next to the `LMNR_PROJECT_API_KEY` that `setup` wrote. `lmnr-cli` fills every `${VAR}` placeholder from your shell first, then from `.env.local` and `.env`. If one is missing, it stops before anything starts.

Keep secrets and Harbor's job output out of git:

```text .gitignore theme={null}
.env
evals/*/jobs/
```

**7. Run the evals**

From the repo root:

```bash theme={null}
npx lmnr-cli@latest eval run
```

`lmnr-cli` builds `refund-agent-evals` with your Docker, runs every task in every suite, and prints a link to the results. The `support` suite becomes an evaluation group in your project, and this run is a new evaluation in it.

**8. Read the results**

Open the evaluation. Each trial is a datapoint, with a `refund` score and a `reply` score. Because the app calls `Laminar.initialize()`, the trial's trace shows the model calls and the `list_charges` / `refund_charge` tool calls as a [transcript](https://tracevell.com/docs/platform/viewing-traces.md), so a `0` takes one click to explain.

Locally, Harbor's raw output is in `evals/support/jobs/<timestamp>/`. Each trial has `agent/` (your `result.json` and logs) and `verifier/` (`reward.json` and `test-stdout.txt`, which includes the judge's reasoning).

## Next steps

> [card] [Run evals with lmnr-cli](https://tracevell.com/docs/agent-evaluations/running-evals.md)
>
> Filter suites and tasks, run trials in Tracevell sandboxes, and wire it into CI.

> [card] [The evals folder](https://tracevell.com/docs/agent-evaluations/evals-folder.md)
>
> Reference for `lmnr.yaml`, `harbor.yaml`, `task.toml`, and secrets.

> [card] [Build a production harness](https://tracevell.com/docs/agent-evaluations/production-harness.md)
>
> Grow this single module into a full stack with services and seeded data.

> [card] [Read a failing trial](https://tracevell.com/docs/platform/viewing-traces.md)
>
> Open a low-scoring datapoint and read the app's run as a transcript.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.