> ## Documentation Index > Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # The Evals Folder Reference for lmnr.yaml, harbor.yaml, and task.toml This page is the reference for the files `lmnr-cli eval run` reads. For a guided first suite, start with the [quickstart](https://tracevell.com/docs/agent-evaluations/quickstart.md). `lmnr-cli eval run` treats every directory directly inside `evals/` that has a `harbor.yaml` as a suite. A fuller layout, for an app with services and a database: ``` evals/ README.md # how to run, how to add a case lmnr.yaml image/ Dockerfile # services, app, shared scripts, baseline data shared/ start_stack.sh # boot dependencies + the app, wait for readiness seed_baseline.* # (optional) data every case shares seed_case.* # load this case's data from environment/ run_case.* # drive the entry point, write result files export_trajectory.* # (optional) rebuild the trajectory if the app isn't instrumented / harbor.yaml agent/ __init__.py agent.py tasks/ / task.toml instruction.md environment/ case.json # any format seed_case reads tests/ test.sh expected.json # (optional) reference answer ``` ## lmnr.yaml and the image `lmnr.yaml` lists the images tasks can use, keyed by the name a task puts in `docker_image`: ```yaml evals/lmnr.yaml theme={null} images: myapp-evals: # referenced as docker_image = "myapp-evals" dockerfile: image/Dockerfile context: .. # the repo root, so COPY can reach the app's source myapp-evals-browser: # a second image, e.g. with a headless browser dockerfile: image/browser/Dockerfile # no context: builds from image/browser/ ``` | Key | Required | What it does | | - | - | - | | `images.` | Yes | The image name tasks reference with `docker_image`. | | `images..dockerfile` | Yes | Path to the Dockerfile, relative to `evals/`. | | `images..context` | No | Docker build context, relative to `evals/`. Defaults to the Dockerfile's directory. | A run builds only the images its selected tasks reference. Entries that no selected task uses are skipped. `lmnr-cli` rebuilds them on every run. When nothing changed, Docker reuses the cached build from earlier runs, so there are no tags to bump and no registry to push to. `lmnr.yaml` is optional: without it, tasks pull `docker_image` from a registry or build their own `environment/Dockerfile`. Two things matter when you write the Dockerfile: * **Build from your live source**, so the evals exercise your production code. * **`COPY` the specific paths you need instead of `COPY . .`.** A broad copy from the repo root would bake every suite's `tests/` folder, including the reference answers, into the image where the agent can read them. From `evals/`, copy only `image/shared/`. It also helps to order layers from rarely to often changing (system packages, dependencies, shared scripts, app code), so an app change rebuilds one small layer. `.dockerignore` applies, and builds started by `lmnr-cli` never receive `.git` or `.env*` files. ## harbor.yaml A suite's `harbor.yaml` is a plain Harbor job config: `agents`, `datasets`, `n_concurrent_trials`, etc. Relative paths resolve from the suite directory, and the suite directory is on `PYTHONPATH`, so `import_path: agent.entry:ClassName` points at `ClassName` in `/agent/entry.py`. ## task.toml Set `docker_image` to an image name from `lmnr.yaml` and leave the Dockerfile out of `environment/`. Every trial then starts from the same built image, and Harbor uploads the contents of `environment/` into `workdir` after the container starts. That is how `case.json` reaches a shared image. ```toml task.toml theme={null} [environment] docker_image = "myapp-evals" workdir = "/case" # where environment/ is uploaded; defaults to the image's WORKDIR cpus = 2 memory_mb = 4096 [agent] timeout_sec = 600 ``` A full stack per case is heavy, so set `cpus` and `memory_mb`. Locally all trials share your machine, so keep `n_concurrent_trials` low. Remotely, each trial gets its own sandbox, so you can raise concurrency for that run with `--n-concurrent`. An image name that isn't in `lmnr.yaml` is pulled from its registry, so public images work as-is. ### A Dockerfile for one task This is a rare case: most suites never need it. If one case needs a different system setup, put a Dockerfile in that task's `environment/` folder and remove `docker_image` from its `task.toml` (while `docker_image` is set, Harbor ignores the Dockerfile). The `environment/` folder then becomes the build context, and nothing is uploaded when the container starts. So the Dockerfile has to `COPY` the case data itself. It also has to start `FROM` a public base image, because images from `lmnr.yaml` aren't available to it. ## Secrets and env vars Write every secret and setting as a `${VAR}` placeholder, and `lmnr-cli` fills them in when the run starts. Where you put the placeholder decides who receives the value: ```yaml evals/support/harbor.yaml theme={null} agents: - import_path: agent.agent:SupportAgent env: ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY} # the agent and the app it starts LOG_LEVEL: ${LOG_LEVEL:-info} # optional, with a default ``` ```toml task.toml theme={null} [verifier.env] ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}" # the verifier, e.g. for an LLM judge [environment.env] TZ = "UTC" # the container itself ``` Values come from your shell first, then from `.env.local` and `.env` in the current directory. With `--remote` they travel with the run as environment variables, and files that contain secrets are never uploaded. Commit a `.env.example` that lists the names, so teammates know what to fill in. ## Next steps > [card] [Run evals with lmnr-cli](https://tracevell.com/docs/agent-evaluations/running-evals.md) > > Local and remote modes, flags, CI, and sandbox limits. > [card] [Build a production harness](https://tracevell.com/docs/agent-evaluations/production-harness.md) > > Run the real services, seed realistic data, and keep the eval honest. > [card] [Example project](https://github.com/lmnr-ai/lmnr-python/tree/main/examples/agent-evals) > > A complete `evals/` folder for a support agent, ready to run with `lmnr-cli eval run`. > [card] [Troubleshooting](https://tracevell.com/docs/agent-evaluations/troubleshooting.md) > > Fixes for stale images, missing env vars, and empty traces. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.