Skip to main content
This page is the reference for the files lmnr-cli eval run reads. For a guided first suite, start with the quickstart. lmnr-cli eval run treats every directory directly inside evals/ that has a harbor.yaml as a suite. A fuller layout, for an app with services and a database:

​
lmnr.yaml and the image

lmnr.yaml lists the images tasks can use, keyed by the name a task puts in docker_image:
evals/lmnr.yaml
A run builds only the images its selected tasks reference. Entries that no selected task uses are skipped. lmnr-cli rebuilds them on every run. When nothing changed, Docker reuses the cached build from earlier runs, so there are no tags to bump and no registry to push to. lmnr.yaml is optional: without it, tasks pull docker_image from a registry or build their own environment/Dockerfile. Two things matter when you write the Dockerfile:
  • Build from your live source, so the evals exercise your production code.
  • COPY the specific paths you need instead of COPY . .. A broad copy from the repo root would bake every suite’s tests/ folder, including the reference answers, into the image where the agent can read them. From evals/, copy only image/shared/.
It also helps to order layers from rarely to often changing (system packages, dependencies, shared scripts, app code), so an app change rebuilds one small layer. .dockerignore applies, and builds started by lmnr-cli never receive .git or .env* files.

​
harbor.yaml

A suite’s harbor.yaml is a plain Harbor job config: agents, datasets, n_concurrent_trials, etc. Relative paths resolve from the suite directory, and the suite directory is on PYTHONPATH, so import_path: agent.entry:ClassName points at ClassName in <suite>/agent/entry.py.

​
task.toml

Set docker_image to an image name from lmnr.yaml and leave the Dockerfile out of environment/. Every trial then starts from the same built image, and Harbor uploads the contents of environment/ into workdir after the container starts. That is how case.json reaches a shared image.
task.toml
A full stack per case is heavy, so set cpus and memory_mb. Locally all trials share your machine, so keep n_concurrent_trials low. Remotely, each trial gets its own sandbox, so you can raise concurrency for that run with --n-concurrent. An image name that isn’t in lmnr.yaml is pulled from its registry, so public images work as-is.

​
A Dockerfile for one task

This is a rare case: most suites never need it. If one case needs a different system setup, put a Dockerfile in that task’s environment/ folder and remove docker_image from its task.toml (while docker_image is set, Harbor ignores the Dockerfile). The environment/ folder then becomes the build context, and nothing is uploaded when the container starts. So the Dockerfile has to COPY the case data itself. It also has to start FROM a public base image, because images from lmnr.yaml aren’t available to it.

​
Secrets and env vars

Write every secret and setting as a ${VAR} placeholder, and lmnr-cli fills them in when the run starts. Where you put the placeholder decides who receives the value:
evals/support/harbor.yaml
task.toml
Values come from your shell first, then from .env.local and .env in the current directory. With --remote they travel with the run as environment variables, and files that contain secrets are never uploaded. Commit a .env.example that lists the names, so teammates know what to fill in.

​
Next steps

Run evals with lmnr-cli

Local and remote modes, flags, CI, and sandbox limits.

Build a production harness

Run the real services, seed realistic data, and keep the eval honest.

Example project

A complete evals/ folder for a support agent, ready to run with lmnr-cli eval run.

Troubleshooting

Fixes for stale images, missing env vars, and empty traces.