FAQ
Fixes for the most common problems when running agent evaluations withlmnr-cli eval run.
A change to the app or the image doesn't show up
A change to the app or the image doesn't show up
lmnr-cli rebuilds the lmnr.yaml images on every run, but Docker’s layer cache can still hide a change, for example a RUN step that downloads “latest”. Check the file inside a running container, then clear the cache with docker builder prune or bump a cache-busting ARG placed before the stale step.The app can't see an env var from harbor.yaml
The app can't see an env var from harbor.yaml
agents[].env reaches every command the agent runs. If the start script launches the app with env -i, it must forward each name explicitly.The trial trace has no LLM or tool calls
The trial trace has no LLM or tool calls
Either the app isn’t instrumented with Tracevell, or it didn’t receive
LMNR_TRAJECTORY_PATH. See Capture the trajectory.Job output shows [REDACTED] in odd places
Job output shows [REDACTED] in odd places
A secret’s value is very short, so every occurrence of it in the output is redacted. Use long, distinctive placeholder values.
Where are the logs for a failed trial?
Where are the logs for a failed trial?
For local runs, in
evals/<suite>/jobs/<timestamp>/. For --remote runs, Harbor’s job output stays in the sandbox: open the trial’s trace in Tracevell instead.Each local trial has agent/ (your logs and result.json), verifier/ (reward.txt, test-stdout.txt), and exception.txt with the agent’s traceback if it raised.Next steps
Run evals with lmnr-cli
Local and remote modes, flags, CI, and sandbox limits.
Capture the trajectory
Get the app’s LLM and tool calls into every trial’s trace.
Drive and grade a trial
Write the driver and the verifier, and choose where the verifier runs.
Harbor integration
Already running
harbor run on a benchmark? Add the Tracevell plugin directly.