Evaluation concepts

​

Datapoints

{
    "data": { "country": "France" },
    "target": "Paris",
    "metadata": { "category": "europe" },
}

• data is whatever the executor receives. It can be a primitive, a dict, or a deeply nested object. Whatever shape you put in, the executor’s first parameter has to match.
• target is optional. It’s the reference value evaluators compare against. You can omit it entirely if your evaluators don’t need it (for example if an evaluator only checks output format).
• metadata is optional. Use it to filter or query datapoints later: model name, category, source dataset, anything you want on the row.

​

Tasks and trials

evaluate({
  data: [
    { data: { country: 'France' }, target: 'Paris', taskId: 'france' },
    { data: { country: 'Japan' }, target: 'Tokyo', taskId: 'japan' },
  ],
  executor: capitalOfCountry,
  evaluators: { accuracy },
  groupName: 'capitals',
  config: { trials: 3 },
});

from lmnr import Datapoint, evaluate

evaluate(
    data=[
        Datapoint(data={"country": "France"}, target="Paris", task_id="france"),
        Datapoint(data={"country": "Japan"}, target="Tokyo", task_id="japan"),
    ],
    executor=capital_of_country,
    evaluators={"accuracy": accuracy},
    group_name="capitals",
    trials=3,
)

1. Explicit taskId / task_id on the datapoint.
2. The dataset row’s task id, when the datapoint comes from a Tracevell dataset.
3. The datapoint’s position in the source data, as a string ("0", "1", …). This is the position before any take / select / shuffle, so running a subset of the dataset keeps each datapoint’s task id.

​

Executor

def capital_of_country(data: dict) -> str:
    response = client.chat.completions.create(
        model="gpt-5-mini",
        messages=[{"role": "user", "content": f"Capital of {data['country']}?"}],
    )
    return response.choices[0].message.content or ""

​

Evaluators

def accuracy(output: str, target: str) -> int:
    return 1 if target.lower() in output.lower() else 0

• A number: one score dimension, named after the evaluator key.
• A dict of numbers: many score dimensions from one function. Useful when one pass over the output produces multiple metrics ({ "precision": 0.9, "recall": 0.8 }).

​

Two kinds of evaluator you’ll actually write

• Code evaluators. Pure functions. Exact match, regex, JSON schema check, string length. Fast, deterministic, free.
• LLM-as-a-judge evaluators. A function that calls an LLM to score the output. Use when code can’t capture the quality dimension (tone, helpfulness, faithfulness).

​

Groups

• The progression chart on the evaluations page shows the average of each score dimension over time for the group. This is how you see regressions.
• Side-by-side comparison between two runs is only enabled for runs in the same group.

​

The trace every trial produces

• Root span: EVALUATION span covering the entire Trial.
• Executor child: one EXECUTOR span with the executor’s input and output.
• Any LLM / tool spans your executor makes, auto-instrumented, nested under the executor.
• Evaluator children: one EVALUATOR span per evaluator, each recording the score and the input it scored.

SELECT trace_id, input, output
FROM spans
WHERE span_type = 'EXECUTOR'
  AND evaluation_id = '<evaluation_id>'
  AND trace_id IN (
    SELECT trace_id FROM spans
    WHERE span_type = 'EVALUATOR'
      AND name = 'accuracy'
      AND attributes['lmnr.span.output'] = '0'
  )

​

What gets stored where

• Postgres evaluations: one row per run. Holds the name, group, and metadata.
• ClickHouse evaluation_datapoints: one row per Trial per run. Holds data, target, metadata, executor output, scores, task_id, trace_id, duration, cost. The SQL editor exposes it as evaluation_trials (evaluation_datapoints still works as an alias).
• ClickHouse spans: the EVALUATION / EXECUTOR / EVALUATOR spans and everything nested under them.

​

Next steps