> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets for evaluations

Hardcoding datapoints in the evaluation file is fine for a quickstart. Once you're actually iterating, the dataset lives in Tracevell and the eval file just points at it. That way anyone on the team can edit the dataset without touching code, new datapoints added from production traces flow in automatically, and the dataset versioning lives with the data instead of with the script.

## Point `evaluate()` at a Tracevell dataset

Wrap the dataset name in `LaminarDataset` and pass that as `data`.

> [tab] TypeScript
>
> ```typescript theme={null}
> import { evaluate, LaminarDataset } from '@lmnr-ai/lmnr';
>
> evaluate({
>   data: new LaminarDataset('capitals-of-the-world'),
>   executor: capitalOfCountry,
>   evaluators: { accuracy, lengthOk },
>   groupName: 'capitals',
> });
> ```

> [tab] Python
>
> ```python theme={null}
> from lmnr import evaluate, LaminarDataset
>
> evaluate(
>     data=LaminarDataset("capitals-of-the-world"),
>     executor=capital_of_country,
>     evaluators={"accuracy": accuracy, "length_ok": length_ok},
>     group_name="capitals",
> )
> ```

The dataset name matches what you see in the [Datasets](https://tracevell.com/docs/datasets/introduction.md) section of Tracevell. Each dataset row's `data`, `target`, and `metadata` fields become the datapoint's `data`, `target`, `metadata`, and the row's `task_id` becomes its [task id](https://tracevell.com/docs/evaluations/concepts.md#tasks-and-trials). Because the task id travels with the row, a run over `take` / `select` / `shuffle` of the dataset still compares cleanly against a full run.

> [!NOTE]
>
> `LaminarDataset` takes an optional `fetchSize` / `fetch_size` parameter controlling how many datapoints are fetched per network round-trip (default `25`). For performance, set it to a multiple of the evaluation concurrency / batch size (default `5`).

## Getting datapoints into the dataset

Three paths, in rough order of how often they get used:

* **Add from a trace**. On any trace in Tracevell, click **Add to dataset**. This is how you turn production failures into regression fixtures. See [Adding data](https://tracevell.com/docs/datasets/adding-data.md).
* **Upload a CSV or JSONL**. Useful for bootstrapping with external data.
* **Push via the SDK**. See the [client reference](https://tracevell.com/docs/sdk/client.md) for `LaminarClient.datasets`.

## Custom dataset sources

If the data lives somewhere Tracevell doesn't own (a production database, an S3 bucket, a remote API), extend `EvaluationDataset` and implement two methods: `size` and `get`.

> [tab] TypeScript
>
> ```typescript theme={null}
> import { EvaluationDataset, evaluate } from '@lmnr-ai/lmnr';
>
> class DatabaseDataset extends EvaluationDataset {
>   constructor(private rows: Array<{ country: string; capital: string }>) {
>     super();
>   }
>
>   async size() {
>     return this.rows.length;
>   }
>
>   async get(index: number) {
>     const row = this.rows[index];
>     return { data: { country: row.country }, target: row.capital };
>   }
> }
>
> evaluate({
>   data: new DatabaseDataset(await loadFromDb()),
>   executor: capitalOfCountry,
>   evaluators: { accuracy },
> });
> ```

> [tab] Python
>
> ```python theme={null}
> from lmnr import EvaluationDataset, Datapoint, evaluate
>
>
> class DatabaseDataset(EvaluationDataset):
>     def __init__(self, rows: list[dict]):
>         super().__init__()
>         self.rows = rows
>
>     def __len__(self):
>         return len(self.rows)
>
>     def __getitem__(self, index: int) -> Datapoint:
>         row = self.rows[index]
>         return Datapoint(
>             data={"country": row["country"]},
>             target=row["capital"],
>         )
>
>
> evaluate(
>     data=DatabaseDataset(load_from_db()),
>     executor=capital_of_country,
>     evaluators={"accuracy": accuracy},
> )
> ```

The class is modelled on PyTorch's `Dataset`. Tracevell's evaluation runner calls `size()` once, then `get(i)` for every index, and fans execution out in parallel.

## Export evaluator outputs back into a dataset

After an evaluation completes, you can turn any subset of its rows into a new dataset. From the **[SQL editor](https://tracevell.com/docs/platform/sql-editor.md)**, write a query against `evaluation_trials` or `spans`, then click **Export to Dataset**. That dataset becomes your regression set for the next iteration of the prompt or model.

## Next steps

> [card] [Compare runs](https://tracevell.com/docs/evaluations/comparing-runs.md)
>
> Keep the dataset constant so comparisons across runs are apples-to-apples.

> [card] [Datasets overview](https://tracevell.com/docs/datasets/introduction.md)
>
> How datasets are modelled in Tracevell and how to populate them.

> [card] [SDK reference](https://tracevell.com/docs/sdk/evaluations.md)
>
> Full parameters for `evaluate`, `LaminarDataset`, and `EvaluationDataset`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.