> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# eve observability

> Trace eve durable agents in TypeScript: every model turn, tool call, token count, and cost, readable as a conversation in Tracevell.

## Overview

Tracevell is an open-source, OpenTelemetry-native observability platform for AI agents. Trace, debug, and monitor every run of an [eve](https://eve.dev/docs/introduction) durable agent by adding Tracevell as an OpenTelemetry destination. eve emits spans for agent turns, model calls, and tool calls; Tracevell captures the full run without per-call wiring. Self-host via Helm or use managed cloud.

What Tracevell can capture from an eve agent, depending on eve's content settings:

* User messages and system instructions sent to the model.
* Model output and reasoning for each turn.
* Tool calls, their arguments, and tool results.
* Token counts, latency, and cost per call.
* The model name and provider behind each call.

eve uses the AI SDK's OpenTelemetry support under the hood, so its spans follow the GenAI semantic conventions. Tracevell reads those directly and renders the run as a transcript: the user question, each model turn, tool calls, and the final answer, in order.

![eve durable agent trace in Tracevell](https://mintcdn.com/laminarai/YvT3Mg9KfKYBnvKx/images/integrations/eve-trace-view.png?fit=max\&auto=format\&n=YvT3Mg9KfKYBnvKx\&q=85\&s=0c6eee2d8322bead0ebc167a5876f121)

An eve agent turn in Tracevell: the user question, model turns, a tool call, and the final answer rendered as a transcript, with the span timeline on the right.

## Getting Started

eve discovers files under `agent/instrumentation/` automatically. Add a Tracevell destination using eve's `otelIntegration()`; eve owns the tracer provider and the agent's trace context. This setup requires `@lmnr-ai/lmnr` 0.8.51 or newer and eve 0.76.3 or newer (Node.js 24).

**1. Install**

```bash theme={null}
npm install @lmnr-ai/lmnr@latest eve@latest
```

**2. Set environment variables**

```bash theme={null}
# .env
LMNR_PROJECT_API_KEY=your-tracevell-project-api-key
```

To get the project API key, go to the Tracevell dashboard, click the project settings,
and generate a project API key. This is available both in the cloud and in the self-hosted version of Tracevell.

Specify the key at `Tracevell` initialization. If not specified,
Tracevell will look for the key in the `LMNR_PROJECT_API_KEY` environment variable.

**3. Add Tracevell as an OpenTelemetry destination**

Create `agent/instrumentation/tracevell.ts`. eve discovers the file and attaches Tracevell's span processor to its OpenTelemetry pipeline. The processor reads `LMNR_PROJECT_API_KEY` from the agent's environment.

```typescript agent/instrumentation/tracevell.ts {1-2,4-6} theme={null}
import { LaminarSpanProcessor } from "@lmnr-ai/lmnr";
import { otelIntegration } from "eve/instrumentation/otel";

export default otelIntegration({
  spanProcessors: [new LaminarSpanProcessor()],
});
```

Do not call `Laminar.initialize()` in the agent: eve owns the tracer provider.

> [!NOTE]
>
> Set `LMNR_PROJECT_API_KEY` in the **agent server's** environment, not just the eval runner's. The agent exports its own spans; the reporter exports the eval grades separately.

**4. Run your agent**

Build and start the agent as usual. Every turn is traced.

```bash theme={null}
eve build
eve start
```

Send the agent a message and the trace appears in Tracevell.

## Run evals

eve ships its own eval runner (`eve eval`). Add Tracevell as a reporter to see each run as an [evaluation](https://tracevell.com/docs/evaluations/introduction.md), with one datapoint per eval. You can [compare runs](https://tracevell.com/docs/evaluations/comparing-runs.md) and chart scores across a group.

Use eve 0.76.3 or newer with `@lmnr-ai/lmnr` 0.8.51 or newer.

**1. Register the Tracevell reporter**

Add `LaminarReporter` to the `reporters` array in your eval config. The reporter reads `LMNR_PROJECT_API_KEY` from the environment.

```typescript evals/evals.config.ts theme={null}
import { LaminarReporter } from "@lmnr-ai/lmnr";
import { defineEvalConfig } from "eve/evals";

export default defineEvalConfig({
  reporters: [
    new LaminarReporter({
      name: "weather-agent",
      groupName: "weather-agent",
    }),
  ],
});
```

`groupName` is what makes runs comparable: every run under the same group appears on one progression chart. On run start the reporter prints the URL of the evaluation to stdout.

**2. Run the evals**

```bash theme={null}
eve eval
```

Each eval writes a datapoint as soon as eve grades it. Open the URL the reporter printed to see the scores, the assertion detail, and the trace behind each row.

### Trace the judge model calls

`t.judge(...)` assertions run in the runner process, not in the agent, so the agent's instrumentation never sees them. To capture the judge's model call, register Tracevell's AI SDK telemetry in the eval config. eve calls the AI SDK's `experimental_decide` from inside its own bundle, and AI SDK v7 reads registered telemetry from a global registry, so this one registration reaches it.

```typescript evals/evals.config.ts {1-2,5} theme={null}
import { LaminarAiSdkTelemetry, LaminarReporter } from "@lmnr-ai/lmnr";
import { registerTelemetry } from "ai";
import { defineEvalConfig } from "eve/evals";

registerTelemetry(new LaminarAiSdkTelemetry());

export default defineEvalConfig({
  reporters: [new LaminarReporter({ name: "weather-agent" })],
});
```

That one line is the whole setup. Judge spans and eval spans share a single tracer provider in the runner: whichever of the reporter or the telemetry integration runs first initializes it, and the other reuses it. The reporter flushes it when the run finishes, so nothing is lost to the `process.exit()` that ends `eve eval`.

Each judge call lands as its own span under the eval trace. Without this step the scores are still recorded; only the judge's model call is missing.

### What lands on each datapoint

Every eval produces one datapoint with three scores:

| Score | Value |
| - | - |
| `eve.verdict.passed` | 1 when eve's verdict for the eval is `passed` |
| `eve.gates.passed` | 1 when every gate assertion passed |
| `eve.soft_thresholds.passed` | 1 when every soft assertion with a threshold passed |

The scores stay the same across every eval file on purpose. eve eval files assert different things, and per-assertion score columns would leave most rows empty.

Per-assertion detail goes to the datapoint metadata under `assertions`, with a shorter `failedAssertions` list when something fails. The metadata also carries the eve verdict, the session status, the eve session id, the model id, and the tools the agent called. On the trace itself, each assertion becomes an `EVALUATOR` span, so you read the grade and the run that earned it in one place.

### Reporter options

| Option | Default | Description |
| - | - | - |
| `name` | eve run name | Name of the Tracevell evaluation created for the run. |
| `groupName` | none | Group the run belongs to. Runs in one group share a progression chart. |
| `metadata` | none | Extra metadata attached to the evaluation. |
| `projectApiKey` | `LMNR_PROJECT_API_KEY` | Project API key. |
| `baseUrl` | `https://api.lmnr.ai` | Tracevell API base URL. Set this for self-hosted Tracevell. |

## Track outcomes with Signals

Traces answer *what happened on this run*. **[Signals](https://tracevell.com/docs/signals/introduction.md) answer the cross-trace question**: *how often does the agent call a tool that returns an empty result, when does a turn run more steps than expected, how often does the agent answer without calling the tool it should have*. A Signal pairs a plain-language instruction with a JSON output schema. Tracevell runs it live on new traces (Triggers) or backfills it across history (Jobs) and records a structured event every time it matches. From there you [query](https://tracevell.com/docs/platform/sql-editor.md), [cluster](https://tracevell.com/docs/signals/clusters.md), and [alert](https://tracevell.com/docs/signals/alerts.md) on events across every run.

> [!NOTE]
>
> Every new project ships with a **Failure Detector** Signal that categorizes issues on any trace over 1000 tokens. Open it from the Signals sidebar to see events as soon as your eve traces arrive.

## Query across traces

* **[SQL editor](https://tracevell.com/docs/platform/sql-editor.md)** for ad-hoc queries across traces, spans, signals, and evals.
* **SQL API** for programmatic access from scripts and pipelines.
* **[CLI](https://tracevell.com/docs/platform/cli.md)** (`lmnr-cli sql query`) for terminal-driven queries and piping JSON into shell tools or coding agents.
* **[MCP server](https://tracevell.com/docs/platform/mcp.md)** to query Tracevell from Claude Code, Cursor, or Codex.

## Troubleshooting

> [accordion] I don't see any traces in Tracevell
>
> * Confirm `LMNR_PROJECT_API_KEY` is set in the agent server's environment, not just the eval runner's.
> * Confirm `agent/instrumentation/tracevell.ts` default-exports `otelIntegration({ spanProcessors: [new LaminarSpanProcessor()] })` and that you restarted or rebuilt the agent after editing it.
> * Check that the agent and reporter use the same project key. On an existing Vercel deployment, check its tracing sampling settings too: an unsampled agent trace cannot be exported.

> [accordion] I see the model and tool calls but no message content
>
> eve controls whether prompts and responses are included in its spans. Tracevell cannot display content eve did not record. Check your agent's content settings in [eve's OpenTelemetry guide](https://eve.dev/docs/observability/otel).

> [accordion] I see the eval scores but no judge model call
>
> Add `registerTelemetry(new LaminarAiSdkTelemetry())` to `evals/evals.config.ts`. Judge assertions run in the eval runner, which the agent's instrumentation never sees.

> [accordion] A running eval row stays pending in the UI
>
> Reload the evaluation page. Rows written by the eve reporter do not stream their result into an open page yet. The stored data is complete, so a reload shows every score.

> [accordion] I see two tracer providers or duplicated spans
>
> Remove `Laminar.initialize()` or other extra tracer-provider setup from the agent. eve owns its tracer provider and attaches `agent/instrumentation/tracevell.ts` to it; adding another provider can duplicate or detach spans.
>
> This does not apply to `evals/evals.config.ts`. That file runs in the eval runner, where the reporter and the AI SDK telemetry integration share one provider.

## What's next

> [card] [Viewing traces](https://tracevell.com/docs/platform/viewing-traces.md)
>
> Read the transcript view, filter, and search across traces.

> [card] [Signals](https://tracevell.com/docs/signals/introduction.md)
>
> Detect behaviors and failures across every run, then query, cluster, and alert on them.

> [card] [SQL editor and MCP server](https://tracevell.com/docs/platform/sql-editor.md)
>
> Query traces programmatically from the UI, API, or your IDE.

> [card] [Evaluations](https://tracevell.com/docs/evaluations/introduction.md)
>
> Score agent behavior on a dataset and track it as the agent changes.

> [card] [Comparing runs](https://tracevell.com/docs/evaluations/comparing-runs.md)
>
> Read score deltas between two runs of the same eval group.

> [card] [Tracing structure](https://tracevell.com/docs/tracing/structure/overview.md)
>
> Sessions, metadata, and tags for deeper control.

## Related integrations

> [card] [Vercel AI SDK](https://tracevell.com/docs/integrations/vercel-ai-sdk.md)
>
> eve is built on the AI SDK. Trace `generateText` and `streamText` directly here.

> [card] [Mastra](https://tracevell.com/docs/integrations/mastra.md)
>
> Trace Mastra agents, workflows, and tools in TypeScript.

> [card] [Temporal](https://tracevell.com/docs/integrations/temporal.md)
>
> Trace agents that run as Temporal workflows.

> [card] [OpenAI](https://tracevell.com/docs/integrations/openai.md)
>
> Trace the OpenAI SDK directly in TypeScript and Python.

> [card] [OpenAI Agents SDK](https://tracevell.com/docs/integrations/openai-agents-sdk.md)
>
> Trace agent runs, handoffs, and tool calls.

> [card] [All integrations](https://tracevell.com/docs/integrations.md)
>
> Browse every provider, framework, coding agent, and browser integration Tracevell supports.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.