> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

## My JS auto-instrumentation is not working.

### Problem

I have instrumented my JavaScript code with Tracevell, and I can see the manual spans in the Tracevell UI, but I don't see the auto-instrumented spans.

### Cause

Tracevell auto-instrumentations are based on OpenTelemetry's auto-instrumentations,
and they are optimized for `commonjs` modules.

### Solution

**1. Make sure that you pass `instrumentModules`**

The `instrumentModules` option is required to instrument the modules you need.

```javascript {7-11} theme={null}
import { Tracevell } from '@lmnr-ai/lmnr';
import { OpenAI } from 'openai';
import Anthropic from '@anthropic-ai/sdk';

Laminar.initialize({
  projectApiKey: process.env.LMNR_PROJECT_API_KEY,
  instrumentModules: {
    OpenAI: OpenAI,
    anthropic: Anthropic,
    // ... other modules you want to instrument
  }
});
```

If this didn't solve your issue, continue with the next steps.

**2. Check if you are using ESM modules**

Due to the differences between Node versions, bundlers, and TypeScript compilers, it is not easy to definitely determine if you are using ESM modules.

Some strong indicators:

* You have a `package.json` file with `"type": "module"` – this is a 100% indicator.
* Your files have `.mjs` extension – this is a strong indicator.
* You are using `import` statements **and** this is not TypeScript – this is a strong indicator.

See more in [Node.js documentation](https://nodejs.org/api/modules.html#enabling) and [Node.js documentation on determining module system](https://nodejs.org/api/packages.html#determining-module-system).

If you are not sure, you can try to run `node --input-type=module yourfile.js` and see if it works.

**3. Try to migrate to CommonJS**

For modern Node versions, migrating to CommonJS is straightforward and most code actually
is written in ESM syntax.

It might be as simple as changing the file extension from `.mjs` to `.js` and adding `"type": "commonjs"` to your `package.json` file.

**4. \[Optional] Further reading**

Refer to the [CJS vs ESM](https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/esm-support.md) guide
in OpenTelemetry documentation for more information.

## My JS traces are not showing up in the Tracevell UI.

### Problem

I have instrumented my JavaScript code with Tracevell, but I don't see any traces in the Tracevell UI.

### Cause

This often happens in Edge runtimes, Lambda functions, or in one-off scripts that are not running in a long-lived process.

JavaScript's OpenTelemetry batch span processor runs in a background async function, and, if the process exits before the function has a chance to
send the traces, they will be lost. In theory, there is a way to force flush the pending spans `onShutdown`, but it is not implemented in the
OpenTelemetry JS SDK. Apparently, doing that may cause the consequent incoming spans to block the process.

See [originating commit](https://github.com/open-telemetry/opentelemetry-js/commit/23db7f0ba383a3043964eb03be11c09df3f7453a) where `onShutdown` was only implemented for the browser contexts,
but not for Node.js.

### Solution

Use Tracevell's `flush()` function to ensure that the traces are sent before the process exits.

```javascript {13} theme={null}
import { Tracevell } from '@lmnr-ai/lmnr';

Laminar.initialize({
  projectApiKey: process.env.LMNR_PROJECT_API_KEY,
  instrumentModules: {
    // your instrumented modules
  }
});

// your code here

// at the end of your script, when you are done tracing
await Laminar.flush(); // NOTE: don't forget to await
```

Use `Laminar.flush()` (and `shutdown()` when needed) to ensure spans are sent before process exit; see the SDK reference for details.

## My AI SDK v7 calls are not traced.

### Problem

I'm on Vercel AI SDK v7 and set up Tracevell, but no AI SDK spans show up in the Tracevell UI.

### Cause

AI SDK v7 changed its telemetry API. The v5 and v6 setup, `Laminar.initialize()` plus `experimental_telemetry: { isEnabled: true, tracer: getTracer() }` on every call, is not the v7 setup. On v7, Tracevell is registered once with the AI SDK's `registerTelemetry`, and calls are traced without any per-call options.

Check your version with `npm ls ai`. If your code imports `registerTelemetry` from `ai` or passes a `telemetry` block to calls, you are on v7.

### Solution

Register Tracevell once at your application's entry point, before the first AI SDK call. In Next.js, put this in `instrumentation.ts`.

```typescript theme={null}
import { registerTelemetry } from 'ai';
import { LaminarAiSdkTelemetry } from '@lmnr-ai/lmnr';

registerTelemetry(new LaminarAiSdkTelemetry());
```

`registerTelemetry` comes from `ai`, not from `@lmnr-ai/lmnr`. If you'd rather import from a single package, `registerAiSdkTelemetry()` from `@lmnr-ai/lmnr` does the same thing.

Then check the following:

* Remove `experimental_telemetry` and `getTracer()` from your call sites. They are the v5/v6 pattern.
* Make sure no call passes `telemetry: { isEnabled: false }`, which turns tracing off for that call.
* Don't call `Laminar.initialize()` separately. `LaminarAiSdkTelemetry` initializes Tracevell itself. Pass init options through `laminarOptions`.
* In short-lived scripts and serverless functions, call `await Laminar.shutdown()` before the process exits.
* To see what the SDK is doing, set `laminarOptions: { logLevel: 'debug' }`. There is no `debug: true` option.

See [Migrating from v5 or v6 to v7](https://tracevell.com/docs/integrations/vercel-ai-sdk.md#migrating-from-v5-or-v6-to-v7) for the full walkthrough.

## My Python auto-instrumentation is not working.

### Problem

I have instrumented my Python code with Tracevell, and I can see the manual spans in the Tracevell UI, but I don't see the auto-instrumented spans.

### Cause

Most likely, you have not installed the right extras that enable the auto-instrumentations.

### Solution

Install the extras for LLM providers you are using and wish to instrument. For instance,

```sh theme={null}
pip install -U lmnr[vertexai]
```

See the relevant integration setup docs for required Python extras: [Integrations](https://tracevell.com/docs/integrations.md).

## I can see the LLM spans and token counts in the Tracevell UI, but prices are 0.

### Problem

LLM spans are instrumented and token counts are shown, but the prices are all at 0.

### Cause

LLM providers do not return pricing information in the API responses, but they do return
token counts.

Tracevell calculates the prices based on the model name and the provider name stored in the Tracevell database.

### Solution

Make sure the model you are using is supported by Tracevell.
If the model/provider pair is not in Tracevell's price table, add your own per-token prices for it in [Custom model costs](https://tracevell.com/docs/platform/custom-model-costs.md). Custom prices match on the span's model and provider names, so copy them exactly as they appear on the span.

## I can see traces and spans in the UI, but they are not showing inputs and/or outputs.

### Problem

I see traces and spans in the Tracevell UI, but they are not showing the inputs and/or outputs. They have attributes and duration, but no inputs or outputs.

### Cause

One possible reason is that we send span inputs and outputs as span attributes, and OpenTelemetry has a limited set of possible attribute types, namely
`string`, `number`, `boolean`, and `array` of each. If the input or output is not one of these types, it will not be sent as an attribute.

To work around this, we serialize the input and output to JSON and send it as a string. We do our best to serialize the inputs and outputs, but it may be
that the serialization fails.

### Solution

Make sure inputs and outputs to your functions or spans are serializable to JSON. If you are using custom objects, make sure their fields are serializible.
In Python, that may mean having to implement `default()` method on some of them, or on the parent object.

## Many of my JS route requests are instrumented in the Tracevell UI, even though I don't want that.

### Problem

I see many traces and spans in the UI, but most of them are just noise, they merely are route requests.

Some examples of noise spans:

* `GET /my/route`
* `POST /my/other/route`
* `dns.lookup`
* `middleware - query`
* `tcp - connect`
* `fs statSync`

### Cause

Even though Tracevell only enables OpenLLMetry-specific instrumentations, i.e. just model calls, some other dependency in your code
may have enabled other instrumentations for Node libraries. In particular, we have seen this issue when such initialization is done
*before* Tracevell's initialization. For example:

```javascript theme={null}
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require(
  '@opentelemetry/auto-instrumentations-node'
);
const { BatchSpanProcessor, SamplingDecision } = require(
  '@opentelemetry/sdk-trace-base'
);

// this can also be not raw OpenTelemetry, but some other observability library
const sdk = new NodeSDK({
  instrumentations: [getNodeAutoInstrumentations()],
  spanProcessors: new BatchSpanProcessor(exporter),
});

const { Tracevell } = require('@lmnr-ai/lmnr');
Laminar.initialize({ projectApiKey: "...", });
```

[Read more](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/metapackages/auto-instrumentations-node#usage-auto-instrumentation) on OpenTelemetry's Node.js auto-instrumentations in OpenTelemetry.

### Solution

The most effective solution is to update lmnr-ai/lmnr to the latest version. Since version 0.6.0 Tracevell has its own Tracer provider,
and it's not affected by other auto-instrumentations.

## SSL Certificate and GRPC Connection Issues for Tracevell

### Problem

Experiencing errors related to SSL certificate verification and GRPC connections in Tracevell. Error messages include `CERTIFICATE_VERIFY_FAILED` and `StatusCode.UNAVAILABLE` when attempting to export traces to a GRPC server.

### Cause

1. **SSL Certificate Verification Error**:
   * The certificate chain may be incomplete or incorrect.
   * The client is not configured to accept the server's certificate.

2. **GRPC Connection Error**:
   * SSL configuration issues prevent establishing a secure connection.
   * The server may actually be unavailable.

### Solution

To resolve these errors, follow these steps:

1. **Import the Certifi Library**:
   * Use the `certifi` library to obtain an updated set of trusted CA certificates.
   * Install using pip: `pip install -U certifi`

2. **Verify the Certifi Path**:
   * After installation, you can verify the path to the `certifi` CA bundle by running:
     ```
     python -m certifi
     ```
   * This command will print the path to the `cacert.pem` file, which you can use for further configurations.

3. **Configure Environment Variables**:
   * **`SSL_CERT_FILE`**: Set the path to the trusted CA certificate file.
     ```
     export SSL_CERT_FILE=$(python -m certifi)
     ```
   * **`GRPC_DEFAULT_SSL_ROOTS_FILE_PATH`**: Configure this variable so that GRPC uses the same CA certificate file.
     ```
     export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=$(python -m certifi)
     ```

## Observe decorator is not working in Python with async generators, i.e. model stream responses

### Problem

`@observe` decorator in Python is not correctly capturing the spans when I use async generators, e.g. when I am streaming responses from a model.

### Cause

Async generators only end when they are exhausted, i.e. when the `async for` loop is done.
Given the nature of Python's async, the chance of the interruptions within stream, and the
way OpenTelemetry's contexts work, there is no way to ensure that `@observe` works in 100% of the cases.

### Solution

One possible workaround is to observe only the outer functions that are calling the async generator,
and handle the final result, not the generator itself.

However, if you want to track partial stream results as well, you can create the spans manually.
Refer to [manually creating spans](https://tracevell.com/docs/tracing/structure/manual-span-creation.md) for more information.

## ERR\_BUFFER\_OUT\_OF\_BOUNDS in Node.js

### Problem

I see the following error in my Node.js application:

```sh theme={null}
node:internal/buffer:1066
      throw new ERR_BUFFER_OUT_OF_BOUNDS('length');
      ^
```

**full error trace**

```sh theme={null}
RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: "length" is outside of buffer bounds
    at proto.utf8Write (node:internal/buffer:1066:13)
    at Op.writeStringBuffer [as fn] (/Users/sebastianscheibe/Code/XXX/node_modules/protobufjs/src/writer_buffer.js:61:13)
    at BufferWriter.finish (/Users/sebastianscheibe/Code/XXX/node_modules/protobufjs/src/writer.js:453:14)
    at /Users/sebastianscheibe/Code/XXX/node_modules/@grpc/proto-loader/build/src/index.js:177:109
    at Array.map (<anonymous>)
    at createPackageDefinition (/Users/sebastianscheibe/Code/XXX/node_modules/@grpc/proto-loader/build/src/index.js:177:39)
    at Object.fromJSON (/Users/sebastianscheibe/Code/XXX/node_modules/@grpc/proto-loader/build/src/index.js:230:12)
    at GrpcClient.loadProtoJSON (/Users/sebastianscheibe/Code/XXX/node_modules/google-gax/build/src/grpc.js:228:51)
    at new ImageAnnotatorClient (/Users/sebastianscheibe/Code/XXX/node_modules/@google-cloud/vision/build/src/v1/image_annotator_client.js:148:38)
    at Object.<anonymous> (/Users/sebastianscheibe/Code/XXX/test.js:5:16) {
  code: 'ERR_BUFFER_OUT_OF_BOUNDS'
}

Node.js v22.7.0
```

### Cause

This is most likely due to a bug in NodeJS versions 22.6 and 22.7 with
handling utf-8 buffers. See more in the [Node.js issue](https://github.com/nodejs/node/issues/54518#issuecomment-2307035124).

### Solution

Upgrade to Node.js version 22.8.0 or higher. You can also downgrade to 22.5.

## My span exports are rejected as too large (gRPC `OUT_OF_RANGE`, HTTP `413 Payload Too Large`).

### Problem

Spans stop arriving, and the exporter logs a rejection. Which error you see depends on the transport:

* **gRPC** (the default): status `OUT_OF_RANGE`, with the message `Error, decoded message length too large: found <N> bytes, the limit is: <M> bytes`.
* **HTTP** (when you set `forceHttp` / `force_http=True`): `413 Payload Too Large`, with the body `Payload too large: the request body exceeds the server's HTTP payload limit. Send smaller batches, or raise HTTP_PAYLOAD_LIMIT if you are self-hosting.`

A related symptom is `DEADLINE_EXCEEDED` on a request that carries only a few dozen spans: the batch is not rejected outright, but it is big enough to outlast the export timeout.

### Cause

The batch span processor sends a batch when it reaches a span count or when a scheduled delay elapses. Neither trigger looks at how many bytes the batch holds, and agent spans carry whole prompts, completions, and tool payloads as string attributes. A handful of large spans is enough to build a request over the ingest limit.

### Solution

Enable the byte-size flush trigger, which exports the buffer when the next span would push it past a limit:

> [tab] TypeScript
>
> ```typescript theme={null}
> Laminar.initialize({
>   projectApiKey: process.env.LMNR_PROJECT_API_KEY,
>   flushBySize: true,
>   maxExportBatchSizeBytes: 8 * 1024 * 1024,
> });
> ```

> [tab] Python
>
> ```python theme={null}
> Laminar.initialize(
>     flush_by_size=True,
>     max_export_batch_size_bytes=8 * 1024 * 1024,
> )
> ```

Lowering `maxExportBatchSize` / `max_export_batch_size` also helps and is worth trying first, but it bounds the span count rather than the payload. See [Batching](https://tracevell.com/docs/sdk/batching.md) for how the two interact and what the byte limit measures.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.