> ## Documentation Index
> Fetch the complete documentation index at: https://tracevell.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Helm Configuration Reference for Self-hosted Tracevell

Every feature of a self-hosted Tracevell is configured through values in `tracevell.yaml`, the overrides file you pass to `helm upgrade -i tracevell tracevell/tracevell -f tracevell.yaml`. This page covers the settings most deployments need: AI features, Signals, Slack, OAuth login, storage, TLS, PII redaction, secrets, and image pinning. For the full value list, see [`values.yaml`](https://github.com/lmnr-ai/lmnr-helm/blob/main/charts/tracevell/values.yaml) in the chart repo.

Apply any change by re-running the install command:

```bash theme={null}
helm upgrade -i tracevell tracevell/tracevell -f tracevell.yaml
```

## AI features and LLM provider

Chat-with-trace and SQL-with-AI run through one unified LLM provider configuration. Set the provider on all three application pods (they default to `gemini`), and supply the key:

* `LLM_PROVIDER`: `gemini` (default), `openai`, or `bedrock`. Set via `frontend.env.llmProvider`, `appServer.env.llmProvider`, and `appServerConsumer.env.llmProvider`.
* `LLM_API_KEY`: the key for `gemini` or `openai`. Set via `secrets.data`.
* `LLM_BASE_URL`: optional, for OpenAI-compatible gateways (LiteLLM, OpenRouter, vLLM) or custom Gemini endpoints. Set via `*.env.llmBaseUrl`.
* `LLM_MODEL_SMALL` / `LLM_MODEL_MEDIUM` / `LLM_MODEL_LARGE`: optional per-tier model overrides. Per-provider defaults apply when unset. For Bedrock, these are Inference Profile IDs.

AI features in the frontend activate as soon as `LLM_API_KEY` (or AWS credentials for Bedrock) is populated.

> [!NOTE]
>
> [Signals](https://tracevell.com/docs/signals/introduction.md) and the [playground](https://tracevell.com/docs/playground.md) do not use these values. Each Signal runs on an [LLM profile](https://tracevell.com/docs/platform/llm-profiles.md): a workspace-level entry holding a provider, its credentials, and the models you allow. Set `LLM_API_KEY` for the features above, and add profiles in workspace settings for Signals and the playground.

### Gemini (default)

Just supply the key:

```yaml theme={null}
secrets:
  data:
    LLM_API_KEY: "your-gemini-key"
```

### OpenAI or OpenAI-compatible gateway

```yaml theme={null}
frontend:
  env:
    llmProvider: "openai"
    # Optional gateway (LiteLLM, OpenRouter, vLLM):
    # llmBaseUrl: "http://my-gateway:4000"

appServer:
  env:
    llmProvider: "openai"

appServerConsumer:
  env:
    llmProvider: "openai"

secrets:
  data:
    LLM_API_KEY: "your-openai-or-gateway-key"
```

### AWS Bedrock

Bedrock uses AWS credentials from `secrets.data` instead of `LLM_API_KEY`:

```yaml theme={null}
frontend:
  env:
    llmProvider: "bedrock"

appServer:
  env:
    llmProvider: "bedrock"

appServerConsumer:
  env:
    llmProvider: "bedrock"

secrets:
  data:
    AWS_ACCESS_KEY_ID: "your-aws-access-key-id"
    AWS_SECRET_ACCESS_KEY: "your-secret-access-key"
    AWS_REGION: "us-east-1"
```

### Model overrides

Pin specific models per size tier (for Bedrock, values are Inference Profile IDs):

```yaml theme={null}
appServerConsumer:
  env:
    llmModelSmall: "gemini-3-flash-preview"
    llmModelMedium: "gemini-3-flash-preview"
    llmModelLarge: "gemini-3-pro-preview"
```

## Signals

[Signals](https://tracevell.com/docs/signals/introduction.md) extract structured events from your traces. On a self-hosted deployment, each Signal runs on an [LLM profile](https://tracevell.com/docs/platform/llm-profiles.md) that a workspace member creates in the app: the provider, its credentials, and the models the Signal may use live in the profile, not in `tracevell.yaml`. Signals are **enabled by default** in the frontend, and no `LLM_API_KEY` is required to turn them on.

Signals created before profiles existed keep running on the `LLM_PROVIDER` configuration above until you point them at a profile. Signals that use a profile ignore `LLM_PROVIDER`, `LLM_BASE_URL`, and the `LLM_MODEL_*` overrides entirely.

To disable Signals:

```yaml theme={null}
frontend:
  env:
    signalsEnabled: "false"
```

To disable chat-with-trace independently:

```yaml theme={null}
frontend:
  env:
    agentChatEnabled: "false"
```

> [!NOTE]
>
> Signal extraction runs in the `app-server-consumer` pod. For a Signal without a profile, that pod's `llmProvider` (and any model overrides) governs how the Signal is computed, so keep the provider consistent across `frontend`, `appServer`, and `appServerConsumer` unless you have a reason to split them. Profile credentials are encrypted by the frontend and decrypted by the app server, both with `AEAD_SECRET_KEY`, so that value has to match across pods (the chart sets one key for all of them).

## Slack notifications

Tracevell posts [alerts](https://tracevell.com/docs/signals/alerts.md) and notifications to Slack through a broker run by Tracevell Cloud: your instance uses Tracevell Cloud's official Slack app, so you do not register a Slack app of your own. Tracevell Cloud runs both legs of the OAuth flow on your behalf; the bot token is returned to your instance server-to-server and encrypted at rest. Your instance authenticates to the broker with an **enterprise license key**.

Set the license key in `tracevell.yaml`:

```yaml theme={null}
secrets:
  data:
    LMNR_LICENSE_KEY: "your-enterprise-license-key"
```

When the license key is set, the "Connect Slack" button in workspace settings uses the brokered flow. `SLACK_BROKER_URL` defaults to Tracevell Cloud's origin (`https://tracevell.com`), so you normally only set the license key. To get a license key, contact [jacob@tracevell.com](mailto:jacob@tracevell.com).

Slack bot tokens are encrypted at rest with `SLACK_ENCRYPTION_KEY`. You do not normally set this: when left empty, the chart defaults it to your `AEAD_SECRET_KEY` so there is one encryption key to manage. Set it explicitly in `secrets.data` only if you want Slack tokens encrypted under a separate key.

## OAuth login

Tracevell supports OAuth login with GitHub, Google, Azure AD, Okta, and Keycloak. Add provider credentials to `secrets.data` and set the frontend URLs to your real domain. Omit a provider's credentials to disable it.

> [!WARNING]
>
> Callback / redirect URLs must match your `nextauthUrl` exactly, or the OAuth flow fails after the provider redirects back.

```yaml theme={null}
secrets:
  data:
    NEXTAUTH_SECRET: "your-nextauth-secret"

    # GitHub: callback https://app.yourdomain.com/api/auth/callback/github
    AUTH_GITHUB_ID: "your-github-client-id"
    AUTH_GITHUB_SECRET: "your-github-client-secret"

    # Google: redirect https://app.yourdomain.com/api/auth/callback/google
    AUTH_GOOGLE_ID: "your-google-client-id"
    AUTH_GOOGLE_SECRET: "your-google-client-secret"

    # Azure AD: redirect https://app.yourdomain.com/api/auth/callback/azure-ad
    AUTH_AZURE_AD_CLIENT_ID: "your-azure-client-id"
    AUTH_AZURE_AD_CLIENT_SECRET: "your-azure-client-secret"
    AUTH_AZURE_AD_TENANT_ID: "your-azure-tenant-id"

    # Okta OIDC
    AUTH_OKTA_CLIENT_ID: "your-okta-client-id"
    AUTH_OKTA_CLIENT_SECRET: "your-okta-client-secret"
    AUTH_OKTA_ISSUER: "https://your-okta-domain.com/oauth2/default"

    # Keycloak OIDC
    AUTH_KEYCLOAK_ID: "your-keycloak-id"
    AUTH_KEYCLOAK_SECRET: "your-keycloak-secret"
    AUTH_KEYCLOAK_ISSUER: "https://your-keycloak-domain.com/realms/My_Realm"

frontend:
  env:
    nextauthUrl: "https://app.yourdomain.com"
    nextPublicUrl: "https://app.yourdomain.com"
```

If your provider credentials already live in a Kubernetes Secret (created by the Keycloak operator, sealed-secrets, or external-secrets-operator), reference them with [`extraEnv`](#secrets-management) instead of inlining them.

## Secrets management

The chart reads secrets from three backends:

1. **Kubernetes Secrets** (default): values in `secrets.data`.
2. **AWS Secrets Manager**: for EKS with IRSA, via the Secrets Store CSI Driver.
3. **HashiCorp Vault**: for on-premises or multi-cloud.

To pull a secret from AWS Secrets Manager, list its keys under `secrets.awsSecretsManager` and leave placeholders in `secrets.data`:

```yaml theme={null}
secrets:
  awsSecretsManager:
    enabled: true
    region: "us-east-1"
    serviceAccount:
      create: false
      name: "lmnr-secrets-sa"
    clusterName: "production"
    secretKeys:
      - NEXTAUTH_SECRET
      - POSTGRES_PASSWORD
  data:
    NEXTAUTH_SECRET: ""
    POSTGRES_PASSWORD: ""
    AWS_REGION: "us-east-1"
```

You can mix backends (auth tokens from AWS, database passwords from Vault, the rest from `secrets.data`). See the chart's [Secrets Management guide](https://github.com/lmnr-ai/lmnr-helm/blob/main/CONFIGURATION.md#secrets-management) for the Vault and mixed-source setups.

### Referencing existing Secrets with extraEnv

`extraEnv` injects environment variables into `frontend`, `appServer`, or `appServerConsumer`, supporting plain values, `secretKeyRef`, `configMapKeyRef`, and `fieldRef`. Entries override matching keys from `secrets.data`, so you can selectively swap individual values without restructuring the secrets config:

```yaml theme={null}
frontend:
  extraEnv:
    - name: AUTH_KEYCLOAK_ID
      valueFrom:
        secretKeyRef:
          name: keycloak-realm
          key: client-id
    - name: AUTH_KEYCLOAK_SECRET
      valueFrom:
        secretKeyRef:
          name: keycloak-realm
          key: client-secret
    - name: AUTH_KEYCLOAK_ISSUER
      value: "https://keycloak.example.com/realms/my-realm"
```

## Container images

The two Tracevell containers (frontend and app server) are pulled from public `ghcr.io/lmnr-ai/*-ee` images, so most installs never touch this section. For production, pin a specific tag instead of `latest` so a pod restart does not pick up a newer build. The two are released together: keep their tags in sync.

```yaml theme={null}
images:
  pullPolicy: IfNotPresent
  frontend:
    tag: "0.1.628"
  appServer:
    tag: "0.1.628"
```

> [!NOTE]
>
> Chart `0.2.0` and newer require an app-server tag of `0.1.628` or newer and will fail at render time if you pin an older one. Earlier charts shipped a separate query-engine container; it has since been folded into the app-server image, so there is no longer a `queryEngine` image to pin.

Available tags are listed under the [Packages tab on GitHub](https://github.com/orgs/lmnr-ai/packages). To mirror images into a private registry, retag and push each image, point `images.repository` at the mirror, and attach `imagePullSecrets` to the default ServiceAccount (the chart does not template this; see the [chart reference](https://github.com/lmnr-ai/lmnr-helm/blob/main/CONFIGURATION.md#container-images)).

## Storage

ClickHouse and Quickwit hold the bulk of Tracevell's data, and both can be backed by S3.

### ClickHouse on S3

```yaml theme={null}
clickhouse:
  s3:
    enabled: true
    endpoint: "https://my-bucket.s3.us-east-1.amazonaws.com/"
    region: "us-east-1"
    useEnvironmentCredentials: true  # use the node IAM role on EKS
    cache:
      enabled: true
      maxSize: "50Gi"
```

On GCP, ClickHouse's S3 backend needs **HMAC credentials** (the GKE metadata server returns OAuth2 tokens that GCS's S3 API rejects). Point the endpoint at `https://storage.googleapis.com/YOUR_BUCKET/` and supply HMAC keys. The chart is designed for one HMAC key pair to back both ClickHouse and Quickwit. See the chart's [ClickHouse S3 guide](https://github.com/lmnr-ai/lmnr-helm/blob/main/CONFIGURATION.md#clickhouse-s3-storage).

### Quickwit (full-text search)

Quickwit powers full-text [search](https://tracevell.com/docs/platform/search.md) over spans. `quickwit.s3.defaultIndexRootUri` is the master switch for the whole Quickwit stack: leave it empty and the Quickwit workloads are skipped and search degrades gracefully (the rest of the platform is unaffected). Set it to a bucket you own to enable search:

```yaml theme={null}
quickwit:
  s3:
    defaultIndexRootUri: "s3://my-bucket/indexes"
    region: "us-east-1"
```

When `global.cloudProvider: gcp`, the chart auto-fills the GCS flavor and endpoint for you; supply the same HMAC keys used for ClickHouse via `quickwit.extraEnv`.

### Persistent volumes

The chart creates an EBS storage class scoped to the availability zones you list. Pods with persistent volumes can only schedule on nodes in the same zone, so make sure your nodes run in these zones:

```yaml theme={null}
storage:
  storageClass:
    type: "gp3"
    reclaimPolicy: "Retain"
    zones:
      - "us-east-1a"
      - "us-east-1b"
      - "us-east-1c"
```

Each stateful service (`postgres`, `clickhouse`, `rabbitmq`) can override its storage class and size independently under `<service>.persistence`.

## TLS and ingress

Tracevell exposes two public endpoints: the frontend (web UI) and the app server (HTTP trace ingestion on `443`, gRPC on `8443`). TLS has three paths.

### cert-manager (automatic, recommended for GCP and generic clusters)

cert-manager provisions and renews free Let's Encrypt certificates. The hostname must be publicly DNS-resolvable so Let's Encrypt can complete the HTTP-01 challenge:

```yaml theme={null}
frontend:
  ingress:
    hostname: "app.yourdomain.com"
    className: "traefik"
    tls:
      enabled: true
      clusterIssuer: "letsencrypt"
      secretName: "tracevell-frontend-tls"
  env:
    nextauthUrl: "https://app.yourdomain.com"
    nextPublicUrl: "https://app.yourdomain.com"
```

### AWS ACM (recommended for AWS)

On AWS, the NLB and ALB terminate TLS via an ACM certificate ARN annotation, with no in-cluster cert management:

```yaml theme={null}
frontend:
  ingress:
    hostname: "app.yourdomain.com"
    annotations:
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
      alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:region:account:certificate/xxx"
      alb.ingress.kubernetes.io/ssl-redirect: '443'
  env:
    nextauthUrl: "https://app.yourdomain.com"
    nextPublicUrl: "https://app.yourdomain.com"
```

### Pre-existing certificate

Import a PEM certificate as a Kubernetes TLS secret and reference it with an empty `clusterIssuer`. See the chart's [NETWORKING guide](https://github.com/lmnr-ai/lmnr-helm/blob/main/NETWORKING.md) for the full app-server TLS and DNS options, including external-dns automation.

> [!NOTE]
>
> On AWS, the app server's NLB already exposes HTTP (`443`) and gRPC (`8443`) externally; you usually do not need a separate app-server Ingress. On GCP the load balancer is pure TCP passthrough and cannot terminate TLS: front the app server with Traefik (or another ingress controller) to get TLS on both ports.

## PII redaction

The [PII redactor](https://tracevell.com/docs/platform/pii-redaction.md) is an optional CPU-only gRPC service that detects personally identifiable information in spans at ingest. It backs both redaction modes: **On** replaces PII before the span is persisted, and **Dual** stores the original text once and redacts it per reader, so owners and admins see the original while members, API keys, and CLI sessions see placeholders. The service is **disabled by default** and has no database or secret dependencies (the detection model is baked into the image). When enabled, the chart injects `PII_REDACTOR_URL` into the `app-server` and `app-server-consumer` pods, activating detection during span processing; the mode is then chosen per project under **Project Settings → Security**.

```yaml theme={null}
piiRedactor:
  enabled: true
```

PII detection is CPU-intensive ONNX inference and the model weighs about 5.5 GB in memory, so pin it to a dedicated compute-optimized node and size it generously. For more throughput, add replicas rather than a bigger node:

```yaml theme={null}
piiRedactor:
  enabled: true
  nodeSelector:
    node.kubernetes.io/instance-type: c8i.2xlarge
  resources:
    requests:
      cpu: "2"
      memory: "8Gi"
    limits:
      cpu: "8"
      memory: "16Gi"
```

## Sharing a Postgres database

By default Tracevell puts its tables in the `public` schema. To share a Postgres database with another service, point it at a dedicated schema with `global.postgresSchema`. The single value is dispatched to all three application pods, so they resolve the same `search_path`. Set it before the first deploy:

```yaml theme={null}
global:
  postgresSchema: "tracevell"
```

On first boot the frontend runs `CREATE SCHEMA IF NOT EXISTS` for a non-`public` schema. If the schema is pre-provisioned or the DB role lacks `CREATE`, disable that step with `frontend.env.postgresCreateSchema: "false"`.

## What's next

> [card] [Kubernetes (Helm)](https://tracevell.com/docs/self-hosting/kubernetes.md)
>
> Back to the install walkthrough.

> [card] [Signals](https://tracevell.com/docs/signals/introduction.md)
>
> Outcome and failure extraction over your traces.

> [card] [PII redaction](https://tracevell.com/docs/platform/pii-redaction.md)
>
> How redaction renders in the UI and SQL.

> [card] [Start tracing](https://tracevell.com/docs/tracing/introduction.md)
>
> Send your first traces to your instance.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.