Skip to content

Monitoring

This page covers how to monitor a self-hosted XTM One instance in production: health probes, metrics, request tracing, and logs. Filigran SaaS instances are monitored by Filigran.

The environment variables referenced here are documented in full in the Configuration section — this page explains what each signal tells you and how to act on it.

Health probes

The platform exposes two unauthenticated endpoints for orchestrators and uptime checks:

Endpoint Use for Returns 200 when
GET /api/health Liveness probe The HTTP server is running
GET /api/health/ready Readiness probe Database, Redis, and object storage are reachable and migrations are complete

Use /api/health as a liveness probe (restart the container if it fails) and /api/health/ready as a readiness probe (stop routing traffic until it passes). The Docker Compose stack already wires /api/health/ready into its health checks so dependent services wait for full startup.

Kubernetes probes

Point livenessProbe at /api/health and readinessProbe at /api/health/ready. Readiness returns non-200 until the database, Redis, and object storage are reachable and migrations have completed, so traffic is held back until the platform is fully started.

Metrics

XTM One exposes Prometheus metrics on a token-protected /metrics endpoint.

  1. Set PROMETHEUS_ENABLED=true.
  2. Set PROMETHEUS_BEARER_TOKEN to a secret of at least 16 characters.
  3. Scrape /metrics with an Authorization: Bearer <token> header.

The token is mandatory

When PROMETHEUS_ENABLED=true, the platform refuses to expose /metrics unless a valid PROMETHEUS_BEARER_TOKEN is set. This prevents an unauthenticated metrics endpoint from leaking operational data.

The endpoint exposes usage metrics in the Prometheus text format. The main application metric is xtmone_quota_usage, a gauge of quota consumption labelled by user_id, usage_type, model, period (daily, monthly, yearly), and platform_id. The standard per-process metrics from the Prometheus client library (Python runtime and garbage-collection counters) are included as well.

Scrape this endpoint to chart quota usage per user and per period, and combine it with the health probes to build a basic availability dashboard.

Request tracing

XTM One records a trace for every LLM interaction so you can inspect what an agent did, which tools it called, how many tokens it used, and where time was spent.

Native trace store

By default, traces are stored inside the platform's own database and are browsable from the application under Traces. The store backend is selected with TRACING_STORE_PROVIDER:

  • postgres (default) — traces live in the bundled PostgreSQL. No extra service required.
  • clickhouse — traces are read from a dedicated ClickHouse instance for higher-volume analytics. Provide the CLICKHOUSE_* connection settings.

Trace payload storage

Trace payloads — the prompt, the conversation, the model's reasoning, tool arguments and results — are stored whole. They are audit material, and the trace store holds the only copy, so nothing is clipped on the way in.

Large payloads are moved out of the database rather than truncated. Above TRACE_PAYLOAD_OFFLOAD_THRESHOLD_BYTES (default 1 MiB), the payload is written to object storage and the trace row keeps a pointer with the exact size, a SHA-256 digest and a short excerpt. In the Traces UI those payloads show a Load full payload action that fetches the complete value on demand; the panes show a bounded preview until you ask for it, so opening a trace with megabyte-sized payloads stays fast.

Images and PDFs inside a payload are stored the same way, one object each: the payload keeps a reference with the blob's content type, byte size and SHA-256, and the span inspector lists them as attachments that render the actual image on demand. This is what makes the screenshot an agent received — or the chart it produced — recoverable from its trace; previously media was replaced by an [image] placeholder, and since the trace store holds the only copy, the picture was gone for good.

Variable What it controls
TRACE_PAYLOAD_OFFLOAD_THRESHOLD_BYTES Size above which a payload goes to object storage (default 1 MiB). 0 disables offloading and always stores inline.
TRACE_PAYLOAD_HARD_MAX_BYTES Inline fallback ceiling used only when object storage is unreachable (default 32 MiB). Below it the payload is stored inline anyway; above it the trace records that the payload could not be stored, so the gap is visible instead of silent.
TRACE_MEDIA_OFFLOAD_ENABLED Whether media blobs are stored (default true). Set to false to replace them with an [image] placeholder instead, as earlier versions did.

Offloading requires the same object storage as file attachments (the S3_* settings). Everything one trace owns — offloaded payloads and media alike — lives under a traces/<trace_id>/ prefix. Plan disk for both volumes accordingly. Trace retention bounds how long payloads and their objects are kept — the retention purge deletes the objects along with the rows, under both store backends.

Langfuse export

When Langfuse is configured, the payloads exported to it still have media replaced by placeholders and long strings truncated, regardless of TRACE_MEDIA_OFFLOAD_ENABLED. Langfuse ingestion has its own size limits, and its SDK would otherwise upload every blob to its own separate media storage. Only the export is reduced; the durable trace keeps everything.

Trace retention

How long traces are kept is a platform setting, not an environment variable: set it under Settings → Observability → Trace retention (days) (1–365, 90 by default). It applies to both store backends, in the same maintenance cycle, and each cycle deletes the offloaded payload objects belonging to the traces it expires.

  • PostgreSQL — expired traces are deleted in batches roughly hourly. A day's aggregated statistics (llm_daily_stats) are built before its raw traces are deleted and outlive them, so dashboards keep reporting on days whose traces are gone.
  • ClickHouse — the same cycle sweeps the expired window: it reads the payload pointers off the expiring rows, deletes the rows, then deletes the objects they pointed at. The tables also carry a TTL as a backstop, re-declared whenever the retention setting changes and re-checked on each cycle, so retention still applies to tables that already exist and to a platform whose maintenance loop is not running.

Why ClickHouse keeps a few extra days

The ClickHouse TTL is deliberately set a week beyond the retention window rather than exactly on it. Offloaded payload objects can only be found through the rows that point at them, so the rows have to outlive the object purge — a TTL on the exact boundary would let ClickHouse drop rows before the sweep had read their pointers, orphaning objects in the bucket. In normal operation the sweep enforces the exact window and the TTL never fires. If the maintenance loop stays down for longer than that margin, the TTL still expires the rows (retention is never silently extended) but their objects are left behind; that is a storage cost, and the sweep logs when it cannot reach object storage.

Langfuse (optional)

You can additionally forward traces to Langfuse for external observability. Set LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, and LANGFUSE_SECRET_KEY. LANGFUSE_HOST defaults to Langfuse Cloud; point it at your self-hosted Langfuse to keep trace data on your own infrastructure.

Langfuse is complementary to the native store — enabling it does not disable native traces. Payloads sent to Langfuse are bounded per field, because its ingestion rejects oversized events; the native store keeps the unabridged copy.

Logs

The platform emits structured logs to stdout, which your container runtime or log aggregator collects.

Variable What it controls
LOG_LEVEL Verbosity (debug, info, warning, error, critical)
LOG_FORMAT json (structured, for aggregators) or console (human-readable)

One structured http_request line is emitted per HTTP request on the app.access logger. Successful requests to /api/health and /metrics are logged at debug so routine probes and scrapes do not flood your logs; client errors are logged at warning and server errors at error.

In production, keep LOG_FORMAT=json and ship logs to a central aggregator (Loki, Elasticsearch, CloudWatch, etc.) so you can search and alert across replicas.

Sandbox call logs

Every OpenSandbox-routed call (code interpreter, custom JS tool, custom Python tool) emits one additional structured line, sandbox_call, carrying CPU/RAM usage, execution time, egress policy, and outcome. It ships through the same stdout path as everything else above — no extra infrastructure required. The full field reference (names, types, units, when each is present) lives in Configuration § Sandbox call log fields; this section covers how to query it once it's in Grafana.

Example line (LOG_FORMAT=json):

{"event": "sandbox_call", "usage_type": "code_interpreter", "outcome": "success", "user_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "cpu_used_percentage": 12.5, "cpu_count": 1.0, "memory_used_in_mib": 64.2, "memory_total_in_mib": 128.0, "tte_ms": 842, "egress_default_action": "deny", "egress_allow_list": [], "level": "info", "logger": "app.services.sandbox_call_logging", "timestamp": "2026-07-31T12:00:00.000000Z"}

With logs shipped into Loki, these LogQL queries build directly on the fields above — swap {container="platform"} for whatever stream label your Promtail/Grafana Agent config actually attaches (container name, job, Kubernetes pod label, etc.):

  • All sandbox calls (base query for a Logs panel):
{container="platform"} | json | event="sandbox_call"
  • Failure rate by usage_type (stat/alert panel — ratio of failing to total calls per interval):
sum by (usage_type) (count_over_time({container="platform"} | json | event="sandbox_call" | outcome="failure" [$__interval]))
/
sum by (usage_type) (count_over_time({container="platform"} | json | event="sandbox_call" [$__interval]))
  • Failure breakdown by reason (table/pie panel — sandbox_ready_timeout, sandbox_create_failed, sandbox_create_error, nonzero_exit, execution_error):
sum by (failure_reason) (count_over_time({container="platform"} | json | event="sandbox_call" | outcome="failure" [$__interval]))
  • Average execution time (TTE) per usage_type (time series panel):
avg_over_time({container="platform"} | json | event="sandbox_call" | outcome="success" | unwrap tte_ms [5m]) by (usage_type)
  • CPU / memory usage trend (swap cpu_used_percentage for memory_used_in_mib or memory_total_in_mib):
avg_over_time({container="platform"} | json | event="sandbox_call" | unwrap cpu_used_percentage [5m]) by (usage_type)
  • Egress policy audit (confirm what's actually live, per call):
{container="platform"} | json | event="sandbox_call" | line_format "{{.usage_type}}: {{.egress_default_action}} allow={{.egress_allow_list}}"

CPU/RAM/TTE fields are simply absent (not null) on records where the sandbox never reached completion — sandbox_ready_timeout, sandbox_create_failed, and sandbox_create_error records have none of them. unwrap queries skip lines missing the unwrapped field rather than erroring, so a TTE/CPU/RAM panel naturally reflects only the calls that actually ran, without extra filtering.

What to watch

At a minimum, alert on:

  • /api/health/ready failing for more than a few minutes (platform not serving)
  • A rising rate of 5xx responses in the access logs
  • Worker containers repeatedly restarting (background jobs and assignment runs stall)
  • PostgreSQL connection saturation — see the connection-budget formula in Configuration
  • Disk usage on the PostgreSQL and object-storage volumes, which grow with knowledge bases and file attachments
  • A rising sandbox_call failure rate or an unfamiliar failure_reason for OpenSandbox-routed code interpreter/custom tool calls — see Sandbox call logs above

Under construction

Recommended alert thresholds and a reference set of Prometheus alert rules for self-hosted deployments are being prepared and will be added here.

Next step

If a signal points to a problem, continue with Troubleshooting for common issues and how to diagnose them.