Monitoring

This page covers how to monitor a self-hosted XTM One instance in production: health probes, metrics, request tracing, and logs. Filigran SaaS instances are monitored by Filigran.

The environment variables referenced here are documented in full in the Configuration section — this page explains what each signal tells you and how to act on it.

Health probes

The platform exposes two unauthenticated endpoints for orchestrators and uptime checks:

Endpoint	Use for	Returns 200 when
`GET /api/health`	Liveness probe	The HTTP server is running
`GET /api/health/ready`	Readiness probe	Database, Redis, and object storage are reachable and migrations are complete

Use /api/health as a liveness probe (restart the container if it fails) and /api/health/ready as a readiness probe (stop routing traffic until it passes). The Docker Compose stack already wires /api/health/ready into its health checks so dependent services wait for full startup.

Kubernetes probes

Point livenessProbe at /api/health and readinessProbe at /api/health/ready. Readiness returns non-200 until the database, Redis, and object storage are reachable and migrations have completed, so traffic is held back until the platform is fully started.

Each web worker process (WEB_WORKERS) runs its own startup, and a probe is answered by whichever worker takes it. A worker can therefore be starting while its pod already receives traffic: the replacement uvicorn starts for a worker recycled after MAX_REQUESTS, or a second worker still starting when the first one made the pod ready. Such a worker holds the requests it receives until its startup completes, for up to 8 seconds, and only then answers 503 with Retry-After. It also skips the migrations and the seeding already applied since the container started, so it is serving within about a second. A seeding step the startup logged as retried "on next startup" (the built-in brand assets while the object store was unreachable, for instance) is therefore retried when the container next starts, not by a replacement worker. A worker whose startup failed answers 503 at once, and its /api/health/ready keeps reporting the error.

Metrics

XTM One exposes Prometheus metrics on a token-protected /metrics endpoint.

Set PROMETHEUS_ENABLED=true.
Set PROMETHEUS_BEARER_TOKEN to a secret of at least 16 characters.
Scrape /metrics with an Authorization: Bearer <token> header.

The token is mandatory

When PROMETHEUS_ENABLED=true, the platform refuses to expose /metrics unless a valid PROMETHEUS_BEARER_TOKEN is set. This prevents an unauthenticated metrics endpoint from leaking operational data.

The endpoint exposes usage metrics in the Prometheus text format. The main application metric is xtmone_quota_usage, a gauge of quota consumption labelled by user_id, usage_type, model, period (daily, monthly, yearly), and platform_id. The standard per-process metrics from the Prometheus client library (Python runtime and garbage-collection counters) are included as well.

Scrape this endpoint to chart quota usage per user and per period, and combine it with the health probes to build a basic availability dashboard.

Sandbox workspace metrics

The code interpreter's sandbox workspace reports:

Metric	Type	Labels	What it counts
`xtmone_sandbox_workspaces`	gauge	`measure` (`count`, `bytes`, `files`, `kernel_restarts`), `platform_id`	Every workspace of the platform, read from the database at scrape time.
`xtmone_sandbox_workspace_calls_total`	counter	`mode` (`stateless`, `session`), `workspace` (`mounted`, `unmounted`, `unavailable`)	`code_interpreter` calls.
`xtmone_sandbox_workspace_bytes_total`	counter	`direction` (`read`, `written`)	Bytes moved into the sandbox and saved from it.
`xtmone_sandbox_workspace_operations_total`	counter	`operation` (`put`, `get`, `list`, `remove`, `ingest`, `materialize_repository`, `install`, `gc_delete`), `outcome`	Workspace operations; `gc_delete` carries the sweep's reason (`idle`, `conversation_archived`, `conversation_deleted`, `run_deleted`, `orphan_objects`).
`xtmone_sandbox_workspace_quota_refusals_total`	counter	`operation`	Change sets refused for the quota or the file limit.
`xtmone_sandbox_kernel_starts_total`	counter	`reason` (`new`, `restarted`, `reset`)	`session=true` kernels started; a rising `restarted` means kernels are lost between calls (idle timeout, timeouts, calls reaching another replica).
`xtmone_turn_resume_results_total`	counter	`kind` (`replayed`, `truncated`)	Tool results an interrupted chat turn handed to the turn that resumed it.
`xtmone_sandbox_workspace_sync_seconds`	histogram	`phase` (`materialize`, `scan`, `upload`, `commit`)	Time moving a workspace in and out of the sandbox.

The counters are per backend process: a scrape reads the process that answers it. The audit log records each file put into a workspace (workspace_put), each repository checkout (materialize_repository) and each workspace the sweep deletes, under the entity type sandbox_workspace. The workspace of one conversation (size, last clean-up, kernel restarts, bytes moved, files) is served to its owner by GET /api/v1/chat/conversations/{id}/sandbox-workspace.

Request tracing

XTM One records a trace for every LLM interaction so you can inspect what an agent did, which tools it called, how many tokens it used, and where time was spent.

A run's trace counts every token of every model call it made, including a run whose calls different providers answered (a provider failover, or a budget downgrade to another provider's model). When different models answered its calls, or their providers report usage in different formats, the run's cost is the sum of its calls' costs, as each call's own trace entry shows them; a run one model answered is priced at that model's rates.

Native trace store

By default, traces are stored inside the platform's own database and are browsable from the application under Traces. The store backend is selected with TRACING_STORE_PROVIDER:

postgres (default) — traces live in the bundled PostgreSQL. No extra service required.
clickhouse — traces are read from a dedicated ClickHouse instance for higher-volume analytics. Provide the CLICKHOUSE_* connection settings.

To investigate an assignment, open it and select View Traces: the Traces page opens scoped to the traces of every run of that assignment, and the other filters (status, failure cause, dates, models) narrow it further. View Traces on a run's page scopes the list to that run alone. The scope shows as a chip above the filters; remove it, or select Clear filters, to see every trace again. Both shortcuts are shown to administrators, who are the ones with access to the Traces page.

What a trace keeps

Two switches under Settings → Observability decide what the native trace store keeps, under both store backends:

`Enable local trace storage`	`Store full payloads`	What is stored
On (default)	On (default)	Every trace with its payloads, as described below
On	Off	Every trace without what was said: the prompts, the conversation, the answers, the model's reasoning, tool arguments and tool results are dropped, and an error message keeps its first 200 characters (a provider error can quote the request it refused). Timings, token counts, cost, model, trace and tool names, status, failure cause and metadata are kept, so the `Traces` list, the analytics and the cost reports keep working
Off	On or off	Nothing: no trace, no span and no user feedback score is written. An administrator's own score on a trace already stored (`POST /api/v1/traces/{id}/scores`) is still recorded

A change applies within a few seconds on every server and worker, to the traces written from then on — including a trace that was still queued when the switch changed. Traces already stored keep what they hold until the retention period deletes them; delete individual traces from the Traces page to remove them sooner. The switches govern the native trace store only: the Langfuse export, while it is enabled, still receives each call's payloads (bounded per field). Turn it off as well to keep payloads inside the platform.

While a switch is off, an agent reading traces (inspect_my_traces, inspect_agent_traces, get_trace_payload, and the same tools on the XTM One MCP server) is told so in the result, so it does not read a run missing from the list, or an empty input or output, as something the run did.

Trace payload storage

While Store full payloads is on, trace payloads — the prompt, the conversation, the model's reasoning, tool arguments and results — are stored whole. They are audit material, and the trace store holds the only copy, so nothing is clipped on the way in.

Large payloads are moved out of the database rather than truncated. Above TRACE_PAYLOAD_OFFLOAD_THRESHOLD_BYTES (default 1 MiB), the payload is written to object storage and the trace row keeps a pointer with the exact size, a SHA-256 digest and a short excerpt. In the Traces UI those payloads show a Load full payload action that fetches the complete value on demand; the panes show a bounded preview until you ask for it, so opening a trace with megabyte-sized payloads stays fast.

Images and PDFs inside a payload are stored the same way, one object each: the payload keeps a reference with the blob's content type, byte size and SHA-256, and the span inspector lists them as attachments that render the actual image on demand. This is what makes the screenshot an agent received — or the chart it produced — recoverable from its trace; previously media was replaced by an [image] placeholder, and since the trace store holds the only copy, the picture was gone for good.

Variable	What it controls
`TRACE_PAYLOAD_OFFLOAD_THRESHOLD_BYTES`	Size above which a payload goes to object storage (default 1 MiB). `0` disables offloading and always stores inline.
`TRACE_PAYLOAD_HARD_MAX_BYTES`	Inline fallback ceiling used only when object storage is unreachable (default 32 MiB). Below it the payload is stored inline anyway; above it the trace records that the payload could not be stored, so the gap is visible instead of silent.
`TRACE_MEDIA_OFFLOAD_ENABLED`	Whether media blobs are stored (default `true`). Set to `false` to replace them with an `[image]` placeholder instead, as earlier versions did.

Offloading requires the same object storage as file attachments (the S3_* settings). Everything one trace owns — offloaded payloads and media alike — lives under a traces/<trace_id>/ prefix. Plan disk for both volumes accordingly. Trace retention bounds how long payloads and their objects are kept — the retention purge deletes the objects along with the rows, under both store backends.

Langfuse export

When Langfuse is configured, the payloads exported to it still have media replaced by placeholders and long strings truncated, regardless of TRACE_MEDIA_OFFLOAD_ENABLED. Langfuse ingestion has its own size limits, and its SDK would otherwise upload every blob to its own separate media storage. Only the export is reduced; the durable trace keeps everything.

Trace retention

How long traces are kept is a platform setting, not an environment variable: set it under Settings → Observability → Trace retention (days) (1–365, 90 by default). It applies to both store backends, in the same maintenance cycle, and each cycle deletes the offloaded payload objects belonging to the traces it expires.

PostgreSQL — expired traces are deleted in batches roughly hourly. A day's aggregated statistics (llm_daily_stats) are built before its raw traces are deleted and outlive them, so dashboards keep reporting on days whose traces are gone. A day whose statistics are due to be rebuilt (a repricing filled in costs they were built without) keeps its traces until that rebuild has run.
ClickHouse — the same cycle sweeps the expired window: it reads the payload pointers off the expiring rows, deletes the rows, then deletes the objects they pointed at. The tables also carry a TTL as a backstop, re-declared whenever the retention setting changes and re-checked on each cycle, so retention still applies to tables that already exist and to a platform whose maintenance loop is not running.

Why ClickHouse keeps a few extra days

The ClickHouse TTL is deliberately set a week beyond the retention window rather than exactly on it. Offloaded payload objects can only be found through the rows that point at them, so the rows have to outlive the object purge — a TTL on the exact boundary would let ClickHouse drop rows before the sweep had read their pointers, orphaning objects in the bucket. In normal operation the sweep enforces the exact window and the TTL never fires. If the maintenance loop stays down for longer than that margin, the TTL still expires the rows (retention is never silently extended) but their objects are left behind; that is a storage cost, and the sweep logs when it cannot reach object storage.

Cost by agent, user and group

The Cost breakdown card on Analytics (administrators only) splits the window's cost three ways, chosen with its By agent / By user / By group switch (kept in the URL as ?breakdown=users / ?breakdown=groups, so a view can be shared):

By user — the users who cost the most over the window, with their email, groups, how many agents they used, traces, tokens, average cost per trace and share of the total. The table holds up to the 500 costliest users, 10 per page and searchable by name, email or group; when more users ran traces, it says how many there were in all and counts the rest together as one N other users row on its last page. A deactivated account is marked Inactive, a deleted one Deleted user.
By group — every group whose members ran traces, with its active users (members who ran at least one trace) out of its current members, the average cost per active user and its share of the total. Users in no group are shown apart as Users in no group.

Both views show the runs no account is behind (anonymous channel senders, public widget visitors, the /mcp/global key, assignment runs without a user) apart, as No user account on their last page, so the user view adds up to the dashboard total.

Groups are read as they are today: traces do not record a group, so a trace counts for the groups its user belongs to now, and moving a user to another group moves their past cost with them. A user in several groups counts in each, so the group rows can add up to more than the total; the card says so when it happens. Both views read the same per-user daily aggregates as the rest of the dashboard on long windows, so they still answer after the traces are purged. The same data is available from GET /api/v1/traces/analytics/cost-by-user (limit, 1-1000, default 100; the users past it are returned together as other_users) and GET /api/v1/traces/analytics/cost-by-group, with the dashboard's filters.

Langfuse (optional)

You can additionally forward traces to Langfuse for external observability. Set LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, and LANGFUSE_SECRET_KEY. LANGFUSE_HOST defaults to Langfuse Cloud; point it at your self-hosted Langfuse to keep trace data on your own infrastructure.

Langfuse is complementary to the native store — enabling it does not disable native traces. Payloads sent to Langfuse are bounded per field, because its ingestion rejects oversized events; the native store keeps the unabridged copy. The Enable local trace storage and Store full payloads switches do not apply to the export: Langfuse receives each call's payloads for as long as it is enabled.

Logs

The platform emits structured logs to stdout, which your container runtime or log aggregator collects.

Variable	What it controls
`LOG_LEVEL`	Verbosity (`debug`, `info`, `warning`, `error`, `critical`)
`LOG_FORMAT`	`json` (structured, for aggregators) or `console` (human-readable)

One structured http_request line is emitted per HTTP request on the app.access logger. Successful requests to /api/health and /metrics are logged at debug so routine probes and scrapes do not flood your logs; client errors are logged at warning and server errors at error.

In production, keep LOG_FORMAT=json and ship logs to a central aggregator (Loki, Elasticsearch, CloudWatch, etc.) so you can search and alert across replicas.

Sandbox call logs

Every OpenSandbox-routed call (code interpreter, custom JS tool, custom Python tool) emits one additional structured line, sandbox_call, carrying CPU/RAM usage, execution time, egress policy, and outcome. It ships through the same stdout path as everything else above — no extra infrastructure required. The full field reference (names, types, units, when each is present) lives in Configuration § Sandbox call log fields; this section covers how to query it once it's in Grafana.

Example line (LOG_FORMAT=json):

{"event": "sandbox_call", "usage_type": "code_interpreter", "outcome": "success", "user_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "cpu_used_percentage": 12.5, "cpu_count": 1.0, "memory_used_in_mib": 64.2, "memory_total_in_mib": 128.0, "tte_ms": 842, "egress_default_action": "deny", "egress_allow_list": [], "level": "info", "logger": "app.services.sandbox_call_logging", "timestamp": "2026-07-31T12:00:00.000000Z"}

With logs shipped into Loki, these LogQL queries build directly on the fields above — swap {container="platform"} for whatever stream label your Promtail/Grafana Agent config actually attaches (container name, job, Kubernetes pod label, etc.):

All sandbox calls (base query for a Logs panel):

{container="platform"} | json | event="sandbox_call"

Failure rate by usage_type (stat/alert panel — ratio of failing to total calls per interval):

sum by (usage_type) (count_over_time({container="platform"} | json | event="sandbox_call" | outcome="failure" [$__interval]))
/
sum by (usage_type) (count_over_time({container="platform"} | json | event="sandbox_call" [$__interval]))

Failure breakdown by reason (table/pie panel — sandbox_ready_timeout, sandbox_create_failed, sandbox_create_error, nonzero_exit, execution_error):

sum by (failure_reason) (count_over_time({container="platform"} | json | event="sandbox_call" | outcome="failure" [$__interval]))

Average execution time (TTE) per usage_type (time series panel):

avg_over_time({container="platform"} | json | event="sandbox_call" | outcome="success" | unwrap tte_ms [5m]) by (usage_type)

CPU / memory usage trend (swap cpu_used_percentage for memory_used_in_mib or memory_total_in_mib):

avg_over_time({container="platform"} | json | event="sandbox_call" | unwrap cpu_used_percentage [5m]) by (usage_type)

Egress policy audit (confirm what's actually live, per call):

{container="platform"} | json | event="sandbox_call" | line_format "{{.usage_type}}: {{.egress_default_action}} allow={{.egress_allow_list}}"

CPU/RAM/TTE fields are simply absent (not null) on records where the sandbox never reached completion — sandbox_ready_timeout, sandbox_create_failed, and sandbox_create_error records have none of them. unwrap queries skip lines missing the unwrapped field rather than erroring, so a TTE/CPU/RAM panel naturally reflects only the calls that actually ran, without extra filtering.

What to watch

At a minimum, alert on:

/api/health/ready failing for more than a few minutes (platform not serving)
A rising rate of 5xx responses in the access logs
Worker containers repeatedly restarting (background jobs and assignment runs stall)
PostgreSQL connection saturation — see the connection-budget formula in Configuration
Disk usage on the PostgreSQL and object-storage volumes, which grow with knowledge bases and file attachments
A rising sandbox_call failure rate or an unfamiliar failure_reason for OpenSandbox-routed code interpreter/custom tool calls — see Sandbox call logs above

Under construction

Recommended alert thresholds and a reference set of Prometheus alert rules for self-hosted deployments are being prepared and will be added here.

Next step

If a signal points to a problem, continue with Troubleshooting for common issues and how to diagnose them.