Skip to content

Evaluations

An evaluation runs one or more agents, on one or more models, against the test cases of a dataset, and scores each answer. Use it to pick a model for an agent, or to check that a new model or persona does not make the answers worse. Evaluations and datasets are administrator tools: they live under Evaluations, and Evaluations > Datasets holds the test cases.

  • A dataset is an ordered list of entries. Each entry has a prompt, the answer you expect, and optionally files attached to the prompt.
  • An evaluation links agents and datasets, says which models to run (the agent's default model, a list of models, or both), and how to score: a judge model compares every answer with the expected one.

The built-in OCR benchmark

XTM One ships one ready-made benchmark: the OCR benchmark dataset and the OCR benchmark evaluation. It measures how faithfully a model reads a document it is given.

The dataset holds six short cyber threat intelligence documents: four text PDFs (an advisory, a table of indicators, a notice in French, a two-page incident report) and two scanned pages (a memo and a table, skewed and speckled like a photocopy). Every entry asks for a verbatim transcription, and the expected answer is the document's exact text. The documents are fictional and written by Filigran for this benchmark; their licence and source are recorded in backend/app/evaluations/seed_data/SOURCES.md.

The documents reach each model the way a file attached in a conversation does:

  • A PDF is read natively by the providers that support it (Anthropic, Google Gemini, AWS Bedrock). Other providers receive the text extracted from the PDF, so for them the entry measures how well the model restores the document's structure from that text.
  • A scanned page is an image, so it needs a vision-capable model. A text-only model fails those two entries: that result is expected, not a platform error.

To run it:

  1. Open Evaluations and select OCR benchmark.
  2. Add the models you want to compare. Out of the box it runs the agent's default model only.
  3. Start a run, then compare the scores per model and per entry.

The evaluation runs the built-in Document Transcriber agent, which is hidden from chat and carries no tools, so the scores compare models, not agent configurations. The evaluation is created once: its models, agents and schedule are yours to change, and restarting the platform does not reset them. If you delete it, it is created again, with its default settings, at the next start. The dataset, on the other hand, is re-synced at every startup; to build a variant of it, export it and import the copy. When an upgrade changes the text or the document of one of its entries, the dataset's entries are replaced: earlier runs keep their scores, but their per-entry results are removed, so a score measured on one document is never shown against another. A run in progress keeps the entries and documents it started with; the dataset is then updated at the next start.

The documents are stored in the platform's object storage when it starts. If the object storage cannot be reached at that moment, the dataset appears without its documents and the OCR benchmark evaluation is not created yet: both are completed at the next start that can store the documents, so the benchmark is not offered before the documents it scores are there. When an upgrade changes a document and the object storage cannot be reached, the benchmark keeps the entries and documents it had until a start that can store the new one.