Skip to main content
An eval set is a list of questions you know the answers to, and questions the assistant should decline. A run asks the assistant each one through its normal answer path and scores what comes back. Run it from Settings → Answer quality in the console, through the API, or from the command line in CI. Each question is one answer against your monthly allowance. No conversation is created. By default nothing is stored: the run is shown to you and recorded in your audit log with its counts only. With kept sets and runs on, a set and its runs are kept on the platform. Running an eval requires the owner, admin or editor role.

​
What is scored

Every score compares text. No model grades an answer, because a model asked whether an answer is right judges how fluent it is, not whether your content supports it. A question passes when its abstention is right and, for a question that should be answered, every check above holds. The pass rate is the share of questions that passed.
These checks count shared words. They cannot tell a paraphrase that contradicts a passage from one that agrees with it, so read the answers behind any surprising pass.

​
The eval-set file

The console, the API and the command line read the same file:
evals.json
Only question is required. A case may name up to 10 facts and 5 sources. In the console, Download file saves the set you built and Import file loads one; the set is also kept in your browser while you edit it.

​
In the console

  1. Open Settings → Answer quality.
  2. Add questions, with the facts and sources a right answer has, or import a file.
  3. Choose the assistant and select Run. Questions are asked one at a time, so you see progress.
  4. Each question shows Pass or Fail and the reason. Select Show answer to read the answer and its sources.
If your deployment has no model provider configured, the run says so and asks nothing. If the model becomes unreachable or your allowance runs out partway, the questions already scored are kept.

​
Kept sets and runs (Beta)

Under the questions, Kept sets and runs (Beta) keeps a set on the platform under a name, runs it again whenever you like, and lists the runs newest first with their pass rate, so you can watch quality move across a change to your knowledge or to the assistant.
  1. Add the questions, enter a Set name and select Keep this set.
  2. Select Run beside the set. The platform scores it a few questions at a time and the screen asks again until the run is done, so you see it finish. A run that cannot go on (no model, the model unreachable, the allowance used) is listed as Could not go on with what it scored.
  3. Each run shows of passed, the set it ran and when it started.
Each run spends one answer per question, like any other. Keeping and running require the owner, admin or editor role, as running does. Off by default, and then not shown at all. Your deployment’s operator turns on kept history. While it is off the platform answers 404 and Settings → Answer quality shows only what is above. Access on request. The API is POST and GET /assistants/{assistant_id}/eval-sets, POST /assistants/{assistant_id}/eval-sets/{set_id}/runs (no body starts a run; {"run_id"} scores the next cases) and GET /assistants/{assistant_id}/eval-runs.

​
From the API

POST /assistants/{assistant_id}/evals takes the same cases, up to 10 per request.

​
In CI

Run the file against an assistant with flask mithunai-evals run on the API host:
The run acts as the account you name, which must hold the owner, admin or editor role in that organization. It reads up to 200 questions. It exits 0 at or above --min-pass-rate (default 1.0), 1 below it, and 2 when it could not run. Add --json for the API’s response shape.
Last modified on November 12, 2026