> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mithunai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Score an assistant against a question set

> Write questions you know the answers to, then score each answer for grounding, citations, expected sources and facts, and abstention, in the console or CI.

An eval set is a list of questions you know the answers to, and questions the assistant should decline. A run asks the assistant each one through its normal answer path and scores what comes back. Run it from **Settings → Answer quality** in the console, through the API, or from the command line in CI.

Each question is one answer against your [monthly allowance](https://docs.mithunai.com/analyze/usage-and-limits.md). No conversation is created. By default nothing is stored: the run is shown to you and recorded in your [audit log](https://docs.mithunai.com/administration/audit-log.md) with its counts only. With [kept sets and runs](#kept-sets-and-runs-beta) on, a set and its runs are kept on the platform. Running an eval requires the owner, admin or editor role.

## What is scored

Every score compares text. No model grades an answer, because a model asked whether an answer is right judges how fluent it is, not whether your content supports it.

| Check | Passes when |
| - | - |
| **Grounded** | At least 60% of the answer's content words appear in the passages it cites. |
| **Citations support it** | Every cited passage shares words with the answer, and at least one passage is cited. |
| **Expected source cited** | One of the sources you named appears in the title or address of a cited passage. |
| **Expected facts** | Each fact appears in the answer, or every word of it does. |
| **Abstention** | The assistant declined exactly when you said it should. |

A question passes when its abstention is right and, for a question that should be answered, every check above holds. The pass rate is the share of questions that passed.

> [!NOTE]
>
> These checks count shared words. They cannot tell a paraphrase that contradicts a passage from one
> that agrees with it, so read the answers behind any surprising pass.

## The eval-set file

The console, the API and the command line read the same file:

```json evals.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "cases": [
    {
      "question": "How do I rotate an API key?",
      "expected_facts": ["one hour"],
      "expected_sources": ["api-keys"],
      "should_abstain": false
    },
    { "question": "What is our CEO's salary?", "should_abstain": true }
  ]
}
```

Only `question` is required. A case may name up to 10 facts and 5 sources. In the console, **Download file** saves the set you built and **Import file** loads one; the set is also kept in your browser while you edit it.

## In the console

1. Open **Settings → Answer quality**.
2. Add questions, with the facts and sources a right answer has, or import a file.
3. Choose the assistant and select **Run**. Questions are asked one at a time, so you see progress.
4. Each question shows **Pass** or **Fail** and the reason. Select **Show answer** to read the answer and its sources.

If your deployment has no model provider configured, the run says so and asks nothing. If the model becomes unreachable or your allowance runs out partway, the questions already scored are kept.

## Kept sets and runs (Beta)

Under the questions, **Kept sets and runs (Beta)** keeps a set on the platform under a name, runs it again whenever you like, and lists the runs newest first with their pass rate, so you can watch quality move across a change to your knowledge or to the assistant.

1. Add the questions, enter a **Set name** and select **Keep this set**.
2. Select **Run** beside the set. The platform scores it a few questions at a time and the screen asks again until the run is done, so you see it finish. A run that cannot go on (no model, the model unreachable, the allowance used) is listed as **Could not go on** with what it scored.
3. Each run shows **{passed} of {total} passed**, the set it ran and when it started.

Each run spends one answer per question, like any other. Keeping and running require the owner, admin or editor role, as running does.

**Off by default, and then not shown at all.** Your deployment's operator turns on kept history. While it is off the platform answers `404` and **Settings → Answer quality** shows only what is above. Access on request.

The API is [`POST`](https://docs.mithunai.com/api-reference/assistants/save-eval-set) and [`GET /assistants/{assistant_id}/eval-sets`](https://docs.mithunai.com/api-reference/assistants/list-eval-sets), [`POST /assistants/{assistant_id}/eval-sets/{set_id}/runs`](https://docs.mithunai.com/api-reference/assistants/run-eval-set) (no body starts a run; `{"run_id"}` scores the next cases) and [`GET /assistants/{assistant_id}/eval-runs`](https://docs.mithunai.com/api-reference/assistants/list-eval-runs).

## From the API

[`POST /assistants/{assistant_id}/evals`](https://docs.mithunai.com/api-reference/assistants/run-assistant-eval) takes the same `cases`, up to 10 per request.

## In CI

Run the file against an assistant with `flask mithunai-evals run` on the API host:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flask mithunai-evals run \
  --organization-id "$ORGANIZATION_ID" \
  --as-user "$ACCOUNT_ID" \
  --assistant-id "$ASSISTANT_ID" \
  --file evals.json \
  --min-pass-rate 0.9
```

The run acts as the account you name, which must hold the owner, admin or editor role in that organization. It reads up to 200 questions. It exits `0` at or above `--min-pass-rate` (default `1.0`), `1` below it, and `2` when it could not run. Add `--json` for the API's response shape.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.