What is measured
The evaluation uses a fictional corpus for two organisations, so every planted fact has exactly one right answer, and one page deliberately carries a prompt injection.
Every metric is computed deterministically from the answer, its citations and the retrieval ranking. No model grades another model, so a result is reproducible. Two properties fail the run whatever the model: any cross-organisation leak, and any other organisation’s passage shown to the model.
Latest results
Run of 2026-09-25, withqwen2.5:3b-instruct answering and bge-m3 embeddings, on the same 70 items as the first real-model run of 2026-09-24:
On the first run, retrieval put a supporting document first for 88% of questions and found every needed document in the top 8 (hit@1 0.88, recall@8 1.00).
What these numbers do not tell you
- They are not your content. The corpus is fictional and small. Test your own assistant on your own questions before launch: see Evaluate an assistant.
- They are not a production model. A 3-billion-parameter model on a CPU is a floor. Each of the nine answerable questions it missed had the answer in the passages it was shown; it failed on output format, which a larger hosted model is expected to handle better. That expectation is not yet measured.
- Latency is not representative. The runs took tens of seconds per answer on a shared CPU, which says nothing about a hosted provider.
- Groundedness is measured lexically. A sentence that reuses the evidence’s words can still misstate it; unsupported numbers are checked exactly, but full entailment is not.
The evidence gate is calibrated per embedding model
The score an answer’s best passage must reach depends on the embedding model, because different models score similarity on different scales. The threshold is measured with this evaluation for each embedding model:bge-m3 has a measured value, and a model not yet measured, including the default text-embedding-3-small, uses a default. See Retrieval pipeline for how the gate works.