Automated Grounding Benchmarks & Hallucination Auditing

MITHUNAI evaluates answer quality using automated factual grounding benchmarks that verify every generated sentence against retrieved source passages before deployment. Unlike subjective human spot-checks, our evaluation framework runs quantitative scoring across three rigorous metrics: factual faithfulness, citation recall, and abstention precision on unanswerable negative queries.

Core Grounding Metrics

The platform evaluates assistants against four empirical metrics:

Metric	What It Measures	Target Threshold
Faithfulness Score	Percentage of claims in the answer that are logically entailed by retrieved passages.	$\ge 99.0\%$
Citation Precision	Accuracy of source URLs and line ranges cited for specific factual assertions.	$\ge 98.5\%$
Abstention Precision	Accuracy of declining to answer queries when required information is absent from the corpus.	$\ge 99.5\%$
Answer-First Compliance	Whether the primary conclusion or solution is stated in the lede sentence.	$100\%$

Running Golden Dataset Evaluations

A Golden Dataset consists of paired test items containing realistic user questions, required source passages, and expected canonical answers:

Golden Test Item Schema

{
  "test_id": "eval_pay_01",
  "question": "What is the maximum file size for webhook payload delivery?",
  "expected_behavior": "answer",
  "required_citation_urls": ["https://docs.example.com/webhooks/limits"],
  "expected_keywords": ["10 MB", "compressed", "413 Payload Too Large"]
}

Testing Negative / Out-of-Scope Queries

Crucially, 30% of your evaluation suite should consist of negative queries—questions that resemble valid inquiries but whose answers are deliberately missing from your documentation:

Negative Test Item Schema

{
  "test_id": "eval_neg_04",
  "question": "How do I configure GraphQL subscriptions for legacy v0.9 endpoints?",
  "expected_behavior": "abstain",
  "prohibited_behavior": "guess_or_hallucinate",
  "expected_abstention_flag": true
}

If the assistant attempts to guess or generate hypothetical GraphQL schemas instead of abstaining, the evaluation gate fails automatically.

CI/CD Regression Testing

Automate grounding verification in your build pipeline using the MITHUNAI CLI:

Automated Benchmark Run

# Run evaluation suite against test assistant
mithunai eval run \
  --assistant asst_staging_01 \
  --dataset ./tests/golden-questions.json \
  --min-faithfulness 0.98 \
  --output ./eval-report.json

# Exit code 0 if all tests pass; 1 if regressions are detected
echo "Evaluation passed successfully."

By enforcing these evaluation gates in CI/CD, documentation and engineering teams ensure that new documentation syncs never introduce factual regressions into customer-facing assistants.

Get started

Knowledge

Assistants & Agents

Channels & Integration

Architecture & Solutions

Administration

Trust and support

Automated Grounding Benchmarks & Hallucination Auditing

Core Grounding Metrics

Running Golden Dataset Evaluations

Testing Negative / Out-of-Scope Queries

CI/CD Regression Testing

​Core Grounding Metrics

​Running Golden Dataset Evaluations

​Testing Negative / Out-of-Scope Queries

​CI/CD Regression Testing

Core Grounding Metrics

Running Golden Dataset Evaluations

Testing Negative / Out-of-Scope Queries

CI/CD Regression Testing