Sync sources and keep answers current

Ingestion reads a source, extracts its text, splits it into passages and indexes them for retrieval. It runs in the background as a job, because a large site can take minutes.

Start a sync

In the console, a new source starts syncing when you save it, and Re-sync starts another run. The Ingestion tab on Sources lists recent jobs and their outcomes, shows a running job’s progress, and can cancel one. With the API:

curl --request POST "$MITHUNAI_URL/mithunai/api/v1/knowledge/sources/$SOURCE_ID/sync" \
  --header "Authorization: Bearer $MITHUNAI_API_KEY"

The response is 202 Accepted with the job. While a run is active, syncing again returns the same job rather than starting a second one, so it is safe to call from a retrying script.

Follow a job

Poll Get a job, or Get a source’s current job, until is_active is false.

Status	Meaning
`pending`, `queued`	Waiting for a worker, including between retry attempts
`running`	Reading and indexing the source
`succeeded`	Every document was ingested, or was already current
`partial`	The run finished, but some documents failed
`failed`	The run could not proceed. See `error_code` and `error_message`
`cancelled`	Stopped on request. Documents ingested before the stop remain
`skipped`	A scheduled re-sync found the source unchanged; nothing re-read

Each job reports counters: documents discovered, ingested, unchanged, skipped, failed and deleted, plus passages written and bytes fetched. documents_skipped counts files refused before any work, such as unsupported types, excluded paths or files over the size limit. Transient failures of a whole run are retried automatically; attempt and max_attempts show where the job is.

import os
import time
import requests

BASE = f"{os.environ['MITHUNAI_URL']}/mithunai/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['MITHUNAI_API_KEY']}"}

job = requests.post(f"{BASE}/knowledge/sources/{os.environ['SOURCE_ID']}/sync", headers=HEADERS, timeout=30).json()
while job["is_active"]:
    time.sleep(10)
    job = requests.get(f"{BASE}/knowledge/jobs/{job['id']}", headers=HEADERS, timeout=30).json()

print(job["status"], job["counters"])

Cancel a job

Cancel a pending, queued or running job with Cancel a job. Documents already ingested in that run stay in the collection.

What changes on a re-sync

Documents are identified by their content, so a re-sync only re-indexes what changed:

New and changed documents are ingested.
Unchanged documents are recognised and left alone.
Documents no longer in the source are removed, unless that would remove most of the source (below).

When a sync would remove most of a source

If a re-sync would remove more than 45% of a source’s documents, nothing is removed. The documents are still used for answers, and the run shows a warning on Ingestion with how many documents it concerns. A listing that suddenly lost most of its pages (a moved sitemap, a renamed branch, a page that now asks for a login) looks the same as a real deletion, so a person decides:

Remove them starts a sync that removes the documents the source no longer offers.
Keep them removes nothing. Scheduled syncs then skip the source until it changes; a manual sync checks again.

Either choice needs a role that can delete documents and is recorded in the audit log. Over the API, see Confirm a held removal and Keep held documents. The percentage is set per deployment by its operator. A sync you start, from the console or the API, always re-reads the source, so a run over a source in which nothing changed ends succeeded, with every document counted as documents_unchanged. A scheduled re-sync first asks whether the source has changed at all, and when it has not, ends skipped without re-reading it. A website or sitemap cannot be asked that without crawling it, so its scheduled runs never end skipped.

Keep knowledge current

A deployment can re-sync sources on a schedule, but that is off unless its operator has turned it on, and it runs on one interval for every source. See Scheduled re-sync. Whether or not it is on, trigger a sync yourself when a change must be picked up:

from your documentation publishing pipeline, after each deploy;
from a scheduled job, at a frequency that matches how often the content changes;
manually, with Re-sync, after a significant change.

Use an API key with the Knowledge operator role for automated syncs. It can manage knowledge, and read the organisation’s details and member list, and nothing else.

Archived collections

An archived collection keeps answering questions but accepts no new content. A sync against a source in an archived collection is accepted and the job ends failed. Restore the collection to ingest again.

When a job fails

A job that ends failed or ingests fewer pages than you expected usually has one of a handful of causes: the site is unreachable or slower than the fetch timeouts allow, the crawl hit its depth or page limit, the include and exclude patterns filtered out the pages you wanted, or a URL resolved to a private network address and was refused by design. Troubleshooting works through each with the fix.

MITHUNAI RAG

Knowledge sources

Answers and assistants

Deploy

Analyze

Sync sources and keep answers current

Start a sync

Follow a job

Cancel a job

What changes on a re-sync

When a sync would remove most of a source

Keep knowledge current

Archived collections

When a job fails

​Start a sync

​Follow a job

​Cancel a job

​What changes on a re-sync

​When a sync would remove most of a source

​Keep knowledge current

​Archived collections

​When a job fails

Start a sync

Follow a job

Cancel a job

What changes on a re-sync

When a sync would remove most of a source

Keep knowledge current

Archived collections

When a job fails