Skip to main content
Ingestion reads a source, extracts its text, splits it into passages and indexes them for retrieval. It runs in the background as a job, because a large site can take minutes.

​
Start a sync

In the console, a new source starts syncing when you save it, and Re-sync starts another run. With the API:
The response is 202 Accepted with the job. While a run is active, syncing again returns the same job rather than starting a second one, so it is safe to call from a retrying script.

​
Follow a job

Poll Get a job, or Get a source’s current job, until is_active is false. Each job reports counters: documents discovered, ingested, unchanged, skipped, failed and deleted, plus passages written and bytes fetched. documents_skipped counts files refused before any work, such as unsupported types, excluded paths or files over the size limit. Transient failures of a whole run are retried automatically; attempt and max_attempts show where the job is.

​
Cancel a job

Cancel a pending, queued or running job with Cancel a job. Documents already ingested in that run stay in the collection.

​
What changes on a re-sync

Documents are identified by their content, so a re-sync only re-indexes what changed:
  • New and changed documents are ingested.
  • Unchanged documents are recognised and left alone.
  • Documents no longer in the source are removed.
If nothing in the source changed since the last successful run, the job ends skipped.

​
Keep knowledge current

MITHUNAI does not re-sync on a schedule yet. Trigger a sync:
  • from your documentation publishing pipeline, after each deploy;
  • from a scheduled job, at a frequency that matches how often the content changes;
  • manually, with Re-sync, after a significant change.
Use an API key with the Knowledge operator role for automated syncs. It can manage knowledge and nothing else.

​
Archived collections

An archived collection keeps answering questions but accepts no new content. A sync against a source in an archived collection is accepted and the job ends failed. Restore the collection to ingest again.

​
When a job fails

A job that ends failed or ingests fewer pages than you expected usually has one of a handful of causes: the site is unreachable or slower than the fetch timeouts allow, the crawl hit its depth or page limit, the include and exclude patterns filtered out the pages you wanted, or a URL resolved to a private network address and was refused by design. Troubleshooting works through each with the fix.
Last modified on September 26, 2026