> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mithunai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sync sources and keep answers current

> Ingestion reads a source, extracts its text, splits it into passages and indexes them. Start a sync, follow the job, read its counters, cancel a bad run.

Ingestion reads a source, extracts its text, splits it into passages and indexes them for retrieval. It runs in the background as a **job**, because a large site can take minutes.

## Start a sync

In the console, a new source starts syncing when you save it, and **Re-sync** starts another run. With the API:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST "$MITHUNAI_URL/arukz/api/v1/knowledge/sources/$SOURCE_ID/sync" \
  --header "Authorization: Bearer $MITHUNAI_API_KEY"
```

The response is `202 Accepted` with the job. While a run is active, syncing again returns the same job rather than starting a second one, so it is safe to call from a retrying script.

## Follow a job

Poll [Get a job](/api-reference/knowledge/get-job), or [Get a source's current job](/api-reference/knowledge/get-source-job), until `is_active` is `false`.

| Status              | Meaning                                                         |
| ------------------- | --------------------------------------------------------------- |
| `pending`, `queued` | Waiting for a worker, including between retry attempts          |
| `running`           | Reading and indexing the source                                 |
| `succeeded`         | Every document was ingested, or was already current             |
| `partial`           | The run finished, but some documents failed                     |
| `failed`            | The run could not proceed. See `error_code` and `error_message` |
| `cancelled`         | Stopped on request. Documents ingested before the stop remain   |
| `skipped`           | The source had not changed since its last successful run        |

Each job reports counters: documents discovered, ingested, unchanged, skipped, failed and deleted, plus passages written and bytes fetched. `documents_skipped` counts files refused before any work, such as unsupported types, excluded paths or files over the size limit.

Transient failures of a whole run are retried automatically; `attempt` and `max_attempts` show where the job is.

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import time
import requests

BASE = f"{os.environ['MITHUNAI_URL']}/arukz/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['MITHUNAI_API_KEY']}"}

job = requests.post(f"{BASE}/knowledge/sources/{os.environ['SOURCE_ID']}/sync", headers=HEADERS, timeout=30).json()
while job["is_active"]:
    time.sleep(10)
    job = requests.get(f"{BASE}/knowledge/jobs/{job['id']}", headers=HEADERS, timeout=30).json()

print(job["status"], job["counters"])
```

## Cancel a job

Cancel a pending, queued or running job with [Cancel a job](/api-reference/knowledge/cancel-job). Documents already ingested in that run stay in the collection.

## What changes on a re-sync

Documents are identified by their content, so a re-sync only re-indexes what changed:

* **New and changed** documents are ingested.
* **Unchanged** documents are recognised and left alone.
* **Documents no longer in the source** are removed.

If nothing in the source changed since the last successful run, the job ends `skipped`.

## Keep knowledge current

MITHUNAI does not re-sync on a schedule yet. Trigger a sync:

* from your documentation publishing pipeline, after each deploy;
* from a scheduled job, at a frequency that matches how often the content changes;
* manually, with **Re-sync**, after a significant change.

Use an API key with the **Knowledge operator** role for automated syncs. It can manage knowledge and nothing else.

## Archived collections

An archived collection keeps answering questions but accepts no new content. A sync against a source in an archived collection is accepted and the job ends `failed`. [Restore the collection](/api-reference/knowledge/update-collection) to ingest again.

## When a job fails

A job that ends `failed` or ingests fewer pages than you expected usually has one of a handful of causes: the site is unreachable or slower than the fetch timeouts allow, the crawl hit its depth or page limit, the include and exclude patterns filtered out the pages you wanted, or a URL resolved to a private network address and was refused by design. [Troubleshooting](/resources/troubleshooting) works through each with the fix.