Connect a website, sitemap or GitHub

A source tells MITHUNAI where to read content from. Create it once, then sync it whenever the content changes. For files on your computer, see File uploads.

Console
API

Open Knowledge and select Connect a knowledge source.
Choose Website, Sitemap, GitHub or File upload, and fill in the fields described below.
Save. The source appears in the list and its first ingestion starts; its status updates as the job runs.

Use Re-sync to ingest changes, and Remove to delete a source.

curl --request POST "$MITHUNAI_URL/arukz/api/v1/knowledge/sources" \
  --header "Authorization: Bearer $MITHUNAI_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "collection_id": "<collection id>",
    "source_type": "website",
    "name": "Product documentation",
    "configuration": {
      "seed_url": "https://docs.example.com/",
      "path_prefix": "/docs",
      "max_depth": "3",
      "max_pages": "500"
    }
  }'

Then sync the source to ingest it. Every configuration value is a string, and path lists are comma-separated inside one string. See Create a source.

Sitemap

Reads exactly the URLs listed in a sitemap.xml, without following links. This is the most predictable way to ingest a documentation site: you ingest what you publish, nothing more.

Setting	Required	Description
`seed_url`	Yes	The sitemap URL, such as `https://docs.example.com/sitemap.xml`. Sitemap indexes are followed
`max_pages`	No	Maximum pages read per run, from 1 to 5,000. Default 500
`path_prefix`, `include_paths`, `exclude_paths`	No	Filter the listed URLs, as for a website

Listed URLs must be on the same origin as the sitemap.

Website

Crawls a site from a start URL, following links on the same origin (scheme, host and port). Use it when no sitemap exists.

Setting	Required	Description
`seed_url`	Yes	Where the crawl starts. An absolute `http` or `https` URL
`path_prefix`	No	Only crawl paths at or below this prefix, such as `/docs`. Matches whole path segments, so `/docs` does not match `/docs-internal`
`max_depth`	No	How many links to follow from the start URL, from 1 to 10. Default 3
`max_pages`	No	Maximum pages per run, from 1 to 5,000. Default 500
`include_paths`	No	Comma-separated path patterns. When set, only matching pages are ingested
`exclude_paths`	No	Comma-separated path patterns to skip. Exclusions win over inclusions

Redirects are not followed, and links to images, media, archives, fonts, stylesheets and scripts are skipped.

GitHub

Reads files from a repository at a branch, tag or commit.

Setting	Required	Description
`owner`	Yes	The GitHub account or organisation, such as `acme`
`repository`	Yes	The repository name, such as `platform-docs`
`ref`	No	Branch, tag or commit SHA. Empty follows the default branch
`include_paths`	No	Comma-separated path patterns, such as `docs/,README.md`. When set, only matching files are ingested
`exclude_paths`	No	Comma-separated patterns to skip. Replaces the default list rather than adding to it

By default MITHUNAI skips .git/, node_modules/, vendor/, dist/, build/, target/, .venv/, __pycache__ directories, minified JavaScript and lock files. If you set exclude_paths, include any of those you still want skipped. Repositories are read with the GitHub access your MITHUNAI deployment is configured with. To connect a private repository, ask your MITHUNAI administrator to grant that access. You never paste a token into a source.

Path patterns

In include_paths and exclude_paths:

Pattern	Matches
`*`	Anything within one path segment
`**`	Anything across segments
`?`	One character
A trailing `/`	Everything below a directory

For example, docs/**/*.md,README.md ingests Markdown under docs/ and the root README.

Choose the right source

If your content is	Use
A documentation site with a sitemap	Sitemap
A site without a sitemap	Website, with `path_prefix` to stay in the docs
Docs-as-code in a repository	GitHub, with `include_paths`
PDFs, Word documents or exported files	File upload

Security

Source settings are visible to every member of your organisation, so they never hold credentials. Crawls cannot reach private or internal network addresses, and URLs containing a username or password are refused. See Security.

Get started

Knowledge

Assistants & Agents

Channels & Integration

Architecture & Solutions

Administration

Trust and support

Connect a website, sitemap or GitHub

Sitemap

Website

GitHub

Path patterns

Choose the right source

Security

​Sitemap

​Website

​GitHub

​Path patterns

​Choose the right source

​Security

Sitemap

Website

GitHub

Path patterns

Choose the right source

Security