Connect a website, repository, API schema or help centre

A source tells MITHUNAI where to read content from. Create it once, then sync it whenever the content changes. For files on your computer, see File uploads.

Console
API

Open Sources and select Connect source. The Connect a knowledge source dialog opens.
Choose a source type and fill in the fields described below. The choices are under four headings. Websites and files: Website, Sitemap, File upload, OpenAPI schema and Custom answers. Knowledge bases: Zendesk Help Center, Confluence and Notion. Issues and support: Jira, Jira Service Management and Zendesk tickets. Community: Discourse forum and GitHub. For GitHub, also choose what to index: Files, Issues, Pull requests or Discussions, and tick This repository is private for a private one. Confluence, Notion, Jira, Jira Service Management, Zendesk tickets and a private repository then ask for a saved connection; their steps are on Confluence, Notion and Private GitHub.
Select Connect source in the dialog. The source appears in the list and its first ingestion starts; its status updates as the job runs.

Most types need one thing: an address, a repository or a file. Select Change filters to narrow what is read.Each source has a menu with Edit, Rename, Pause or Resume, Re-sync and Remove. See Edit, pause or rename a source.

curl --request POST "$MITHUNAI_URL/mithunai/api/v1/knowledge/sources" \
  --header "Authorization: Bearer $MITHUNAI_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "collection_id": "<collection id>",
    "source_type": "website",
    "name": "Product documentation",
    "configuration": {
      "seed_url": "https://docs.example.com/",
      "path_prefix": "/docs",
      "max_depth": "3",
      "max_pages": "500"
    }
  }'

Then sync the source to ingest it. Send every configuration value as a string; path lists are comma-separated inside one string. See Create a source.

Sitemap

Reads exactly the URLs listed in a sitemap.xml, without following links. This is the most predictable way to ingest a documentation site: you ingest what you publish, nothing more.

Setting	Required	Description
`seed_url`	Yes	The sitemap URL, such as `https://docs.example.com/sitemap.xml`. Sitemap indexes are followed
`max_pages`	No	Maximum pages read per run, from 1 to 5,000. Default 500
`path_prefix`, `include_paths`, `exclude_paths`	No	Filter the listed URLs by their path, as for a website

Listed URLs must be on the same origin as the sitemap.

Website

Crawls a site from a start URL, following links on the same origin (scheme, host and port). Use it when no sitemap exists.

Setting	Required	Description
`seed_url`	Yes	Where the crawl starts. An absolute `http` or `https` URL
`path_prefix`	No	Only crawl paths at or below this prefix, such as `/docs`. Matches whole path segments, so `/docs` does not match `/docs-internal`
`max_depth`	No	How many links to follow from the start URL, from 1 to 10. Default 3
`max_pages`	No	Maximum pages per run, from 1 to 5,000. Default 500
`include_paths`	No	Comma-separated path patterns. When set, only matching pages are ingested
`exclude_paths`	No	Comma-separated path patterns to skip. Exclusions win over inclusions

Redirects are not followed, and links to images, media, archives, fonts, stylesheets and scripts are skipped.

GitHub

Reads files from a repository at a branch, tag or commit.

Setting	Required	Description
`owner`	Yes	The GitHub account or organisation, such as `acme`
`repository`	Yes	The repository name, such as `platform-docs`
`ref`	No	Branch, tag or commit SHA. Empty follows the default branch
`include_paths`	No	Comma-separated path patterns, such as `docs/,README.md`. When set, only matching files are ingested
`exclude_paths`	No	Comma-separated patterns to skip. Replaces the default list rather than adding to it

By default MITHUNAI skips .git/, node_modules/, vendor/, dist/, build/, target/ and .venv/ at the root of the repository, and __pycache__ directories, minified JavaScript and lock files at any depth. A node_modules/ inside a package directory is not skipped by default; add **/node_modules/ to skip it. If you set exclude_paths, include any of those you still want skipped. A public repository needs no token. To connect a private one, save a read-only GitHub token of your own as a connection and tick This repository is private: see Connect a private GitHub repository. A source never holds a token itself, and one that uses your connection never falls back to the deployment’s own GitHub access.

GitHub issues, pull requests and discussions

A repository’s conversations often hold answers its files do not: the workaround in a closed issue, the reasoning in a pull request, the accepted answer in a discussion. Each is a source of its own, so a repository can be connected for its files and again for its issues. In the console, choose GitHub, enter the repository as owner/repo, and choose Issues, Pull requests or Discussions. That is enough to connect it; select Change filters to narrow what is read. With the API, use the source type and settings below.

What to index	`source_type`	One document per
Issues	`github_issues`	Issue, with its comments in order. Pull requests are not included
Pull requests	`github_pull_requests`	Pull request, with its description and comments. The diff is not read
Discussions	`github_discussions`	Discussion, with its accepted answer and comments. Beta: see the note below

Comments written by bots are left out. A citation links to the issue, pull request or discussion on GitHub. All three take the same settings:

Setting	Required	Description
`owner`	Yes	The GitHub account or organisation, such as `acme`
`repository`	Yes	The repository name, such as `platform`
`state`	No	`open`, `closed` or `all`. Default `all`: a closed issue is often the one with the answer
`max_age_days`	No	Skip anything last updated longer ago than this, from 0 to 3,650. Default 365. `0` means no limit
`include_labels`	No	Comma-separated labels. When set, only items carrying at least one of them are read
`exclude_labels`	No	Comma-separated labels to skip. Exclusions win over inclusions
`max_items`	No	How many to read, newest first, from 1 to 1,000. Default 200
`max_comments`	No	Comments read per item, from 0 to 100. Default 30

A label filter takes up to 50 labels of up to 100 characters each. Pull requests take three more:

Setting	Default	Description
`exclude_closed_unmerged`	`true`	Skip pull requests that were closed without being merged
`include_conversation_comments`	`true`	Read the comments on the pull request’s conversation
`include_review_comments`	`false`	Also read the review comments left on lines of code

With the API these three are the strings "true" and "false", like every other configuration value.

Discussions are in beta, and GitHub serves them only to a signed-in caller, so they need a GitHub token even for a public repository: either the GitHub access your MITHUNAI deployment is configured with, or a GitHub connection of your own (tick This repository is private and choose it). Without either, the source is created and its sync fails.

OpenAPI schema

Reads one OpenAPI 3.x or Swagger 2.x schema, in JSON or YAML, and turns each operation, one method on one path, into its own document: its summary, parameters, request body and responses. A question about one endpoint then finds that endpoint.

Setting	Required	Description
`schema_url`	Yes	The schema’s address. An absolute `http` or `https` URL
`max_operations`	No	Maximum operations read, from 1 to 2,000. Default 500

The schema must be readable without signing in.
It must be no larger than your deployment’s fetch limit, which is 1 MiB by default.
A $ref to another file or another host is not followed. If your schema is split across files, bundle it into one before you connect it.

A citation links to the schema.

Custom answers

Write the answer yourself for the questions where wording matters: a refund policy, a support address, a deprecation notice. A custom answers source holds question-and-answer pairs you type, and each pair becomes one document. In the console, choose Custom answers, name the set, and add your pairs.

Limit	Value
Pairs per source	1 to 25. For more, connect another source
Question	Up to 500 characters, on one line
Answer	Up to 4,000 characters, written in Markdown

The same question cannot appear twice in one source; case and spacing are ignored when comparing.

A custom answer is retrieved like any other passage of your content. The assistant uses it when it is the best match for what was asked, and is not guaranteed to return it word for word.

To change your answers, select Edit answers on the source, save, then sync. See Edit, pause or rename a source. With the API, the pairs are configuration keys: question_1 and answer_1, question_2 and answer_2, and so on. The numbers set the order.

Zendesk Help Center

Reads the articles of a Zendesk help centre that a signed-out visitor can read. Each article becomes one document, and a citation links to the article.

Setting	Required	Description
`help_center_url`	Yes	Any page of the help centre. Only the site’s address is kept
`locale`	No	A language of the help centre, such as `en-us`. Empty reads its default language
`category_id`	No	Read one category only: the number in the category’s address
`section_id`	No	Read one section only: the number in the section’s address. Set a category or a section, not both
`max_articles`	No	Maximum articles read, from 1 to 2,000. Default 500

Only public help centres can be connected. Drafts, internal articles and articles restricted to signed-in users are not read, and a help centre that requires signing in fails to sync. Images and attachments are not read.

Discourse forum

Reads the topics of a public Discourse forum. Each topic becomes one document holding its first posts, and a citation links to the topic.

Setting	Required	Description
`forum_url`	Yes	Any page of the forum. Only the site’s address is kept
`category_id`	No	Read one category only: the number at the end of the category’s address
`tag`	No	Read one tag only. Lower case: letters, digits, hyphens, dots and underscores. Set a category or a tag, not both
`solved_only`	No	`true` or `false`. Default `false`. Reads only topics with an accepted answer
`max_topics`	No	Maximum topics read, latest first, from 1 to 500. Default 100
`max_posts_per_topic`	No	Posts read from each topic, from 1 to 20. Default 20

Only public forums can be connected. Private categories and forums that require signing in are not read.
Only the first 20 posts of a topic are read, so an answer posted later in a long topic is not included.
solved_only needs a forum running the Solved plugin. On a forum without it, the sync fails rather than reading every topic.
A forum installed under a path, such as example.com/forum, cannot be connected.

Anyone who can register on a forum can post to it, so a forum is the least reviewed content you can connect. Connect a forum you moderate, and consider solved_only.

Path patterns

In include_paths and exclude_paths:

Pattern	Matches
`*`	Anything within one path segment
`**`	Anything across segments
`?`	One character
A trailing `/`	Everything below a directory

A pattern must match the whole path. For a GitHub source the path is relative to the repository root: docs/**/*.md,README.md ingests Markdown under docs/ and the root README. For a website or sitemap source the path is the URL’s, which starts with /: write /docs/**, not docs/**.

Choose the right source

If your content is	Use
A documentation site with a sitemap	Sitemap
A site without a sitemap	Website, with `path_prefix` to stay in the docs
Docs-as-code in a repository	GitHub, with `include_paths`
Answers that live in issue threads	GitHub, indexing Issues
An API reference	OpenAPI schema
A help centre on Zendesk	Zendesk Help Center
A wiki on Confluence Cloud	Confluence (beta)
Pages in Notion	Notion (beta)
A private repository	GitHub, with a connection (beta)
A community forum on Discourse	Discourse forum, with `solved_only` if it applies
A few answers that must be worded your way	Custom answers
PDFs, Word documents or exported files	File upload

Edit, pause or rename a source

Open a source’s menu on Sources.

Edit opens the form the source was connected with, filled in with its current settings. A source’s type cannot be changed, and an uploaded file has no settings to edit.
Rename changes the name and nothing else.
Pause stops the source being read, by you and by a scheduled re-sync. Its documents stay, and your assistant still answers from them. A run already in progress is not interrupted. Resume makes it readable again. Re-sync is unavailable while a source is paused.

Saving an edit does not read the source. Your assistant keeps answering from what the source held before, until you sync it. The sync adds what the new settings match and removes what they no longer do.

After you save, the console offers Sync now. If you choose Not now, the source is marked as edited since it was last synced until you sync it. Saving and syncing are separate permissions, so you may be able to save an edit and need someone else to sync it; your edit is kept either way. With the API, update the source: send PATCH /knowledge/sources/{source_id} with any of name, configuration and status (active or paused). configuration replaces the stored configuration whole, so send every setting, not only the one you changed. Then sync the source.

Security

Source settings are visible to every member of your organisation, so they never hold credentials. Crawls cannot reach private or internal network addresses, and URLs containing a username or password are refused. See Security.

MITHUNAI RAG

Knowledge sources

Answers and assistants

Deploy

Analyze

Connect a website, repository, API schema or help centre

Sitemap

Website

GitHub

GitHub issues, pull requests and discussions

OpenAPI schema

Custom answers

Zendesk Help Center

Discourse forum

Path patterns

Choose the right source

Edit, pause or rename a source

Security

​Sitemap

​Website

​GitHub

​GitHub issues, pull requests and discussions

​OpenAPI schema

​Custom answers

​Zendesk Help Center

​Discourse forum

​Path patterns

​Choose the right source

​Edit, pause or rename a source

​Security

Sitemap

Website

GitHub

GitHub issues, pull requests and discussions

OpenAPI schema

Custom answers

Zendesk Help Center

Discourse forum

Path patterns

Choose the right source

Edit, pause or rename a source

Security