> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mithunai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Supported Knowledge Connectors & Data Sources

> Connect code repositories, websites, cloud storage, documentation spaces, and support platforms to your MITHUNAI knowledge base with automated delta synchronization.

MITHUNAI connectors continuously synchronize documents, codebases, and ticketing archives from your existing tools into partitioned pgvector collections without manual data export. Each connector tracks document versions, executes incremental delta updates, and applies automated secret redaction before text chunking and vector indexing.

```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
flowchart LR
    subgraph Sources["Enterprise Sources"]
        GH["GitHub / GitLab\n(Repos & PRs)"]
        DOCS["Websites & Sitemaps\n(HTML & Markdown)"]
        WKS["Notion / Confluence\n(Docs & Wikis)"]
        CLOUD["Google Drive / S3\n(PDFs & Office)"]
        SUP["Zendesk / Jira\n(Tickets & Articles)"]
    end

    subgraph Pipeline["Ingestion Pipeline"]
        ING["Universal Ingestor"]
        RED["Secret & PII Redactor"]
        CHK["AST & Semantic Chunker"]
        VEC["Hybrid Embedder\n(Dense + BM25)"]
    end

    subgraph Store["Knowledge Store"]
        PG[("pgvector Partition\nTenant-Isolated")]
    end

    Sources --> ING --> RED --> CHK --> VEC --> PG
```

## Available Connectors

MITHUNAI provides native connectors across six enterprise data categories:

<Columns cols={2}>
  <Card title="Code Repositories" icon="github">
    **GitHub & GitLab**: Ingest source code, READMEs, Markdown documentation, and pull request
    discussions with branch filtering and file glob rules.
  </Card>

  <Card title="Documentation & Websites" icon="globe">
    **Web Crawler & Sitemap Parser**: Automated crawling of public documentation, developer portals,
    and marketing domains with same-origin fencing.
  </Card>

  <Card title="Workspace Knowledge Bases" icon="book-bookmark">
    **Notion & Confluence**: Enterprise OAuth integrations syncing spaces, pages, databases, and
    structured blocks with hierarchy preservation.
  </Card>

  <Card title="Cloud Object Storage" icon="cloud">
    **Google Drive & AWS S3**: Real-time webhook and S3 event-driven synchronization for PDFs, Word
    (.docx) files, and technical whitepapers.
  </Card>

  <Card title="Support & Ticketing" icon="headset">
    **Zendesk, Jira & Linear**: Ingest resolved customer support tickets, help center articles, and
    bug tracker threads for agent deflection.
  </Card>

  <Card title="Team Communication" icon="message-square">
    **Slack & Discord**: Index public technical support channels and knowledge-sharing threads with
    noise filtering and author attribution.
  </Card>
</Columns>

***

## Code Repositories: GitHub & GitLab

Repository connectors allow MITHUNAI to answer technical implementation questions, explain API signatures, and cross-reference documentation against live code.

### Configuration Parameters

| Parameter       | Type      | Required | Description                                                                                |
| :-------------- | :-------- | :------- | :----------------------------------------------------------------------------------------- |
| `repository`    | string    | Yes      | Fully qualified repository name (e.g., `acme/platform-core`).                              |
| `branch`        | string    | Yes      | Target branch to index (e.g., `main`, `release/v2`).                                       |
| `include_globs` | string\[] | No       | File matching patterns (e.g., `["docs/**", "**/*.md", "src/**/*.ts"]`).                    |
| `exclude_globs` | string\[] | No       | Exclusion patterns (e.g., `["**/node_modules/**", "**/*.test.ts"]`).                       |
| `sync_on_push`  | boolean   | No       | Automatically trigger delta re-indexing upon GitHub webhook push events (default: `true`). |

```bash Connect via API theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://app.mithunai.com/arukz/api/v1/knowledge/sources \
  -H "Authorization: Bearer $MITHUNAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "collection_id": "col_prod_01",
    "type": "github",
    "name": "Platform Core Repo",
    "config": {
      "repository": "acme/platform-core",
      "branch": "main",
      "include_globs": ["docs/**/*.md", "api/**/*.py"]
    }
  }'
```

***

## Web Crawling & Sitemap Connector

The sitemap and web crawler connector indexes public documentation portals, API reference catalogs, and developer guides.

### Crawl Policy & Guardrails

* **Same-Origin Fencing**: The crawler never follows hyperlinks pointing outside the declared domain or path prefix.
* **Canonical URL Deduplication**: Pages declaring `rel="canonical"` pointing to another indexed document are automatically deduplicated.
* **Robots.txt Adherence**: The crawler honors `Disallow` rules and crawl-delay directives declared by the target server.
* **Dynamic JavaScript Rendering**: Headless Chromium execution ensures single-page applications (React, Vue, Next.js) render complete DOM trees prior to content extraction.

***

## Workspace Knowledge Bases: Confluence & Notion

Confluence spaces and Notion workspaces contain institutional memory, architecture decision records (ADRs), and internal runbooks.

### Hierarchy & Table Preservation

* **Breadcrumb Navigation**: Every indexed chunk retains its complete workspace breadcrumb path (e.g., `Engineering > Infrastructure > Disaster Recovery`).
* **Markdown Table Extraction**: Multi-column tables are extracted into GitHub-flavored Markdown tables to preserve cell relationships during vector search.
* **Access Control Mapping**: Notion and Confluence access permissions can be mapped directly to MITHUNAI collection access roles.

***

## Delta Synchronization & Webhook Triggers

MITHUNAI minimizes computational overhead and API rate limits by evaluating document content hashes (`SHA-256`) before re-embedding.

```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
sequenceDiagram
    autonumber
    participant Ext as Upstream Source (GitHub / Notion)
    participant WH as MITHUNAI Webhook Ingestion
    participant Worker as Background Ingestion Worker
    participant DB as pgvector & Relational DB

    Ext->>WH: Webhook event (push / document.updated)
    WH->>Worker: Enqueue sync job with source ID
    Worker->>Ext: Fetch modified document metadata & content
    Worker->>Worker: Compute SHA-256 hash
    alt Hash matches stored record
        Worker->>DB: Update last_checked_at timestamp (skip embedding)
    else Hash differs (content changed)
        Worker->>Worker: Parse AST / Markdown & chunk
        Worker->>Worker: Generate dense & sparse embeddings
        Worker->>DB: Upsert chunks & update collection index
    end
    Worker-->>WH: Job completed telemetry emitted
```