> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mithunai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# MithunAI Inference Gateway

> High-throughput reverse proxy with sub-45ms semantic caching, automated prompt fencing, and resilient multi-provider model routing.

The MithunAI Gateway is a high-throughput proxy layer that terminates inference requests, evaluates semantic cache hits under 45ms, and shields language models from adversarial prompt injection.

Applications route their inference calls through the Gateway endpoint rather than calling upstream LLM providers directly.

```
Client Application ──> [ MithunAI Gateway ] ─── (Cache Hit: < 45ms) ───> Instant Verified Answer
                              │
                      (Cache Miss / Cold)
                              ▼
                       [ Prompt Fence ]
                              ▼
                [ Multi-Provider Model Router ] ──> (Anthropic / OpenAI / Bedrock)
```

## Core Capabilities

<Columns cols={2}>
  <Card title="Sub-45ms Semantic Cache" icon="gauge">
    Evaluates prompt cosine similarity in pgvector memory before invoking model inference. Cached
    turns return immediately with zero model latency and zero token cost.
  </Card>

  <Card title="Cryptographic Prompt Fencing" icon="shield-check">
    Wraps user inputs in perimeter fences with randomized nonces. Instructions are strictly parsed
    as passive text, preventing instruction overrides and prompt injection.
  </Card>

  <Card title="Multi-Model Failover" icon="shuffle">
    Automatically routes requests to configured fallback providers if the primary LLM provider
    suffers rate limits, high latency, or API downtime.
  </Card>

  <Card title="Tenant-Scoped Rate Limiting" icon="sliders">
    Enforces per-organization token and request quotas at the HTTP edge, preventing runaway billing
    and resource starvation.
  </Card>
</Columns>

## Gateway Configuration

Route requests to the Gateway by updating your base URL in client libraries:

<CodeGroup>
  ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
  curl -X POST "https://app.mithunai.com/arukz/api/v1/gateway/chat" \
    -H "Authorization: Bearer mth_live_..." \
    -H "Content-Type: application/json" \
    -H "x-mithunai-tenant-id: org_sovereign_770" \
    -d '{
      "model": "claude-3-5-sonnet",
      "messages": [
        {"role": "user", "content": "Explain tenant boundaries in MithunAI"}
      ],
      "semantic_cache": true,
      "temperature": 0.0
    }'
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
  from mithunai import MithunClient

  client = MithunClient(
      api_key="mth_live_...",
      tenant_id="org_sovereign_770"
  )

  response = client.gateway.chat(
      model="claude-3-5-sonnet",
      messages=[{"role": "user", "content": "Explain tenant boundaries in MithunAI"}],
      semantic_cache=True
  )

  print(response.content)
  print(f"Cache Hit: {response.cache_hit} ({response.latency_ms}ms)")
  ```
</CodeGroup>

## Invariants

* **Zero Outbound Telemetry**: In sovereign clusters, all inference logs and cache embeddings remain within your VPC boundaries.
* **Fail-Closed Execution**: If prompt fencing detects suspicious instruction injection patterns, the Gateway declines the turn with an explicit security alert.