Skip to main content
The MithunAI Gateway is a high-throughput proxy layer that terminates inference requests, evaluates semantic cache hits under 45ms, and shields language models from adversarial prompt injection. Applications route their inference calls through the Gateway endpoint rather than calling upstream LLM providers directly.

​
Core Capabilities

Sub-45ms Semantic Cache

Evaluates prompt cosine similarity in pgvector memory before invoking model inference. Cached turns return immediately with zero model latency and zero token cost.

Cryptographic Prompt Fencing

Wraps user inputs in perimeter fences with randomized nonces. Instructions are strictly parsed as passive text, preventing instruction overrides and prompt injection.

Multi-Model Failover

Automatically routes requests to configured fallback providers if the primary LLM provider suffers rate limits, high latency, or API downtime.

Tenant-Scoped Rate Limiting

Enforces per-organization token and request quotas at the HTTP edge, preventing runaway billing and resource starvation.

​
Gateway Configuration

Route requests to the Gateway by updating your base URL in client libraries:

​
Invariants

  • Zero Outbound Telemetry: In sovereign clusters, all inference logs and cache embeddings remain within your VPC boundaries.
  • Fail-Closed Execution: If prompt fencing detects suspicious instruction injection patterns, the Gateway declines the turn with an explicit security alert.
Last modified on September 27, 2026