LLM Gateways in Production: Comparing LiteLLM, Portkey, Kong AI Gateway, and Cloudflare AI Gateway Architecture, Fallback Cascades, and Serving Economics

As enterprise production architectures scale from single prototype endpoints to multi-model agentic pipelines, coupling application code directly to foundation model provider APIs creates severe operational bottlenecks. Direct client integration leads to fractured telemetry, credential sprawl across microservices, unhandled upstream rate limits, and vendor lock-in.

To isolate application logic from downstream provider volatility, engineering teams deploy dedicated LLM gateways. These specialized reverse proxies sit between client applications and AI inference endpoints, providing protocol normalization, intelligent routing, fallback cascades, semantic caching, rate limiting, cost tracking, and security guardrails.

This analysis evaluates the architecture, data plane mechanics, streaming protocol handling, and serving economics across four leading solutions: LiteLLM, Portkey, Kong AI Gateway, and Cloudflare AI Gateway.

Core Engine and Data Plane Architectures

The architectural foundation of an LLM gateway dictates its latency profile, deployment footprint, memory scaling characteristics, and operational overhead.

1. LiteLLM: Model-Centric Abstraction Layer

• Data Plane Engine: Python, FastAPI, and AsyncIO event loops.
• State Storage: Redis for distributed caching and rate limits; PostgreSQL via Prisma ORM for virtual key management and spend ledgers.
• Proxy Latency Overhead: 15 ms to 40 ms under typical production workloads.
• Deployment Model: Self-hosted Docker container or Python package within private VPCs.
• Architectural Focus: LiteLLM treats the model request as the primary object to normalize, meter, and govern. It translates more than 100 model provider formats into standard OpenAI-compatible REST and streaming payloads. It is the most common self-hosted choice for engineering teams requiring full data plane custody.

2. Portkey: AI Operations and Observability Gateway

• Data Plane Engine: High-throughput edge runtime built on TypeScript and Rust.
• State Storage: Multi-tenant managed cloud storage with distributed Redis backends.
• Proxy Latency Overhead: 10 ms to 25 ms for request routing and trace-assembly.
• Deployment Model: Managed enterprise SaaS with optional open-source container binaries for self-hosting.
• Architectural Focus: Portkey approaches LLM traffic as an AI operations lifecycle. Its gateway runtime handles request routing, retries, fallbacks, and parameter mapping, while emitting granular trace spans to its observability platform. Routing topologies and fallback ladders are declared via declarative JSON and YAML configurations.

3. Kong AI Gateway: Enterprise API Management Plane

• Data Plane Engine: Nginx, OpenResty, C, and LuaJIT hooks, with Envoy filter extensions.
• State Storage: Redis and in-memory shared dictionaries with declarative DB-less syncing.
• Proxy Latency Overhead: 1 ms to 3 ms at scale.
• Deployment Model: High-performance Kubernetes ingress/mesh data planes (EKS, GKE) paired with centralized Kong Konnect control planes.
• Architectural Focus: Kong treats AI traffic as an extension of standard API management. By running AI plugins directly inside OpenResty C and Lua execution paths, Kong achieves sub-millisecond proxy overhead. It enables enterprises with existing API gateway infrastructure to govern LLM traffic without introducing an additional proxy layer.

4. Cloudflare AI Gateway: Distributed Edge Reverse Proxy

• Data Plane Engine: V8 Isolates running across Cloudflare's global Anycast edge network.
• State Storage: Cloudflare KV, Vectorize, and D1 globally distributed databases.
• Proxy Latency Overhead: 5 ms to 15 ms at the edge hop.
• Deployment Model: Fully managed edge service requiring zero compute provisioning.
• Architectural Focus: Cloudflare approaches LLM calls as edge traffic to cache, route, and protect. Clients change the base URL to route through Cloudflare Anycast IPs. It provides edge caching and analytics out of the box, though it requires downstream requests to transit Cloudflare infrastructure.

Streaming Protocol Normalization and Chunk Handling

A core responsibility of an LLM gateway is streaming protocol normalization. Foundation model providers implement divergent Server-Sent Events (SSE) schemas, chunk structures, and final usage reporting conventions.

SSE Event Format Discrepancies

• OpenAI and vLLM: Emits raw data: {"choices": [{"delta": {"content": "..."}}]} events, terminating with data: [DONE]. Token usage metadata is omitted from streaming deltas unless explicitly requested via stream_options: {"include_usage": true}.
• Anthropic Claude: Emits typed event envelopes (event: message_start, event: content_block_delta, event: message_delta, event: message_stop). Text deltas reside in delta.text, while input token counts arrive in message_start and output token counts arrive in message_delta.
• Google Gemini: Uses streamGenerateContent returning chunked JSON arrays containing candidates[0].content.parts[0].text alongside cumulative usageMetadata.

Buffer Management and TTFT Preservation

When translating streaming protocols, naive gateways risk buffering SSE chunks, which artificially inflates Time to First Token (TTFT) and degrades interactive UX.

High-performance gateways implement zero-copy streaming pipelines:

1. Parse incoming SSE chunks incrementally without full JSON payload deserialization where possible.
2. Translate provider-specific field keys into standard OpenAI-compatible delta frames.
3. Immediately flush byte buffers to the downstream client socket without waiting for token batch thresholds.
4. Concurrently accumulate token counts in an asynchronous sidecar worker to record telemetry and update budget ledgers upon stream termination (message_stop or [DONE]).

High-Availability Fallback Cascades and Routing

Downstream foundation model APIs regularly experience localized degradations, HTTP 429 rate limit rejections, HTTP 503 capacity exhaustion, and HTTP 529 overload errors. Gateways mitigate these failures through active traffic routing policies.

Client Request
      │
      ▼
[ LLM Gateway Data Plane ]
      │
      ├── (1) Primary Route (e.g., Anthropic Claude 3.7 Sonnet)
      │         │
      │         └───► [ 429 Rate Limit / 529 Overload ]
      │                     │
      │                     ▼ (Trigger Failover)
      ├── (2) Fallback Route A (e.g., OpenAI GPT-4.5)
      │         │
      │         └───► [ Timeout > 5000ms ]
      │                     │
      │                     ▼ (Trigger Failover)
      └── (3) Fallback Route B (e.g., DeepSeek-V3 on vLLM Cluster)
                │
                └───► [ HTTP 200 OK ] ──► Stream Response to Client

Fallback Mechanics

Gateways implement configurable fallback sequences:

• Status Code Triggers: Automatic failover is initiated on specific HTTP error codes (408, 429, 500, 502, 503, 504, 529) or connection timeouts.
• Exponential Backoff with Jitter: To prevent stampeding herd problems on rate-limited endpoints, retry intervals compute backoff using full jitter: t_sleep = min(t_max, t_base * 2^attempt) * Uniform(0.5, 1.5).
• Dynamic Cooldowns and Circuit Breakers: When a provider endpoint breaches an error rate threshold (such as 10 consecutive failures), the gateway marks the target unhealthy, enters a half-open state, and bypasses it for a configurable cooldown window (for example, 60 seconds) before sending canary probe requests.
• Hedging Strategies: For latency-critical workflows, gateways can dispatch concurrent requests to two distinct providers after a P95 latency threshold, returning the fastest response and cancelling the trailing connection.

Multi-Tier Caching and Prompt Cache Pass-Through

LLM inference queries exhibit high prompt redundancy, particularly in multi-turn agentic workflows where system instructions and tool definitions are repeatedly transmitted. Gateways implement multiple layers of cache optimization.

1. Exact-Match Key-Value Caching

The gateway computes a deterministic hash (SHA-256) of the normalized request parameters: Cache_Key = SHA256(Model || Messages_JSON || Temperature || Top_P || Tools_Schema)

If an identical request arrives before TTL expiration, the gateway returns the cached response with zero downstream provider calls, reducing latency to under 5 ms and cost to zero.

2. Semantic Vector Caching

Semantic caching embeds the incoming prompt using a dense vector embedding model and queries a vector database (such as Redis Vector Search, Cloudflare Vectorize, or Qdrant). If the cosine similarity between the incoming query and a previously answered query exceeds a strict threshold (such as similarity > 0.96), the cached completion is returned.

Operational Caveat: Semantic caching introduces embedding generation latency (15 ms to 50 ms) and requires careful scope isolation to prevent leaking user-specific context across sessions.

3. Native Prompt Cache Header Pass-Through

Modern model providers (Anthropic, OpenAI, Google Gemini) offer native prompt caching on their inference clusters, cutting input token pricing by 50% to 80% on cached prefixes.

Gateways must preserve prefix ordering and forward appropriate provider headers:

• For Anthropic: Formatting system and context blocks with cache_control: {"type": "ephemeral"} breakpoints.
• For OpenAI: Ensuring shared system prompt prefixes remain byte-identical across requests to hit automatic 1024-token boundary cache clusters.

Governance, Virtual Keys, and Budget Enforcement

In multi-team enterprise environments, direct API key distribution poses severe cost and security risks. LLM gateways centralize credential management behind virtual keys.

Virtual Key Mapping

Applications authenticate to the gateway using internal virtual API tokens (sk-gw-...). The gateway validates the token against its metadata database and injects the actual downstream provider API key before forwarding the request. Master provider credentials remain strictly contained within the gateway environment.

Rate Limiting and Token Bucket Enforcement

Gateways enforce multi-tier rate limits:

• Requests Per Minute (RPM): Governed via sliding window counters in Redis.
• Tokens Per Minute (TPM): Estimated on ingress via lightweight BPE tokenizers (such as tiktoken or Hugging Face fast tokenizers) and reconciled post-response using exact provider usage payloads.

Budget Pools and Hard Caps

Gateways assign virtual keys to organizational entities (teams, projects, cost centers). Spend is tracked against predefined budgets (daily, weekly, monthly). When a team reaches 80% of its budget, soft warnings trigger webhook alerts; upon reaching 100%, the gateway immediately rejects subsequent requests with HTTP 429 and custom budget error schemas, preventing runaway spend.

Performance Benchmarks and Latency Economics

Introducing a proxy hop adds network and processing overhead. Choosing the right gateway requires balancing feature richness against data plane latency.

According to independent enterprise benchmarks published by Kong Engineering and comparative analyses by ToolJunction, data plane overhead varies substantially by runtime:

• Kong AI Gateway: Utilizing C and Lua within OpenResty, Kong achieves sub-3 ms latency overhead at 20,000+ RPS under identical hardware allocations, with P95 latency remaining near baseline network limits.
• Cloudflare AI Gateway: Leverages V8 Isolates at the edge, introducing approximately 5 ms to 15 ms of edge processing overhead, often offset by edge caching when user geographic distribution matches Cloudflare edge nodes.
• Portkey: Edge and container proxies introduce 10 ms to 25 ms of routing and trace-assembly overhead.
• LiteLLM: Python AsyncIO execution introduces 15 ms to 40 ms of serialization and framework overhead per request under moderate load.

Latency vs. Throughput Trade-Off

For conversational AI and interactive human chat where downstream model generation takes 2,000 ms to 8,000 ms, a 20 ms gateway overhead is negligible (accounting for less than 1% of total response time).

However, in high-frequency automated agent loops and real-time voice pipelines where sub-200 ms Time to First Token (TTFT) is required, sub-millisecond proxy architectures like Kong or native edge runtimes become critical.

Architectural Decision Framework

When to Choose Kong AI Gateway

• Your organization already operates a Kong API Gateway mesh or Kubernetes ingress controller.
• Sub-millisecond data plane latency and maximum throughput (20,000+ RPS) are strict architectural requirements.
• You require unified enterprise security policies (OAuth2, OIDC, mTLS, WAF) covering both standard microservices and LLM endpoints.

When to Choose LiteLLM

• You need rapid self-hosted deployment inside a private VPC with zero third-party data egress.
• Your engineering stack is Python-centric and values direct integration with native OpenAI SDKs.
• You require broad model format translation across 100+ open-weight and proprietary providers.

When to Choose Portkey

• You require an end-to-end AI operations platform with visual routing trees, deep trace observability, and prompt management.
• You need configurable guardrail pipelines and dynamic model failover ladders without writing custom routing logic.
• Managed enterprise compliance and multi-tenant billing analytics are primary drivers.

When to Choose Cloudflare AI Gateway

• You seek a zero-infrastructure managed edge proxy with instant global deployment.
• Your user base is geographically distributed and benefits from Anycast edge caching and rate limiting.
• You want turn-key caching and request logging without operating databases or clusters.

By centralizing multi-provider fallbacks, streaming normalization, budget enforcement, and semantic caching within a dedicated gateway layer, engineering teams build resilient, vendor-agnostic foundation model architectures capable of sustaining production workloads at scale.

Sources

• Kong Engineering: AI Gateway Benchmark: Kong AI Gateway, Portkey, and LiteLLM
• LiteLLM Documentation: Architecture, Load Balancing, and Proxy Configuration
• Portkey Documentation: AI Gateway Architecture and Fallback Routing
• Kong AI Gateway: Product Overview and AI Plugins
• Cloudflare Developers: AI Gateway Overview and Features
• OpenTelemetry: Semantic Conventions for Generative AI Operations
• ToolJunction: AI Gateway Comparison: Portkey vs LiteLLM vs Kong vs Cloudflare