LLM Gateways and Routing Infrastructure in Production: Comparing LiteLLM, Portkey, Kong AI Gateway, and Cloudflare AI Gateway

In early production architectures, engineering teams frequently integrate Large Language Models (LLMs) by instantiating vendor-specific SDK clients directly within application microservices. While this pattern enables rapid prototyping, it introduces severe architectural bottlenecks at scale: unmitigated upstream provider outages (HTTP 502/503 errors), strict rate limit exhaustion (HTTP 429), uncoordinated token spend across teams, absent audit logging, and tight coupling to proprietary API schemas.

To resolve these operational challenges, production AI infrastructure has converged on the AI Gateway pattern. Siting a dedicated proxy between application services and upstream inference providers centralizes routing, load balancing, dynamic fallbacks, semantic response caching, token-aware rate limiting, and security guardrails.

This analysis evaluates four leading AI gateway architectures deployed in production environments: LiteLLM, Portkey, Kong AI Gateway, and Cloudflare AI Gateway. We examine their underlying data plane engines, benchmarked proxy overhead, semantic caching subsystems, fallback resilience mechanisms, and enterprise governance capabilities.

Architectural Taxonomies and Core Mechanics

AI gateways fall into three distinct operational models based on where and how the data plane executes:

Embedded Application Proxies and Daemon Gateways (LiteLLM, Portkey Core): Lightweight runtime services deployed as standalone microservices or sidecars within a Kubernetes cluster or container mesh. They focus on payload translation, local routing logic, and broad model provider compatibility.
High-Throughput API Gateway Extensions (Kong AI Gateway, Envoy AI Gateway): Native extensions built on top of high-performance C, Rust, or OpenResty/NGINX proxy cores. These gateways treat LLM requests as high-volume API traffic enriched with specialized AI plugins for token tracking, regex prompt guards, and vector database lookups.
Serverless Edge Proxies (Cloudflare AI Gateway): Globally distributed anycast proxy layers executing within lightweight V8 isolates across edge points of presence (PoPs). They terminate client TLS connections close to the user, handle edge caching, and proxy upstream inference without requiring infrastructure management.

AI Gateway Architecture and Routing Pipeline

Deep Architectural Comparison

1. LiteLLM: Python-Native Translation and Open-Source Modularity

LiteLLM is widely adopted as an open-source model gateway due to its extensive protocol translation layer, standardizing over 100 model providers into OpenAI-compatible request and response payloads.

Core Data Plane: Built on Python with FastAPI and AsyncIO. LiteLLM handles schema normalization in memory, mapping provider-specific parameter nuances (such as Anthropic reasoning tokens or AWS Bedrock payload structures) transparently.

Caching Subsystem: Supports multiple caching backends:

In-memory and local disk caches for single-instance development.
Redis and Valkey exact-match caching using SHA-256 hashes of serialized request payloads.
Native semantic caching via redis-semantic (requiring Redis Stack with RediSearch) and qdrant-semantic. The gateway computes text embeddings over input prompt arrays and checks cosine similarity against stored vectors using a configurable similarity threshold.

Routing and Load Balancing: The LiteLLM Router provides RPM (requests per minute) and TPM (tokens per minute) tracking, health check cooldowns for failed endpoints, weighted routing, and dynamic fallback arrays.

Operational Constraints: Because the data plane runs in Python, it exhibits higher baseline CPU consumption and serialization latency compared to compiled reverse proxies. In high-concurrency environments handling thousands of requests per second, horizontal scaling requires careful PostgreSQL connection pool tuning to prevent database lock contention on user and key accounting tables.

2. Portkey: TypeScript Edge Core and Enterprise Orchestration

Portkey AI Gateway operates as an open-source, ultra-lightweight TypeScript/Node.js gateway that can be self-hosted in Docker, deployed as an edge worker (Cloudflare Workers, AWS Lambda), or consumed via a managed SaaS control plane.

Core Data Plane: A fast TypeScript engine that normalizes requests across 250+ model providers while maintaining a minimal binary footprint and sub-10ms execution overhead in local runtimes.

Routing and Resilience: Portkey relies on declarative JSON configurations that define complex routing graphs:

Error-matched fallback chains that switch models or providers upon receiving specific upstream status codes (e.g., falling back to Claude 3.5 Sonnet if GPT-4o returns a 429 or 500).
Automatic retries with exponential backoff up to 5 attempts.
Weighted canary deployments and conditional routing rules based on request metadata or user tiers.

Guardrails and Governance: Portkey integrates over 50 guardrails directly into the request-response lifecycle. These include regex filters, PII detection and masking, toxicity checks, and external webhook validators. It also provides virtual API keys with hard budget limits and real-time cost attribution.

Operational Constraints: While the gateway core is open-source, advanced enterprise features such as deep log analytics dashboards, automated semantic cache management, and fine-grained team access controls are coupled to Portkey's managed control plane.

3. Kong AI Gateway: Sub-Millisecond C/OpenResty Proxying

Kong AI Gateway extends Kong's battle-tested OpenResty/NGINX data plane with a suite of AI-specific Lua plugins, designed for massive enterprise throughput.

Core Data Plane: High-performance C and Lua core. The data plane separates entirely from the Konnect or Kubernetes control plane, running stateless proxy workers that handle TLS termination, request validation, and routing with sub-millisecond overhead.

AI Plugin Ecosystem:

ai-proxy and ai-proxy-advanced: Handle model format conversion, multi-provider routing, and semantic model selection.
ai-semantic-cache: Connects directly to Redis Vector Search to calculate cosine similarity on vector embeddings, returning cached completions before touching upstream networks.
ai-prompt-guard and ai-semantic-prompt-guard: Execute PCRE regular expressions and vector similarity checks against disallowed prompt lists at the proxy level.
ai-rate-limiting-advanced: Enforces rate limits based on exact token usage extracted from stream events, rather than raw HTTP request counts.

Operational Constraints: Kong requires operational familiarity with API gateway deployments, declarative deck configurations, or Kubernetes Ingress controllers. For teams seeking a simple plug-and-play SDK wrapper, Kong presents a steeper setup and infrastructure curve.

4. Cloudflare AI Gateway: Zero-Ops Edge Proxying and Unified Analytics

Cloudflare AI Gateway runs as a globally distributed reverse proxy on Cloudflare's anycast edge network across 300+ locations.

Core Data Plane: Serverless V8 isolates running directly on Cloudflare Workers. Integration requires only modifying the base URL of existing SDK calls to route through Cloudflare's universal gateway endpoint.

Key Features:

Edge Response Caching: Serves exact-match cached completions directly from Cloudflare's key-value edge cache, reducing latency and upstream token costs to zero for duplicate requests.
Dynamic Visual Routing: Provides a visual builder for traffic splitting, A/B testing, and automated fallbacks across 20+ supported AI providers.
Data Loss Prevention (DLP) and Rate Limiting: Inspects outbound payloads for sensitive data patterns and throttles abusive client keys before requests traverse the public internet.

Operational Constraints: Cloudflare operates a closed-source control plane. Routing heuristics and internal cache mechanics are managed entirely by Cloudflare, offering limited visibility into low-level failover state machines. Furthermore, semantic vector caching is not natively integrated at the edge, restricting caching benefits to exact-match queries.

Performance and Latency Benchmarks

Evaluating gateway overhead is critical: while an LLM inference call may take anywhere from 500ms to 10 seconds, the gateway itself must introduce negligible latency to avoid degrading Time-To-First-Token (TTFT) and Inter-Token Latency (ITL).

In controlled benchmark testing on Amazon EKS using WireMock to simulate uniform LLM endpoints under identical 12-CPU resource allocations:

Throughput: Kong AI Gateway data planes achieved over 228% higher throughput than Portkey and over 859% higher throughput than LiteLLM under high concurrent load (400 virtual users generating 1,000 prompt tokens each).
Latency Overhead: Kong demonstrated 65% lower proxy latency than Portkey and 86% lower proxy latency than LiteLLM at p95 and p99 percentiles.
Edge Routing Considerations: Cloudflare AI Gateway introduces between 20ms and 60ms of network proxy hop latency depending on client location, but mitigates this on cache hits by serving responses directly from the nearest edge PoP in under 15ms.

Semantic Caching Mechanics: Exact vs Vector Similarity

Caching LLM responses differs fundamentally from traditional HTTP caching due to the non-deterministic and natural-language nature of prompt inputs.

Exact-Match Caching:

Hashing: Computes SHA-256 over (model, messages, temperature, top_p, tools).
Overhead: Less than 1ms lookup in Redis, Valkey, or Cloudflare KV.
Limitation: Minor prompt variations ("Summarize this article" vs "Please summarize this article") cause a total cache miss.

Semantic Similarity Caching:

Pipeline: The gateway intercepts the prompt, calls an embedding model (e.g., text-embedding-3-small or a local embedding service), and queries a vector index (Redis Vector Search, Qdrant, or Milvus) using Approximate Nearest Neighbor (ANN) search.
Thresholding: If the cosine similarity exceeds a configured threshold (typically 0.85 to 0.95), the cached completion is returned.
Tradeoff: The embedding generation introduces 15ms to 50ms of latency and nominal embedding API cost. However, in repetitive customer support or enterprise search workloads, Redis benchmarks indicate cost reductions of up to 73%.

Architectural Selection Criteria

Selecting the right AI gateway depends on team scale, deployment topology, and compliance requirements:

Select LiteLLM if: You require a 100% open-source, self-hosted proxy with the broadest out-of-the-box model translation layer, native Qdrant/Redis semantic caching, and simple Python-based extensibility.
Select Portkey if: You need deep end-to-end tracing, built-in input/output guardrails, declarative JSON routing with automatic exponential retries, and the option to run on edge runtimes or managed cloud infrastructure.
Select Kong AI Gateway if: Your organization already operates high-throughput Kubernetes API infrastructure, requires enterprise mTLS/RBAC governance, or demands the lowest possible proxy latency and maximum throughput under extreme concurrency.
Select Cloudflare AI Gateway if: You prefer a zero-infrastructure, fully managed edge proxy that integrates immediately with existing web applications, providing instant analytics, DLP, and exact-match caching with minimal configuration.