LLM Load Shedding and Overload Control in Production: Adaptive Admission, Preemption Economics, and Graceful Degradation

Standard web services rely on well-established overload protection patterns: reverse proxies monitor CPU utilization, memory thresholds, or static queue depths and reject excess HTTP requests with 429 Too Many Requests or 503 Service Unavailable status codes. When applied naively to Large Language Model (LLM) serving infrastructure, these conventional heuristics fail catastrophically. LLM inference workloads exhibit extreme compute and memory asymmetries that decouple raw server utilization from actual system stability.

Under sudden traffic surges, an unmanaged LLM cluster can remain at 100% GPU utilization while its effective goodput collapses to near zero. Preventing throughput collapse requires a domain-specific overload control architecture that accounts for two-phase execution dynamics, non-deterministic Key-Value (KV) cache memory expansion, preemption penalties, and cascading client timeouts.

The Asymmetric Overload Dynamics of LLM Serving

To understand why conventional load shedding fails, consider the two operational phases of autoregressive transformer inference:

+-----------------------------------------------------------------------------------+
| PREFILL PHASE: Compute-Bound                                                      |
| - Consumes entire input prompt (N tokens) in a single forward pass                |
| - High arithmetic intensity; saturated Tensor Cores                              |
| - Predictable execution time proportional to prompt length                        |
+-----------------------------------------------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------------+
| DECODE PHASE: Memory-Bandwidth-Bound                                              |
| - Generates output tokens autoregressively (1 token per step)                     |
| - Low arithmetic intensity; memory bandwidth bottlenecked                         |
| - Dynamic, non-deterministic duration (1 to max_tokens steps)                      |
| - Continual KV cache expansion: GPU VRAM commitment increases every step         |
+-----------------------------------------------------------------------------------+

1. Dynamic KV Cache Growth as an Unbounded Memory Commitment

When an engine admits an HTTP request containing a 512-token prompt, it initially allocates KV cache blocks for those 512 tokens. However, admitting the request is an implicit credit commitment to support subsequent decoding. If the client requested up to 2,048 output tokens, the request's memory footprint expands by a factor of five over hundreds of decoding iterations. If dozens of concurrent requests expand simultaneously without strict admission limits, the GPU runs out of physical KV cache pages (Kwon et al., 2023).

2. The Sunk-Cost Penalty of Preemption and Goodput Collapse

In standard microservices, aborting an in-flight request discards minimal computational state. In LLM serving, aborting or preempting a request that has already generated 800 of its 1,000 tokens discards the prefill FLOPs and all 800 forward passes already computed. If the engine is forced to recompute the request later, that GPU time is permanently wasted. Under heavy overload, engines that blindly admit requests spend all their GPU compute swapping, preempting, and recomputing abandoned sequences, causing completed requests per second (goodput) to plummet (Kim et al., 2024).

3. Cascading Client Timeouts and Ghost Generations

When request queues swell, the time an incoming prompt sits waiting before prefill increases. If the queue latency exceeds the client application's HTTP socket timeout (e.g., 30 seconds), the client disconnects. If the serving engine does not continuously verify transport liveness, it will eventually dequeue the request, execute a full compute-heavy prefill, run decoding to completion, and attempt to write tokens to a dead connection. The cluster consumes its scarcest compute cycles generating tokens that no client will ever read.

Multi-Tier Admission Control Architecture

Robust overload mitigation requires decoupling edge ingress filtering from engine-level memory scheduling. A production topology uses a two-tier admission control strategy:

[ Incoming Requests ]
         |
         v
+-------------------------------------------------------------------------------+
| LAYER 1: Edge Gateway Admission Control                                       |
| - Tenant Rate Limiting & Token Buckets                                        |
| - Priority Class Tagging (Tier 0: Interactive UI, Tier 1: Agent, Tier 2: Batch) |
| - Client Timeout Deadline Budgeting (Drop if queue_delay > client_deadline)  |
| - Early Rejection with 429/503 + Retry-After                                  |
+-------------------------------------------------------------------------------+
         | (Admitted Requests)
         v
+-------------------------------------------------------------------------------+
| LAYER 2: Engine-Level Proactive Admission Control                             |
| - Physical KV Block Feasibility Checks (PagedAttention)                       |
| - Iteration-Level Token Budgeting (Max Prefill Tokens per Step)               |
| - Dynamic Concurrency Limits based on TTFT / ITL SLOs                         |
| - Preemption Handler (Swap vs. Recompute vs. Truncate)                       |
+-------------------------------------------------------------------------------+
         |
         v
[ GPU Tensor Cores & High-Bandwidth Memory (HBM) ]

Layer 1: Edge Gateway Admission Control

The edge gateway evaluates incoming requests before they touch GPU worker queues:

• Priority Tiering: Requests are tagged by traffic class. Tier 0 (synchronous user-facing chat) receives strict latency guarantees; Tier 1 (interactive agent tool-calling loops) receives standard priority; Tier 2 (asynchronous evaluation, indexing, or batch summarization) is aggressively queued or rejected during peak load.
• Deadline-Aware Dropping: Clients supply a request timeout deadline header (such as X-Request-Deadline-Ms: 5000). If a request sits in the gateway backlog longer than its remaining deadline minus estimated execution time, the gateway immediately drops it with a 504 Gateway Timeout rather than forwarding dead load downstream (AWS Builder Center, 2024).
• Token-Aware Rate Limiting: Rate limiters track estimated prompt token lengths rather than raw HTTP request counts, preventing token-heavy payload bursts from saturating prefill engines.

Layer 2: Engine-Level Proactive Admission Control

Modern serving runtimes (such as vLLM and SGLang) implement iteration-level scheduling. At each forward pass, the scheduler determines how many waiting requests can transition from the waiting queue to the running batch:

• KV Block Watermarks: The scheduler defines high and low watermarks for free KV memory blocks (such as gpu_memory_utilization = 0.90). A new prompt is only scheduled for prefill if free blocks exceed the watermark and can accommodate the prompt tokens plus a safety margin for decoding steps (Kwon et al., 2023).
• Chunked Prefill Throttling: Rather than processing a 16,000-token prompt in a single massive forward pass that starves all concurrent decode streams, the engine splits the prompt into fixed chunks (e.g., 512 or 1,024 tokens) co-scheduled alongside decode iterations (Bari et al., 2025).

Preemption Economics: Swap, Recompute, or Truncate

When aggregate KV cache consumption reaches 100% despite admission controls, the serving engine must preempt in-flight requests to prevent out-of-memory crashes. Three recovery mechanisms exist, each with distinct operational characteristics:

1. Swap-to-Host (Paging)

• Memory Recovery Speed: Moderate (bounded by PCIe bandwidth).
• Compute Overhead: Low (retains all previously computed tokens).
• Latency Impact: High (blocks on host-to-device memory copy when resumed).
• Optimal Use Case: Long prompts with few remaining decode steps.
• Mechanism: The engine migrates KV blocks of selected running sequences from GPU High-Bandwidth Memory (HBM) to CPU host memory over PCIe (Sheng et al., 2023). While this preserves all forward-pass computation, PCIe bus bandwidth (typically 64 GB/s on PCIe Gen 5 x16) creates a significant latency penalty when paging blocks back to GPU memory (which operates at 2.0 to 3.35 TB/s on modern accelerator hardware).

2. Recomputation

• Memory Recovery Speed: Instantaneous.
• Compute Overhead: High (discards all generated tokens; re-prefills later).
• Latency Impact: Moderate for short sequences; high for long sequences.
• Optimal Use Case: Short prompts early in their generation cycle.
• Mechanism: The engine evicts the sequence's KV blocks completely and pushes the request back to the front of the waiting queue. When scheduled again, the engine concatenates the original prompt with all output tokens generated prior to eviction, executing a single prefill pass to repopulate the KV cache (Kim et al., 2024). For short context lengths, recomputation is often faster than waiting on PCIe host-to-device paging.

3. Graceful Truncation

• Memory Recovery Speed: Instantaneous.
• Compute Overhead: Zero (no wasted compute; returns partial valid output).
• Latency Impact: None (terminates immediately).
• Optimal Use Case: Streaming conversational endpoints and non-strict background tasks.
• Mechanism: For conversational or summarization workloads where a partial response provides utility, the engine immediately appends an End-Of-Sequence (EOS) marker, terminates the generation stream with finish_reason: "length", and deallocates the KV cache. This permanently frees VRAM without incurring future recomputation overhead.

Eviction Policies: FCFS vs. SRPT vs. LAS

Standard schedulers use First-Come-First-Served (FCFS) eviction, which drops the most recently admitted sequence. However, research demonstrates that Shortest Remaining Processing Time (SRPT) and Least Attained Service (LAS) policies maximize goodput:

• LAS (Least Attained Service): Preempts sequences that have consumed the fewest total tokens. This protects long-running requests that represent large sunk compute investments, minimizing total wasted GPU cycles.
• SRPT (Shortest Remaining Processing Time): Prioritizes sequences closest to completion, clearing KV cache blocks as rapidly as possible to relieve memory pressure (Bari et al., 2025).

Adaptive Concurrency Limits via CoDel and Latency SLOs

Static concurrency limits (such as hardcoding max_num_seqs = 256) fail because the computational cost per request varies dramatically based on prompt length, prefix cache hits, and generation length. Production clusters employ dynamic concurrency limits based on queuing theory and adaptive feedback control.

       +-------------------------------------------------------------+
       | Measure Current TTFT and ITL P95 over Sliding Window (10s)  |
       +-------------------------------------------------------------+
                                      |
                 +--------------------+--------------------+
                 |                                         |
                 v                                         v
       [ SLO Breach: ITL > 25ms ]              [ Healthy: ITL < 15ms ]
                 |                                         |
                 v                                         v
   +---------------------------+             +---------------------------+
   | Multiplicative Decrease   |             | Additive Increase         |
   | Limit = Limit * 0.85      |             | Limit = Limit + 2         |
   | Trigger Gateway Shedding  |             | Expand Concurrency Pool   |
   +---------------------------+             +---------------------------+

Applying CoDel to LLM Queues

Controlled Delay (CoDel) distinguishes between acceptable temporary queue spikes and persistent, destructive standing queues (Nichols & Jacobson, 2012).

1. The gateway tracks the minimum queue delay experienced by requests over a rolling window (e.g., 5 seconds).
2. If the minimum queue delay exceeds the target threshold (e.g., 200ms) for an entire window, the controller enters shedding mode.
3. In shedding mode, incoming requests are dropped with increasing probability using an Additive-Increase/Multiplicative-Decrease (AIMD) algorithm until the minimum queue delay drops below the target.

Dynamic Concurrency Tuning Based on ITL and TTFT SLOs

Rather than monitoring GPU load, the autoscaling and admission systems monitor two core LLM service level objectives:

• Time-to-First-Token (TTFT): Measures prefill pipeline saturation and queue delay.
• Inter-Token Latency (ITL) / Time-per-Output-Token (TPOT): Measures decode batch saturation and memory bandwidth contention.

If P95 ITL exceeds the target budget (e.g., 30ms per token for real-time streaming), the ingress proxy multiplicatively reduces the maximum concurrent active requests dispatched to that worker replica.

Graceful Quality Degradation Strategies

When traffic spikes exceed total cluster capacity, load shedding should not be binary. Systems can degrade service quality along predictable dimensions before rejecting requests entirely:

[ Extreme Overload Event ]
            |
            v
[ 1. Disable Extended Reasoning / Thinking Tokens ]
  - Switch reasoning models from deep search (budget: 4k tokens) to zero-budget mode
            |
            v
[ 2. Dynamic Model Cascade Fallback ]
  - Route non-critical queries from 70B+ frontier models to 8B distilled models
            |
            v
[ 3. Prompt & Context Compression ]
  - Prune low-relevance RAG chunks and truncate verbose system prompts
            |
            v
[ 4. Output Token Clamping ]
  - Restrict max_tokens from 2,048 to 512 across interactive endpoints
            |
            v
[ 5. Hard Load Shedding ]
  - Drop Tier 2 batch requests, then Tier 1 agent requests; preserve Tier 0 UI

• Reasoning Budget Clamping: For reasoning models that use dynamic test-time compute, the gateway injects headers capping or disabling thinking token budgets, instantly reducing generation lengths by 60% to 80%.
• Model Cascade Fallback: The API gateway automatically redirects low-tier workloads to smaller, higher-throughput models (e.g., routing summarization queries from a 70B parameter model to an 8B distilled model running with tensor parallelism).
• Context and Retrieval Pruning: The RAG retrieval pipeline dynamically reduces the number of retrieved context passages from k=10 to k=3, cutting input prompt token volume and unburdening prefill engines.
• Output Token Capping: Interactive endpoints dynamically lower the default max_tokens ceiling, shortening the average sequence lifecycle and accelerating KV cache turnover.

Client Backpressure and Transport Cancellation

A critical vulnerability in production LLM gateways is the disconnect between HTTP client state and worker execution state. In streaming architectures based on Server-Sent Events (SSE) or gRPC:

[ Client App ]  <---(Broken TCP Socket)--X--  [ API Gateway ]  ======(gRPC Cancel)=====>  [ LLM Worker Engine ]
(User closes tab / leaves page)             (Detects socket close)                        (Aborts generation loop)
                                                                                          (Deallocates KV cache blocks)

• TCP Socket Heartbeats and Disconnect Detection: The API gateway must continuously monitor the client socket for connection resets or closed file descriptors while streaming output tokens.
• Immediate Downstream Cancellation: Upon detecting a client disconnect, the gateway must emit an asynchronous cancellation signal (such as grpc.Status.CANCELLED or an internal abort RPC) to the inference engine.
• KV Cache Deallocation: Upon receiving the abort signal, the serving engine removes the request from the active batch within the current iteration and immediately marks all allocated PagedAttention blocks as free, making them instantly available for waiting prefill requests.
• Prefill-Decode Disaggregation Safety: In disaggregated architectures where prefill runs on dedicated nodes and transfers KV caches over RDMA/InfiniBand to decode nodes (Patel et al., 2024), cancellation signals must propagate across both node tiers to abort in-flight KV tensor transfers.

Production Implementation Checklist

Deploying effective overload protection for LLM services requires validating controls across the full request lifecycle:

• Edge Rate Limiting: Enforce token-budget rate limits rather than raw request-count limits.
• Client Timeout Propagation: Include X-Request-Deadline headers and discard expired queue items before prefill execution.
• Strict Memory Watermarks: Configure runtime memory utilization targets (such as gpu_memory_utilization = 0.90) to reserve head-room for decode bursts.
• Chunked Prefilling: Enable iteration-level chunked prefill to eliminate decode stream starvation during long prompt ingestion.
• LAS and SRPT Scheduling: Use Least Attained Service or Shortest Remaining Processing Time scheduling during memory contention to minimize wasted GPU cycles.
• Adaptive Concurrency Control: Apply CoDel or AIMD feedback loops driven by P95 TTFT and ITL metrics rather than static concurrency caps.
• Automated Degradation Tiers: Implement fallback cascades that reduce thinking budgets, prune RAG context, and clamp output tokens under stress.
• End-to-End Cancellation Propagation: Ensure client HTTP disconnects immediately trigger backend engine aborts and instant KV cache deallocation.

Sources

• AWS Builder Center: Using Load Shedding to Avoid Overload
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)
• The Effect of Scheduling and Preemption on the Efficiency of LLM Inference Serving (Kim et al., 2024)
• Optimal Scheduling Algorithms for LLM Inference: Theory and Practice (Bari et al., 2025)
• Controlling Queue Delay (CoDel) (Nichols & Jacobson, 2012, ACM Queue)
• Splitwise: Efficient Generative LLM Inference Using Phase Splitting (Patel et al., 2024)
• High-Throughput Generative Inference of Large Language Models with FlexGen (Sheng et al., 2023)
• vLLM Engine Scheduling and Preemption Architecture