LLM Streaming in Production: Server-Sent Events vs. WebSockets, Incomplete JSON Parsing, and Backpressure Architecture

Token streaming is the primary user experience mechanism in generative AI applications. By piping generated tokens to the client as they leave the transformer layers rather than awaiting full sequence completion, systems dramatically reduce perceived latency from tens of seconds to sub-second time to first token (TTFT).

However, moving from batch request-response models to long-lived streaming pipelines introduces brittle failure modes across the networking stack. In production, engineering teams regularly grapple with reverse proxy buffer deadlocks, connection starvation across browser tabs, runaway compute billing when clients disconnect mid-generation, quadratic parsing degradation during structured output streaming, and client-side DOM throttling under high token throughput.

Designing a robust streaming architecture requires understanding transport protocol trade-offs, edge proxy behaviors, and incremental parsing state machines.

LLM Streaming Architecture in Production

Transport Protocol Comparison: SSE vs. WebSockets vs. HTTP/2 Chunked Transfer

Large language model providers (including OpenAI, Anthropic, and open-source serving engines like vLLM and SGLang) overwhelmingly standardize on Server-Sent Events (SSE) over raw HTTP/2 chunked transfer or WebSockets. The protocol selection involves distinct trade-offs across simplicity, connection overhead, and bidirectional capabilities.

Server-Sent Events (SSE)

SSE transmits structured UTF-8 text over standard HTTP connections using the text/event-stream MIME type. Each message is framed with field prefixes (data:, event:, id:, retry:) terminated by double newlines (\n\n).

Advantages of SSE for LLM serving:

Stateless HTTP Infrastructure: SSE relies on standard HTTP requests (typically HTTP POST carrying the generation payload, followed by a streaming response). It natively traverses enterprise firewalls, API gateways, load balancers, and monitoring proxies without protocol upgrades.
Low Connection Overhead: As detailed in architectural comparisons by Tianpan, maintaining idle SSE connections incurs negligible state overhead compared to stateful WebSocket session handlers that require persistent heartbeat ping/pong loops and session-pinning backplanes.
Built-in Event Demultiplexing: Named event types allow backend systems to multiplex token deltas, tool-call fragments, reasoning traces, and metadata across a single stream.

The historical drawback of SSE was the browser connection limit under HTTP/1.1, where browsers strictly capped concurrent connections to 6 per domain. Opening multiple browser tabs with active SSE streams quickly blocked subsequent HTTP requests. Under HTTP/2 multiplexing (RFC 9113), multiple SSE streams run over a single TCP connection within independent stream frames, raising the concurrent limit to default server negotiated limits (typically 100 to 250 streams).

WebSockets

WebSockets establish a persistent, bidirectional, full-duplex TCP connection initiated via an HTTP 101 Switching Protocols handshake.

While WebSockets avoid HTTP header repetition for high-frequency client-to-server messaging, they introduce operational complexity for standard LLM inference:

Stateful Connection Management: WebSockets require sticky load balancer sessions or a distributed synchronization backplane (such as Redis Pub/Sub) to route messages across horizontally scaled workers.
No Native Request-Response Semantics: Application layers must manually implement request IDs, correlation maps, and timeout handlers.
When WebSockets Are Justified: WebSockets become necessary when the interaction pattern requires continuous bidirectional streaming. Key use cases include real-time voice-to-voice agents (piping raw audio chunks back and forth over WebRTC/WebSockets), client-side barge-in interruption, and multi-user collaborative canvas editing.

Raw Chunked Transfer Encoding

Raw HTTP/1.1 chunked transfer or HTTP/2 DATA frames transmit raw bytes without event framing. While lightweight, raw streams lack standard event metadata framing, making it difficult to cleanly distinguish between raw text tokens, structured tool calls, usage statistics, and stream termination signals without custom framing protocols.

Reverse Proxy and CDN Edge Pitfalls

The most frequent production failure mode in SSE deployment occurs at the intermediate reverse proxy layer (Nginx, Envoy, Cloudflare, AWS CloudFront).

Reverse Proxy Response Buffering

By default, reverse proxies buffer upstream HTTP responses until receiving a Content-Length header, a chunked transfer completion marker, or until internal memory buffers (such as 4KB/8KB buffers) fill up. Because token streaming produces small text chunks (often 4 to 30 bytes per token), an unconfigured proxy will withhold tokens until the entire generation completes or buffer thresholds are met. The end-user experiences zero perceived streaming, followed by a sudden burst of text.

To disable buffering across standard proxy infrastructure:

Nginx Header Configuration: The application must emit the X-Accel-Buffering: no response header, or the proxy location block must explicitly disable buffering:

location /api/v1/chat/completions {
    proxy_pass http://llm_backend;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding off;
}

Standard Cache-Control Headers: The application response must send:

Content-Type: text/event-stream
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no

The no-transform directive prevents CDNs and compression middleboxes from applying Gzip/Brotli buffering on active streams.

Keep-Alive Heartbeats: During long prefill phases (processing 100K+ context tokens) or reasoning model deliberation cycles, the model may take 5 to 30 seconds before outputting the first token. To prevent proxy gateways from triggering 504 Gateway Timeouts, backends must send periodic SSE comment frames (: ping\n\n or : keep-alive\n\n), which are ignored by SSE client parsers but keep intermediate TCP sockets active.

Disconnect Propagation: Preventing Runaway GPU Token Burn

In an LLM serving cluster, generative decoding is memory-bandwidth bound and compute-expensive. If an end-user navigates away, closes their browser tab, or clicks "Stop Generating", the client aborts the HTTP connection via AbortController.

If the server does not actively monitor connection termination, the backend engine continues generating tokens through the full completion budget, consuming high-demand GPU SRAM and tensor cores for output that will never be delivered.

In asynchronous Python frameworks like FastAPI and Starlette, as highlighted in streaming implementation guides by Learnixo, the streaming generator must poll the underlying ASGI connection state:

import asyncio
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()

async def token_stream_generator(request: Request, prompt: str):
    try:
        async for token in llm_engine.generate_stream(prompt):
            # Check if client disconnected
            if await request.is_disconnected():
                # Signal upstream inference scheduler to cancel sequence
                await llm_engine.abort(prompt_id)
                break
            
            yield f"data: {{\"token\": \"{token}\"}}\n\n"
            await asyncio.sleep(0) # yield control to event loop
    except asyncio.CancelledError:
        await llm_engine.abort(prompt_id)
        raise
    finally:
        yield "data: [DONE]\n\n"

@app.post("/stream")
async def stream_endpoint(request: Request):
    return StreamingResponse(
        token_stream_generator(request, "user prompt"),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache, no-transform",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",
        }
    )

In vLLM and SGLang, passing cancellation signals directly to the internal continuous batching scheduler frees the allocated KV cache blocks immediately, preventing GPU memory starvation.

Streaming Structured Outputs and the Incomplete JSON Problem

Modern AI workflows frequently require streaming structured JSON data (such as tool-call arguments, form auto-completion, or schema-constrained entities). However, JSON is a context-free grammar designed for static, complete documents. During token-by-token generation, the in-flight JSON payload is syntactically broken at every intermediate step (unclosed quotes, trailing commas, missing braces).

The O(N^2) Full-Repair Anti-Pattern

A common developer misstep is invoking a complete JSON repair tool (such as json-repair) on the accumulated string buffer on every incoming token.

As documented in engineering benchmarks by Aha.io, running an $O(N)$ string repair algorithm across $N$ streamed tokens results in cumulative $O(N^2)$ computational overhead. While unnoticeable for short strings under 2KB, once tool arguments or JSON bodies exceed 5KB to 10KB (such as generating code files or detailed analytical reports), reparsing the entire string on every token generates visible browser main-thread freezes and high CPU utilization.

State-Machine Based Partial JSON Parsing

Production architectures employ stateful, incremental parsers such as partial-json-parser or specialized streaming AST lexers.

Instead of scanning from character zero on each chunk, incremental streaming parsers maintain an active grammar stack tracking open containers:

Container Tracking: A state stack records currently open object braces {, array brackets [, and active string literal quotes ".
Key-Value Isolation: As object keys and primitive scalar values complete, they are committed to an immutable memory record, avoiding re-evaluation.
Synthetic Closure: When a render cycle requires a snapshot of the current state, the parser appends synthetic closing tokens to the working AST slice without mutating the raw stream buffer.

Stream Protocol Specifications (Vercel AI Data Stream Protocol)

To standardize multi-part streaming across natural language, reasoning traces, and structured tool calls, protocols like the Vercel AI SDK Data Stream Protocol encode typed event prefixes over SSE:

0: "text delta" (standard natural language token)
1: {"function_call": {"name": "search", "arguments": "..."}} (tool execution chunks)
2: {"thought": "reasoning step"} (reasoning tokens / chain-of-thought)
d: {"finishReason": "stop", "usage": {"completionTokens": 142}} (final execution metadata)

This explicit framing allows client-side state engines to route different stream fragments directly to dedicated UI components without guessing data boundaries.

Client-Side Rendering and Backpressure Throttling

High-performance inference hardware (such as groq LPUs or Cerebras CS-3 clusters) can stream tokens at rates exceeding 150 to 300 tokens per second. If a frontend application triggers a React state update and DOM re-render for every single token received, the browser main thread quickly drops frames, leading to stuttering scrolling and unresponsive input.

RequestAnimationFrame (RAF) Batching

To maintain a fluid 60 FPS user interface during high-velocity token delivery, clients should buffer incoming tokens into a non-reactive queue and flush updates using requestAnimationFrame:

// Client-side Token Throttling
class TokenStreamBuffer {
  private queue: string[] = [];
  private isRafScheduled = false;
  private onFlush: (text: string) => void;

  constructor(onFlush: (text: string) => void) {
    this.onFlush = onFlush;
  }

  public push(token: string) {
    this.queue.push(token);
    if (!this.isRafScheduled) {
      this.isRafScheduled = true;
      requestAnimationFrame(() => this.flush());
    }
  }

  private flush() {
    if (this.queue.length > 0) {
      const chunk = this.queue.join('');
      this.queue = [];
      this.onFlush(chunk);
    }
    this.isRafScheduled = false;
  }
}

Markdown and Delimiter Splitting

Streaming markdown introduces visual parsing artifacts:

Code Fences: When a model emits `python , the first token might be ` `, followed by , followed by python`. Standard markdown parsers will flicker between inline code spans and code blocks until the full fence is received.
Math Delimiters: LaTeX formulas bounded by $$ or \( require buffered lexing to prevent raw formatting noise from flashing on screen.

Production streaming markdown renderers maintain a lookahead buffer or use incremental Markdown AST parsers (such as remark/rehype pipelines configured with loose unclosed tag tolerance) to smooth visual transitions.

Production Checklist

Building reliable LLM streaming infrastructure requires adhering to five core practices:

Transport: Default to SSE over HTTP/2 for stateless scalability. Reserve WebSockets for bidirectional voice/canvas interactions.
Proxy Settings: Set X-Accel-Buffering: no, Cache-Control: no-transform, and configure keep-alive ping frames.
Compute Protection: Always bind client disconnect detection to engine cancellation APIs (request.is_disconnected() to llm.abort()).
Structured Parsing: Use incremental partial JSON state machines; avoid running $O(N)$ full-string repairs in hot token loops.
UI Rendering: Throttle high-speed token updates with requestAnimationFrame to prevent DOM thread locking.