Vision-Language Model Serving in Production: Comparing LLaVA-NeXT, Qwen2-VL, InternVL2, and Pixtral Architecture, Dynamic Resolution Patching, Visual Token Pruning, and Serving Latency Economics

Deploying Vision-Language Models (VLMs) in production introduces systems challenges distinct from text-only Large Language Models. In standard text serving, input length scales linearly with character and token count. In multimodal systems, a single high-resolution image or short video clip can expand into thousands of visual tokens before textual generation begins. This expansion places immense strain on GPU memory bandwidth, inflates Key-Value (KV) cache allocation, and degrades Time-to-First-Token (TTFT).

Engineers deploying open-weight multimodal architectures must balance spatial visual fidelity against serving latency, batch concurrency, and inference infrastructure cost. Understanding the design trade-offs between leading architectures such as LLaVA-NeXT, Qwen2-VL, InternVL2, and Pixtral is essential for building scalable inference pipelines.

Architectural Anatomy of Vision-Language Serving

Modern open-weight VLMs typically employ a tripartite architectural pipeline:

Vision Encoder: A pre-trained Vision Transformer (ViT) or convolutional-transformer hybrid that processes raw pixel tensors. Common backbones include CLIP-ViT-L/14 (336x336 fixed grid), SigLIP-SO400M, and InternViT-6B.
Multimodal Projector (Connector): An adapter module bridging the visual feature space and the textual embedding space. Implementations range from two-layer Multi-Layer Perceptrons (MLPs) with GELU activations to cross-attention Perceiver resamplers and dynamic convolutional downsamplers.
Autoregressive Language Model: A transformer decoder (such as Llama, Qwen, or InternLM) that processes concatenated visual token embeddings and user text prompt embeddings in a unified sequence.

While conceptually straightforward, this pipeline creates divergent computational profiles during inference. The vision encoder execution is compute-dense and FLOP-bound, running forward passes over large image matrices. Conversely, the subsequent autoregressive decoding phase is memory-bandwidth bound. When both stages share the same GPU resources, serving engines face GPU kernel scheduling conflicts, unbalanced compute utilization, and severe KV cache exhaustion.

VLM Token Compression and Serving Architecture

Dynamic High-Resolution Patching Architectures

Early VLMs like LLaVA-1.5 resized all input images to a fixed square resolution (e.g., 336x336 pixels), producing 576 visual tokens ( $24 \times 24$ patch grid). This fixed resizing severely distorted aspect ratios and blurred critical fine details necessary for Document OCR, chart parsing, and fine-grained visual reasoning.

Modern production VLMs employ distinct dynamic patching strategies to handle arbitrary resolutions and aspect ratios:

1. LLaVA-NeXT: AnyRes Grid Slicing

LLaVA-NeXT introduces the AnyRes (Any-Resolution) algorithm. The serving engine calculates an optimal grid configuration $\{m \times n\}$ based on the source image aspect ratio (such as $1 \times 2$ , $2 \times 2$ , or $1 \times 3$ ), crops the image into sub-images matching the ViT base resolution (336x336), and generates an additional downsampled global thumbnail.

The total number of visual tokens fed to the LLM is: L = (m * n + 1) * T_base

For a $2 \times 2$ grid using a CLIP backbone ( $T_{\text{base}} = 576$ ), AnyRes produces $(4 + 1) \times 576 = 2,880$ visual tokens per image. To prevent out-of-memory errors on extreme aspect ratios, LLaVA-NeXT applies thresholded 2D spatial pooling or bilinear interpolation to cap total sequence length.

2. InternVL2: Dynamic High-Resolution with Pixel Unshuffle

The InternVL 1.5 and InternVL2 series scale visual resolution by slicing input images into 1 to 12 dynamic tiles (and up to 40 tiles for 4K inputs) at $448 \times 448$ resolution, accompanied by a global thumbnail.

A raw $448 \times 448$ tile processed by InternViT-6B (patch size $14 \times 14$ ) produces $32 \times 32 = 1,024$ patch tokens. Feeding $1,024$ tokens per tile into an LLM across 12 tiles would yield over 13,000 visual tokens. InternVL mitigates this via a Pixel Unshuffle operation:

A $2 \times 2$ spatial neighborhood of visual tokens is reshaped into channel dimensions ( $C \to 4C$ ).
A linear projection layer projects the expanded channel dimension down to the LLM hidden dimension.
This compresses visual token count by 75%, yielding 256 tokens per $448 \times 448$ tile. A 5-tile image consumes $(5 + 1) \times 256 = 1,536$ tokens.

3. Qwen2-VL: Naive Dynamic Resolution and M-RoPE

Qwen2-VL avoids fixed grid slicing and aspect-ratio padding entirely. It employs a dynamic-resolution Vision Transformer that processes arbitrary input dimensions with a $14 \times 14$ patch size, followed by a 2D spatial downsampling block ( $2 \times 2 \to 1$ ) to reduce token density by $4\times$ .

To handle variable sequence geometry without positional distortion, Qwen2-VL implements Multimodal Rotary Position Embedding (M-RoPE). M-RoPE splits the RoPE embedding dimension into three separate frequency chunks:

Temporal index ( $T$ ) for video frames or static image indexing.
Height index ( $H$ ) for 2D spatial vertical coordinates.
Width index ( $W$ ) for 2D spatial horizontal coordinates.

For video processing, Qwen2-VL groups consecutive frames into 3D patch pairs, reducing temporal sequence length by half while maintaining fine-grained temporal timestamp alignment.

4. Pixtral: Native Multi-Resolution Transformer

Pixtral 12B departs from separate fixed-tile cropping by utilizing a 400M parameter native vision encoder that ingests images at native aspect ratios and resolutions. The encoder maps $16 \times 16$ image patches into linear sequence tokens separated by structural delimiter tokens ([IMG], [IMG_BREAK], [IMG_END]), allowing direct end-to-end multi-image and mixed text-image document ingestion.

The Systems Cost: KV Cache Inflation and TTFT

The primary operational cost in production VLM serving is not the compute FLOPs of the vision encoder, but the KV cache footprint created by visual tokens in the LLM decoder.

KV Cache Memory Equation

For a transformer model served with 16-bit precision (FP16/BF16), the memory consumed by the KV cache per token across all layers is: Memory_per_token = 2 * n_layers * n_kv_heads * d_head * 2 bytes

For a typical 70B parameter model ( $n_{\text{layers}} = 80$ , $n_{\text{kv\_heads}} = 8$ , $d_{\text{head}} = 128$ ): Memory_per_token = 2 * 80 * 8 * 128 * 2 = 327,680 bytes = 320 KB per token

Under text-only workloads with an average prompt of 500 tokens, each request consumes approximately 160 MB of KV cache.

When serving a VLM with 2,880 visual tokens per image:

Each image requires: 2,880 * 320 KB = 921.6 MB of KV cache memory.
A batch of 32 concurrent multimodal requests demands over 29.5 GB of GPU VRAM exclusively for the visual prompt KV cache.
Serving a 16-frame video at 1,000 tokens per frame requires 5.12 GB of KV cache for a single request.

This massive memory footprint reduces maximum continuous batching concurrency, triggers frequent out-of-memory preemption, and degrades system throughput.

Visual Token Pruning and Compression Techniques

To prevent memory exhaustion during multi-image and video serving, production architectures deploy visual token pruning strategies across two primary tiers:

1. Static Spatial Compression at the Projector

Pixel Unshuffle (InternVL): Downsamples $2 \times 2$ spatial patches into channel features, achieving a fixed $4\times$ reduction without learned attention overhead.
2D Spatial Pooling (LLaVA-OneVision): Applies average pooling or bilinear interpolation with striding across feature grids before the projection layer.
C-Abstractor / Perceiver Resamplers: Uses a fixed set of $K$ learnable latent queries ( $K \in [64, 256]$ ) that cross-attend to the full ViT output sequence, locking visual token count regardless of raw input resolution.

2. Dynamic Attention-Guided Pruning in LLM Layers

Recent research reveals significant spatial redundancy in visual tokens within deep transformer layers. In early layers, visual tokens participate in cross-modal alignment. In deeper layers, text generation attends heavily to textual tokens and only a sparse subset of salient visual patches.

FastV (Chen et al., 2024): Evaluates attention weights assigned to visual tokens after the initial transformer decoder layers (e.g., layer 2). It prunes 50% to 70% of visual tokens with lowest cumulative attention scores. This reduces prefill FLOPs by 36.9% and shrinks KV cache size by over 55% while maintaining benchmark accuracy within 1% of the baseline.
FasterVLM (Zhang et al., 2024): Utilizes [CLS]-to-patch attention maps from the vision encoder to filter out non-informative background patches before token projection, avoiding LLM prefill compute entirely.
PyramidDrop and Balanced Token Pruning (BTP): Progressively drop visual tokens across successive transformer stages (e.g., dropping 25% at layer 8, 25% at layer 16), creating a tapered memory profile.

Serving dynamic token pruning in production requires inference runtimes that support non-uniform sequence lengths without breaking static CUDA graph allocations.

Serving Infrastructure and Runtime Scheduling

Optimizing VLM inference requires tailored runtime scheduling in engines like vLLM and SGLang:

Asynchronous Vision Encoding vs. Co-Located Tensor Parallelism

When serving large VLMs (e.g., 72B) across 4 or 8 GPUs with Tensor Parallelism (TP):

If the Vision Transformer (e.g., 300M to 6B parameters) is replicated across all GPUs, small ViT matrix multiplications suffer from TP communication overhead and low tensor core occupancy.
If the ViT is partitioned across the TP group, all-reduce synchronization latency can bottleneck overall prefill time.
Production architectures increasingly decouple the Vision Encoder onto dedicated auxiliary compute instances or asynchronous worker threads that stream projected embeddings directly into the LLM prefill queue.

Chunked Prefill with Multimodal Budgeting

Standard Chunked Prefill divides long prompts into fixed token chunks (e.g., 512 tokens) to interleave prefill computation with latency-sensitive decode steps. In VLMs, splitting an image's token sequence across chunk boundaries can disrupt spatial positional continuity if positional embeddings rely on monolithic 2D coordinate matrices. Serving engines must enforce image-atomic or tile-atomic chunk boundaries.

Prefix Caching for Multimodal Assets

In applications involving repeated visual assets (e.g., visual document headers, persistent UI elements in web navigation agents, or static video backgrounds), caching the computed KV pairs of visual tokens via RadixAttention yields massive latency gains. Serving engines hash the raw image pixel tensor or vision embedding vector to retrieve precomputed KV blocks, eliminating both ViT forward passes and LLM prefill latency for recurrent assets.

Production Architectural Comparison

The selection of a VLM architecture depends heavily on deployment latency requirements and document complexity:

InternVL2 (8B / 26B / 76B): Best suited for enterprise document understanding, multilingual OCR, and high-resolution chart parsing. The $4\times$ pixel unshuffle compression ensures manageable token counts even when processing up to 12 tiles.
Qwen2-VL (2B / 7B / 72B): Best suited for video understanding, robotic camera feeds, and variable aspect-ratio UI interaction due to native dynamic resolution, M-RoPE positional extrapolation, and 3D temporal patch grouping.
LLaVA-NeXT (8B / 72B): Strong open baseline for general visual reasoning and multi-modal chat, though AnyRes tiling requires careful threshold configuration to avoid KV cache bloat.
Pixtral (12B): Ideal for native multi-image documents and interleaved text-image reasoning with minimal tiling preprocessing complexity.

Implementation Guidelines for Production Deployment

Calculate Maximum Multimodal Batch Limits: Dimension GPU memory pools by calculating worst-case KV cache usage assuming all concurrent requests submit maximum-tiled images.
Apply Dynamic Tile Caps: Enforce strict upper bounds on dynamic resolution grids (e.g., capping InternVL tiles at 6 or 12 for real-time APIs, reserving 40-tile mode for offline batch pipelines).
Enable Paged Attention with 8-Bit KV Caching: Utilizing FP8 KV cache quantization reduces per-token memory overhead by 50%, enabling $2\times$ higher concurrent multimodal batch sizes without noticeable degradation in visual groundedness.
Leverage Visual Prefix Caching: For GUI automation and document workflows with recurring layouts, ensure prefix caching is enabled to reuse image KV blocks across multi-turn agent interactions.