Self-Hosted Embedding and Reranking Serving in Production: TEI vs. Infinity vs. vLLM Architecture, Dynamic Batching, and Serving Economics

While generative large language models dominate inference infrastructure discussions, vector embeddings and cross-encoder rerankers handle order-of-magnitude higher request volumes in production retrieval-augmented generation (RAG) and search pipelines. Serving embedding and reranking models presents fundamentally different computational characteristics than auto-regressive text generation. Without auto-regressive token generation loops or key-value (KV) cache state management, the primary engineering bottlenecks shift entirely to dynamic request batching, CPU-bound tokenization, padding elimination, and GPU kernel pooling.

Production engineering teams evaluating self-hosted embedding infrastructure typically converge on three primary open-source serving engines: Hugging Face Text Embeddings Inference (TEI), Infinity, and vLLM in pooling mode. Each framework optimizes for distinct operational constraints across latency, multi-model density, and hardware portability.

Inference architecture diagram for production embedding and reranking serving

Architectural Bottlenecks in Embedding and Reranking Inference

Unlike generative models where memory bandwidth during the decode phase dominates latency, encoder-only models (such as BERT, RoBERTa, and ModernBERT) and sequence-classification cross-encoders run single forward passes across variable-length inputs. Several architectural factors dictate throughput and tail latency.

1. Tokenization and CPU Scheduling Overhead

In high-throughput embedding endpoints, tokenization is frequently the system bottleneck. Profiling published by Snowflake Engineering on naive embedding inference servers revealed that the actual GPU forward pass (embed()) often accounted for only 10% of total request execution time, with 90% consumed by CPU-bound tokenization, string manipulation, and JSON serialization.

High-performance embedding engines resolve this by running asynchronous tokenization worker pools in native code (such as Rust or optimized C++) ahead of the GPU batch queue, ensuring the GPU compute pipeline remains fully saturated without thread contention.

2. Variable-Length Concatenation vs. Batch Padding

Standard batched inference pads all sequences in a batch to match the length of the longest input. In heterogeneous workloads where sequence lengths range from short 8-token search queries to 512-token document chunks, zero-padding wastes up to 70% of GPU compute cycles on useless multiplication operations.

Modern embedding engines eliminate padding entirely using variable-length attention kernels, such as FlashAttention varlen or FlashInfer. Sequences are flattened into a continuous 1D token buffer accompanied by a cumulative sequence length tensor (cu_seqlens), ensuring zero redundant compute across batch elements.

3. In-Kernel Sequence Pooling and Dimension Truncation

Generating dense vector representations requires aggregating hidden states across token positions. Performing mean pooling, [CLS] token extraction, or last-token pooling on the CPU introduces expensive host-to-device synchronization and high memory bandwidth overhead.

Production runtimes execute pooling, Matryoshka Representation Learning (MRL) dimension slicing, and vector L2 normalization directly within fused GPU kernels before transferring the compact vector tensor back to host memory.

4. Cross-Encoder Pair Expansion and Late Interaction

Reranking workloads compound batch complexity:

Cross-Encoders: Accept pairs of (query, document) formatted as concatenated sequences ([CLS] query [SEP] document [SEP]). For a query evaluated against 100 candidate documents, the engine must dynamically expand, tokenize, and batch 100 sequences simultaneously, extracting scalar relevance logits from the classification head.
Multi-Vector Late-Interaction Models: Architectures like ColBERT and ColPali retain token-level representations rather than collapsing sequences into single vectors. The inference server must return high-dimensional token matrices ([seq_len, dim]), requiring efficient binary or base64 tensor serialization protocols to avoid network I/O bottlenecks.

Engine Breakdown: TEI vs. Infinity vs. vLLM

Hugging Face Text Embeddings Inference (TEI)

Text Embeddings Inference (TEI) is purpose-built by Hugging Face in Rust for ultra-low-latency deployment of transformer encoders, dense embedding models, and sequence-classification rerankers.

Architecture: Standalone Rust daemon with zero Python runtime dependency. It integrates custom Candle and PyTorch C++ bindings, with optional ONNX Runtime execution for CPU environments.
Batching and Scheduling: Dynamic token-level scheduling with sub-millisecond dispatch overhead. Requests are queued and batched based on token limits (--max-batch-tokens) rather than fixed request counts.
Kernels: Native integration of FlashAttention-2, FlashInfer, and custom fused CUDA kernels for sequence pooling and L2 normalization.
Strengths: Lowest p99 latency in single-model deployments, minimal baseline memory footprint, and production-grade gRPC and HTTP interfaces.
Trade-Offs: Limited to single-model instances per container; multi-modal vision-language embedding architectures (such as ColPali) require alternative runtimes.

Infinity

Infinity, developed by Michael Feil, is a high-throughput serving framework designed for embedding, reranking, CLIP, and ColPali models.

Architecture: Hybrid Python/Rust architecture utilizing PyTorch, ONNX, TensorRT, and CTranslate2 backends.
Multi-Model Co-Hosting: Infinity natively supports deploying multiple models within a single server instance on a shared GPU (for example, serving bge-small-en-v1.5 and mxbai-rerank-large-v2 concurrently).
Multi-Modal Support: Full native support for late-interaction vision-language models including ColPali and ColQwen, delivering base64-encoded token matrices for visual document retrieval.
Hardware Portability: Broad accelerator support across NVIDIA CUDA, AMD ROCm, Apple Silicon MPS, AWS Inferentia, and CPU.
Strengths: Operational flexibility, multi-model co-location reducing GPU idle time, and native support for modern multi-modal RAG stacks.
Trade-Offs: Slightly higher scheduling overhead than pure Rust implementations like TEI under extreme sub-10ms latency constraints.

vLLM (Pooling / Embedding Mode)

While primarily known as an auto-regressive generation engine, vLLM supports embedding and sequence classification models via its pooling engine architecture.

Architecture: Continuous batching engine with PagedAttention and multi-GPU distributed orchestration.
Large Backbone Scaling: vLLM excels when deploying 7B+ parameter decoder-based embedding models (such as SFR-Embedding-Mistral, NV-Embed-v2, or Qwen2-Embedding) that exceed the memory capacity of single commodity GPUs.
Tensor Parallelism: Built-in Megatron-style tensor parallelism enables sharding large embedding backbones across multiple GPUs with unified memory management.
Strengths: Unmatched throughput for multi-gigabyte foundation model embeddings; unified infrastructure across generative and embedding workloads.
Trade-Offs: Substantially higher memory footprint and initialization overhead for standard 100M-500M parameter BERT-scale embedding models compared to TEI or Infinity.

Architectural Comparison Matrix

Production Deployment Recipes

Deploying Hugging Face TEI on CUDA

TEI is deployed via pre-built Docker containers tailored to specific GPU architectures:

docker run --gpus all -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-embeddings-inference:hopper-1.9 \
  --model-id BAAI/bge-large-en-v1.5 \
  --max-batch-tokens 16384 \
  --max-client-batch-size 128 \
  --auto-truncate

Deploying Multi-Model Inference with Infinity

Infinity allows consolidating embedding and reranking services onto a single GPU instance:

docker run --gpus all -p 7997:7997 \
  -v $PWD/cache:/app/.cache \
  michaelf34/infinity:latest v2 \
  --model-id michaelfeil/bge-small-en-v1.5 \
  --model-id mixedbread-ai/mxbai-rerank-xsmall-v1 \
  --port 7997 \
  --batch-size 64

Engineering Decision Framework

Selecting the appropriate embedding and reranking engine depends on throughput requirements, model architectures, and infrastructure budgets:

Low-Latency, High-Volume Text Pipelines (TEI): For microservice architectures running dedicated BERT or ModernBERT embedding models with strict SLAs (p99 under 15ms), TEI delivers the highest request density and lowest CPU scheduling overhead per GPU dollar.
Multi-Task and Multi-Modal RAG Stacks (Infinity): For applications combining dense text embeddings, cross-encoder rerankers, and vision-language late-interaction models (ColPali) on constrained GPU allocations, Infinity provides co-hosting capabilities and cross-platform hardware support.
Decoders and Foundation-Scale Embeddings (vLLM): For high-accuracy domain retrieval pipelines requiring 7B+ parameter decoder embedding models distributed across multi-GPU nodes, vLLM provides tensor-parallel execution and unified cluster management.