Vector Compression in Production Search: Comparing SQ, PQ, and RaBitQ Architecture, Recall Retention, and Memory Economics

In production Retrieval-Augmented Generation (RAG) systems and enterprise search platforms, storing raw floating-point embedding vectors in RAM quickly encounters hard hardware limits. A dataset of 100 million 1536-dimensional embeddings stored in FP32 requires over 614 GB of high-speed memory solely for vector coordinates, before accounting for index graph structures like HNSW or DiskANN.

To scale similarity search to billions of vectors while keeping indices memory-resident, production vector databases rely on vector compression algorithms. These techniques trade controlled losses in search recall for 4x to 32x reductions in memory consumption and massive speedups in distance computation throughput.

1. Scalar Quantization (SQ): Uniform Coordinate Compression

Scalar Quantization (SQ) is the most straightforward vector compression technique. Rather than treating the vector as an entangled high-dimensional entity, SQ quantizes each dimension independently across the corpus:

• Mechanics: For each dimension $d$, the minimum and maximum values across the entire dataset are mapped onto a uniform discrete grid. In standard 8-bit scalar quantization (SQ8), continuous 32-bit floating-point values are mapped into discrete uint8 integers $[0, 255]$:

$$q_i = \text{round}\left( \frac{v_i - \min_d}{\max_d - \min_d} \times 255 \right)$$

• Memory Reduction: SQ8 achieves an exact 4x compression ratio (reducing FP32 to 1 byte per dimension) and SQ4 achieves an 8x reduction (0.5 bytes per dimension).
• Distance Computation: Approximate Euclidean distance and inner product calculations can be executed using fast SIMD integer instructions (e.g., AVX-512 VNNI or ARM NEON sdot), delivering 2x to 4x throughput gains compared to FP32 dot products.
• Recall Profile: SQ8 typically retains 98% to 99.5% of uncompressed recall@10 with minimal distortion, making it the industry standard default for datasets up to tens of millions of items.

2. Product Quantization (PQ): Subspace Decomposition and Codebooks

When memory constraints require higher compression than 4x or 8x, Product Quantization (PQ) provides configurable decomposition by breaking the vector space into orthogonal subspaces:

• Mechanics: A $D$-dimensional vector is partitioned into $M$ disjoint sub-vectors of dimension $D/M$. For each sub-vector subspace, k-means clustering is run offline to generate a codebook of $K = 256$ centroids (addressable by a single 1-byte index). The original sub-vector is replaced by the index of its nearest centroid.
• Memory Reduction: A 1536-dimensional vector partitioned into $M = 96$ sub-vectors is represented by 96 bytes (a 16x compression ratio against FP32). Partitioning into $M = 48$ sub-vectors yields a 32x compression ratio (48 bytes per vector).
• Asymmetric Distance Computation (ADC): During search, the incoming query vector $q$ remains in full FP32 precision. The engine precomputes a lookup table of distances between the query's sub-vectors and all 256 centroids in each subspace. Calculating the distance between the query and any quantized database vector requires only $M$ table lookups and additions, completely bypassing floating-point multiplications:

$$\tilde{d}(q, x) = \sum_{m=1}^M \text{LUT}_m[c_m(x)]$$

• Trade-Offs: PQ introduces quantization distortion if the embedding dimensions exhibit strong cross-subspace covariance. Techniques like Inverted Multi-Index (IMI) or Optimized Product Quantization (OPQ) apply orthogonal rotation matrices before partitioning to minimize correlation across subspaces.

3. RaBitQ: Fast Quantization with Rigorous Error Bounds

While 1-bit binary quantization (converting positive dimensions to 1 and negative dimensions to 0) achieves 32x compression, it historically suffered severe recall degradation on non-isotropic vector distributions. RaBitQ (Randomized Binary Quantization, introduced by Gao et al., 2024) solves this by providing provable error bounds and dynamic residual corrections:

• Mechanics: RaBitQ projects normalized database vectors onto random orthogonal hyperplanes and encodes them into 1-bit representations. To mitigate asymmetric distortion, each vector stores a small set of scalar calibration factors (vector norm and mean projection residual) alongside its bit-string.
• Distance Computation: Distance estimation between an unquantized query and a 1-bit database vector is computed via bitwise XOR, population count (popcnt), and a single fused-multiply-add using the stored calibration scalar:

$$\text{Dist}(q, x) \approx \alpha_x \cdot \text{POPCNT}(B_q \oplus B_x) + \beta_x$$

• Performance: On billion-scale benchmarks, RaBitQ achieves over 30x compression while delivering higher recall@100 than aggressive PQ configurations, executing distance scans at memory bandwidth limits.

Architectural Decision Framework

When selecting a vector compression scheme for production retrieval systems, engineering teams should evaluate the following criteria:

| Parameter | Scalar Quantization (SQ8) | Product Quantization (PQ32/PQ64) | RaBitQ (1-bit + Scalars) | | :--- | :--- | :--- | :--- | | Compression Ratio | 4x | 16x to 32x | 24x to 30x | | Recall Retention | 98% - 99.5% | 85% - 94% | 92% - 97% | | Index Build Time | Fast (O(N) min/max scan) | Slow (k-means clustering per subspace) | Fast (Random projection + statistics) | | Distance Engine | SIMD integer arithmetic | LUT table lookups + additions | SIMD popcnt + scalar correction | | Ideal Use Case | Datasets < 50M items where recall precision is critical | Massive multi-billion scale datasets with memory caps | High-throughput filtered search with tight latency budgets |

Re-Ranking and Two-Stage Retrieval

In high-accuracy production architectures, vector compression is typically paired with a two-stage retrieval pipeline:

1. First-Stage Candidate Generation: The compressed index (SQ, PQ, or RaBitQ) scans the dataset to retrieve the top $K \times 5$ candidate IDs at ultra-low latency.
2. Second-Stage Exact Rescoring: The engine fetches the uncompressed FP16 or FP32 vectors from NVMe flash storage or compressed RAM buffers to re-rank the top $K$ candidates using exact cosine similarity.

This hybrid pattern preserves 99.9% of full-precision retrieval accuracy while reducing resident RAM footprints by more than 85%.

Sources

• arXiv: RaBitQ: Quantizing High-Dimensional Vectors with a Bit for Efficient Retrieval (Gao et al., 2024)
• IEEE TPAMI: Product Quantization for Nearest Neighbor Search (Jégou et al., 2011)
• arXiv: Optimized Product Quantization for Approximate Nearest Neighbor Search (Ge et al., 2013)