Distributed Vector Search and Sharding Architecture in Production: Horizontal Partitioning, Scatter-Gather Tail Latency, Filter-Aware Routing, and Dynamic Rebalancing
Scaling vector search beyond tens of millions of high-dimensional embeddings inevitably hits a physical boundary: the single-node memory wall. Because graph-based approximate nearest neighbor (ANN) algorithms such as Hierarchical Navigable Small World (HNSW) require random memory access patterns across graph vertices and high-dimensional vectors, keeping embeddings and index structures in RAM is critical for sub-20ms query latencies. A collection of 100 million 1,536-dimensional float32 vectors
1 min