GPU10 articles

GPU

Articles

FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3
FlashAttention: IO-Aware Exact Attention, Tiling, Online Softmax, and the Evolution to FlashAttention-3 The attention mechanism is the computational bottleneck of every Transformer model. Standard implementations materialize the full $N \times N$ attention matrix in high-bandwidth memory (HBM), incurring $O(N^2)$ memory reads and writes that dominate runtime long before arithmetic intensity saturates the GPU. FlashAttention and its successors eliminate this bottleneck by restructuring the atten
August 28, 20261 min
CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination
CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination In high-throughput large language model serving, the autoregressive generation (decode) phase presents an operational bottleneck distinct from prefill processing. While prefill computation is dominated by compute-bound General Matrix Multiplications (GEMMs) operating across extended sequence lengths, autoregressive decoding processes a single token per sequence at each ite
August 22, 20261 min
GPU Cluster Storage in Production: GPUDirect Storage, NVMe-oF, Parallel File Systems, and Checkpointing Throughput
Training frontier large language models and serving hundred-billion parameter checkpoints places extreme demands on storage subsystems. While compute clusters frequently deploy thousands of GPUs connected via high-bandwidth interconnects like NVLink and InfiniBand, storage architectures often become severe bottlenecks during two critical operational phases: distributed checkpointing and cold-start model weight loading. A standard 70-billion parameter model in BF16 precision generates approximat
August 22, 20261 min
GPU Memory Profiling in Production LLM Serving: CUDA Allocator Internals, PyTorch Snapshots, and VRAM Optimization
In high-throughput large language model serving, memory is the primary constraint governing latency, batch concurrency, and context length. While model parameter footprints are static and easily calculated, runtime GPU memory (VRAM) dynamics are governed by low-level caching allocators, dynamic key-value (KV) cache allocation pools, transient activation spikes, and memory fragmentation. When an inference worker crashes with torch.cuda.OutOfMemoryError, default system diagnostics such as nvidia-
August 22, 20261 min
Fast Model Weight Loading in Production: Safetensors, Tensorizer, and Direct GPU Deserialization
Fast Model Weight Loading in Production: Safetensors, Tensorizer, and Direct GPU Deserialization In modern large language model inference clusters, cold start latency is rarely bounded by GPU compute allocation. Instead, the operational bottleneck centers on storage I/O and weight deserialization. As foundation models scale from 70 billion to 405 billion parameters, raw weight footprints range from 140 GB to over 800 GB in standard 16-bit precision. On naive serving stacks, deserializing these
August 20, 20261 min
GPU Slicing in Production AI Systems: Comparing MIG, MPS, Time-Slicing, and Dynamic Partitioning
GPU Slicing in Production AI Systems: Comparing MIG, MPS, Time-Slicing, and Dynamic Partitioning Modern production AI systems rarely deploy a single standalone large language model. Contemporary compound AI architectures rely on heterogeneous pipelines comprising embedding models (such as BGE or E5), cross-encoder rerankers, safety classifiers (such as Llama Guard), speculative decoding draft models, and vision-language encoders. While primary generation models typically require dedicated multi
August 20, 20261 min
LLM Autoscaling and Cold Starts in Kubernetes: Architecture, KEDA Metrics, Model Weight Caching, and Ephemeral GPU Provisioning
Autoscaling large language model workloads on Kubernetes presents a fundamentally different engineering problem than traditional stateless microservices. While web APIs scale on CPU utilization or request rate within seconds, LLM inference instances require specialized GPU accelerators, massive container images, multi-gigabyte weight tensors, and intensive runtime compilation before serving a single token. Without proactive architectural design, a cold-starting LLM pod on Kubernetes often requi
August 20, 20261 min
Custom LLM Kernel Optimization in Production: Triton vs. CUDA C++ vs. torch.compile vs. CUTLASS
Serving large language models at scale requires extracting maximum performance from modern GPU architectures like NVIDIA Ampere, Hopper, and Blackwell. While early production deployments relied on standard PyTorch eager execution and standard cuBLAS calls, high-throughput serving systems such as vLLM, SGLang, and TensorRT-LLM depend on specialized fused GPU kernels to eliminate memory bandwidth bottlenecks and saturate Tensor Cores. Engineering teams face four primary paradigms for kernel devel
August 20, 20261 min
GPU Cluster Networking in Production: InfiniBand vs. RoCEv2 vs. Ultra Ethernet Architecture, Congestion Control, and NCCL Collective Latency
Distributed training and high-throughput inference workloads are fundamentally bound by the network fabric. While traditional cloud applications rely on asynchronous request-response cycles that absorb latency jitter, distributed deep learning relies on synchronous collective communication. Operations such as All-Reduce, All-Gather, and All-to-All require hundreds or thousands of GPUs to exchange tensors and synchronize at strict barrier points before execution can proceed. In this execution mo
August 20, 20261 min
French startup Kog says smarter software can pull 30x faster inference out of stock GPUs
The race for faster AI inference has pushed labs toward custom silicon, but a French startup argues the cheapest speedup is already sitting in servers companies bought. Kog says it can reach roughly thirty times faster LLM inference on standard data-center GPUs using software alone. Kog demonstrated three thousand tokens per second per request on AMD MI300X and NVIDIA H200 GPUs in May, running a small two-billion-parameter model it has since open-sourced as Laneformer 2B. Chief executive Gael D
August 14, 20261 min