Subscribe
Archive / page 42 of 123

Latest articles, page 42

  1. FP8 Mixed-Precision Serving in Production: Comparing E4M3 vs. E5M2 Formats, Per-Tensor vs. Block-Wise Dynamic Scaling, FP8 KV Cache, and Tensor Core GEMM Economics

    FP8 Mixed-Precision Serving in Production: Comparing E4M3 vs. E5M2 Formats, Per-Tensor vs. Block-Wise Dynamic Scaling, FP8 KV Cache, and Tensor Core GEMM Economics Serving large language models at enterprise scale requires balancing computational throughput against high-bandwidth memory (HBM) capacity. While 16-bit floating-point formats (FP16 and BF16) remain the standard for model pre-training and fine-tuning, their memory footprint and arithmetic bandwidth create severe bottlenecks during hi

    Inference
    6 min read

  2. Odds Ratio Preference Optimization (ORPO): Mathematical Foundations, Monolithic SFT-Alignment, and Reference-Free Odds Penalties

    Post-training large language models typically relies on a decoupled, two-stage pipeline: Supervised Fine-Tuning (SFT) on curated instruction-response pairs, followed by Preference Alignment using algorithms such as Reinforcement Learning from Human Feedback (PPO) or Direct Preference Optimization (DPO). While effective, this multi-stage paradigm introduces substantial operational and computational friction. First, DPO and PPO require maintaining auxiliary reference models in GPU memory to preve

    AI
    9 min read

  3. Ringg Raises $10M from Peak XV to Scale Enterprise Voice AI Agents

    Indian voice AI startup Ringg has secured $10 million in fresh capital from Peak XV Partners as an extension to its Series A round. The funding brings the startup's total Series A capital to $15.5 million, following an initial $5.5 million tranche closed earlier in 2026. The capital injection comes as voice remains the dominant channel for commercial communication in India. According to survey data from Truecaller, over 76% of Indian consumers prefer phone interactions when contacting businesse

    AI Voice
    2 min read

  4. QLoRA: Mathematical Foundations of 4-Bit NormalFloat Quantization, Double Quantization, and Paged Optimizers

    Fine-tuning large language models in full 16-bit precision requires substantial hardware infrastructure. For a 65-billion parameter model, storing weights in 16-bit brain floating point (BF16) requires 130 GB of VRAM. When optimizing those parameters with 32-bit AdamW, storing gradients (130 GB), first moments (260 GB), second moments (260 GB), and master weights (260 GB) drives the minimum memory footprint beyond 780 GB of VRAM, even before accounting for sequence activations. While Low-Rank A

    AI
    7 min read

  5. Projecting Conflicting Gradients (PCGrad): Mathematical Foundations, Orthogonal Projections, and Multi-Task Optimization in Deep Learning

    Projecting Conflicting Gradients (PCGrad): Mathematical Foundations, Orthogonal Projections, and Multi-Task Optimization in Deep Learning In modern machine learning systems, models rarely optimize for a single objective. Foundation models are trained simultaneously on diverse data distributions spanning natural language, source code, mathematical reasoning, and multimodal inputs. Similarly, post-training alignment pipelines must simultaneously optimize for helpfulness, factual accuracy, harmles

    AI
    11 min read

  6. Amazon to Shut Down Mechanical Turk on September 30 After 21 Years

    Amazon Web Services has announced that it will permanently shut down Amazon Mechanical Turk on September 30, 2026. The retirement brings an end to a 21-year-old crowdsourcing platform that served as an early foundation for modern machine learning dataset annotation and human-in-the-loop computing. In a formal notice posted to the Mechanical Turk portal, AWS stated that the decision followed an internal assessment of its service portfolio. The platform had previously halted new customer registra

    Amazon
    3 min read

  7. Hierarchical KV Cache Offloading in Production LLM Serving: Host RAM, Local NVMe, Remote Storage, and LMCache Architecture

    High-concurrency large language model (LLM) serving faces an acute memory capacity bottleneck. While modern GPUs deliver high floating-point compute throughput, High-Bandwidth Memory (HBM) capacity remains severely constrained. In workloads involving multi-turn conversations, agentic coding loops, and long-document retrieval-augmented generation (RAG), Key-Value (KV) cache tensors rapidly consume available GPU memory, forcing engines to either reject requests, preempt running sequences, or recom

    AI
    9 min read

  8. Mixture-of-Depths: Mathematical Foundations, Dynamic Compute Routing, Capacity-Constrained Tensors, and IsoFLOP Scaling

    Mixture-of-Depths (MoD): Mathematical Foundations, Dynamic Compute Routing, Capacity-Constrained Tensors, and IsoFLOP Scaling In standard autoregressive Transformer architectures, computational effort is distributed uniformly across all tokens in a sequence. Every token position $i \in \{1, \dots, S\}$ passes through every layer $l \in \{1, \dots, L\}$, executing identical matrix multiplications across multi-head self-attention and feed-forward networks (FFN). This architectural constraint igno

    AI
    11 min read

  9. LLM Gateways in Production: Comparing LiteLLM, Portkey, Kong AI Gateway, and Cloudflare AI Gateway Architecture, Fallback Cascades, and Serving Economics

    As enterprise production architectures scale from single prototype endpoints to multi-model agentic pipelines, coupling application code directly to foundation model provider APIs creates severe operational bottlenecks. Direct client integration leads to fractured telemetry, credential sprawl across microservices, unhandled upstream rate limits, and vendor lock-in. To isolate application logic from downstream provider volatility, engineering teams deploy dedicated LLM gateways. These specialize

    AI
    8 min read

  10. OpenAI Data Center Lead Chris Malone Departs Amid Infrastructure Realignment

    OpenAI head of data centers Chris Malone has departed the company, marking another high-level transition within the artificial intelligence lab as it restructures its infrastructure divisions ahead of a planned initial public offering. Malone, who joined OpenAI in March 2025 after five years leading data center strategy at Meta and over a decade as an engineer at Google, oversaw technical deployment across OpenAI's compute expansion efforts. Infrastructure Realignment Malone's exit coincides

    AI
    1 min read