Subscribe
Category / page 8

Chips & Infrastructure

314 articles

Articles, page 8

141-160 of 314

  1. SGLang v0.5.18 Cuts LLM Cold Starts by 2.4x with Overlapped Weight Loading and CUDA Graph Capture

    The open-source LLM serving engine SGLang has released version 0.5.18, introducing an overlapped startup engine that significantly reduces cold-start latency for large language models, alongside communication kernel optimizations and expanded architecture support. Comprising 710 pull requests from 212 contributors, the release addresses operational overheads in LLM infrastructure where autoscaling, rolling cluster deployments, and worker node recovery frequently pay steep restart penalties. O

    AI
    2 min read

  2. Texas Governor Greg Abbott Says AI Data Centers 'Dug Their Own Grave' Amid Community Backlash

    Texas Governor Greg Abbott issued a sharp critique of artificial intelligence infrastructure developers on Sunday, stating that data center operators have "dug their own grave" by moving into municipalities without securing local community support or complying with state transparency mandates. Speaking on ABC's This Week, Abbott addressed growing public pushback across Texas over utility grid strain, localized electricity rate increases, and heavy water consumption from cooling facilities. The

    Infrastructure
    2 min read

  3. Distributed Locking and Deadlock Prevention in Production AI Agents: Architecture, Semantic Mutexes, Leases, and Wait-For Graphs

    When autonomous AI agent architectures scale from isolated single-agent loops to concurrent multi-agent fleets, systems engineering teams encounter a fundamental distributed systems reality: concurrency bugs in language models do not merely produce dirty reads. They produce reasoning corruption, circular delegation deadlocks, and cascading execution failures. In single-agent execution pipelines, control flow is strictly sequential: retrieve context, prompt the model, parse tool arguments, apply

    Agents
    8 min read

  4. Israel Outlines Five-Year AI Strategy: 100,000 Accelerators, 5nm Fab, and Quantum Base

    Israel's National Artificial Intelligence Directorate has released its five-year National AI Strategic Plan, outlining formal commitments to construct a sovereign infrastructure footprint spanning at least 100,000 advanced AI accelerators, a domestic quantum computer, and advanced semiconductor fabrication capabilities at 5 nanometers or below. Operating under the Prime Minister's Office, the Directorate published the roadmap to implement Government Resolution 4255, originally approved by the I

    AI
    2 min read

  5. Microscaling Formats in Production LLM Inference: NVFP4 vs. MXFP4 Architecture, Block Quantization, Kernel Scaling, and Perplexity Degradation

    As frontier language models scale into hundreds of billions of parameters, memory bandwidth and VRAM capacity remain the primary bottlenecks in production inference serving. While 8-bit floating-point formats (FP8 E4M3 and E5M2) achieved mainstream adoption across Hopper and Ada generation hardware, the industry is transitioning to 4-bit floating-point (FP4) arithmetic to double inference throughput and halve memory footprints. Directly quantizing deep transformer weights and activations into 4

    AI
    7 min read

  6. Alibaba Plans Record $10.2B Hong Kong Share Offering to Fund Full-Stack AI Infrastructure and Models

    Alibaba Group Holding announced a proposed equity placement in Hong Kong to raise HK$80 billion ($10.2 billion), allocating 100% of net proceeds to expand its full-stack artificial intelligence capabilities across custom silicon, cloud compute infrastructure, and foundation model development. Transaction Structure and Market Scale According to terms reviewed by Reuters, Alibaba is offering 710 million ordinary shares at HK$112.70 per share, representing a 3.6% discount to its latest closing p

    AI
    2 min read

  7. Fill-in-the-Middle (FIM) and Inline Code Completion in Production: Architecture, Prefix-Suffix Context Windows, Speculative Decoding, and Sub-50ms Serving Latencies

    Fill-in-the-Middle (FIM) and Inline Code Completion in Production: Architecture, Prefix-Suffix Context Windows, Speculative Decoding, and Sub-50ms Serving Latencies Inline code completion is the most latency-sensitive deployment of large language models in production software engineering. Unlike conversational assistants or background agentic batch jobs, inline code completion operates directly inside an active editor typing loop. Developers type at speeds ranging from 40 to 120 words per minut

    AI
    7 min read

  8. Predictive KV Cache Warming in Production LLM Serving: Architecture, Session Prefetching, and TTFT Latency Shaving

    Predictive KV Cache Warming in Production LLM Serving: Architecture, Session Prefetching, and TTFT Latency Shaving In long-context large language model (LLM) deployments, Time-to-First-Token (TTFT) represents the primary latency bottleneck. As input prompts scale from 8,000 tokens to 128,000 tokens in multi-turn coding agents, legal document analysis, and enterprise retrieval pipelines, prefill execution consumes between 85% and 95% of total GPU compute time per request. Standard optimization

    AI
    5 min read

  9. Decentralized and Peer-to-Peer LLM Inference in Production: Architecture, Ring Memory Partitioning, and Network Latency

    Decentralized and Peer-to-Peer LLM Inference in Production: Architecture, Ring Memory Partitioning, and Network Latency Frontier open-weight models such as Llama 3.1 405B, DeepSeek-V3, and Command R+ have expanded model capabilities, but their parameter scales exceed the physical memory limits of individual consumer and edge workstations. Running a 405-billion parameter model in 16-bit precision requires over 810 GB of memory, and even 4-bit quantized variants require roughly 230 GB of contiguo

    AI
    7 min read

  10. Continuous Pre-Training in Production: Domain Adaptation, Replay Buffers, Learning Rate Restarts, and Catastrophic Forgetting Mitigation

    Continuous Pre-Training in Production: Domain Adaptation, Replay Buffers, Learning Rate Restarts, and Catastrophic Forgetting Mitigation Adapting general-purpose foundation models to specialized enterprise domains (such as clinical medicine, corporate law, quantitative finance, and proprietary software codebases) presents a fundamental architectural challenge. While Retrieval-Augmented Generation (RAG) and Supervised Fine-Tuning (SFT) remain standard first-line approaches, both exhibit severe s

    LLM
    7 min read

  11. Memory Shortage Drives Nvidia AI Server Prices Up Over 15%

    Nvidia has notified major customers that prices for server systems containing its artificial intelligence accelerators are increasing by more than 15% in many configurations, according to reports from Bloomberg and Fortune. The price adjustments stem from severe supply constraints and rising costs across dynamic random-access memory (DRAM) and high-bandwidth memory (HBM) modules. The price increases will apply to server systems scheduled for delivery starting in early 2027, covering platforms p

    AI
    2 min read

  12. Low-Precision Quantization Kernels in Production: Comparing Marlin, ExLlamaV2, FlashInfer, and BitBLAS

    Low-Precision Quantization Kernels in Production: Comparing Marlin, ExLlamaV2, FlashInfer, and BitBLAS Architecture, Memory Bandwidth, and Decoding Throughput Autoregressive large language model (LLM) serving operates under two distinct compute regimes: a compute-bound prefill phase and a memory-bandwidth-bound decode phase. While processing the initial prompt involves matrix-matrix multiplications (GEMM) with high arithmetic intensity, generating tokens one by one requires matrix-vector multip

    AI
    9 min read

  13. Anthropic Nears $7B Acquisition of AI Infrastructure Startup Decart Ahead of IPO

    Anthropic is finalizing negotiations to acquire Israeli artificial intelligence infrastructure startup Decart in a transaction valued at approximately $7 billion, according to reporting from Calcalist and Reuters. The acquisition, expected to be settled primarily in Anthropic equity, would mark the Claude developer's largest purchase to date as it prepares for a planned initial public offering. Founded in September 2023 by Dr. Dean Leitersdorf and Moshe Shalev, Decart specializes in hardware-ag

    AI
    2 min read

  14. LLM Inference on AMD ROCm in Production: MI300X Architecture, Triton Kernel Parity, and vLLM Serving Benchmarks

    LLM Inference on AMD ROCm in Production: MI300X Architecture, Triton Kernel Parity, and vLLM Serving Benchmarks Serving frontier large language models in enterprise production has historically been synonymous with NVIDIA CUDA infrastructure. However, the deployment of AMD Instinct MI300X accelerators across tier-one hyperscalers and neoclouds has established a viable alternative for high-throughput inference fleets. With 192 GB of high-bandwidth memory (HBM3) and 5.3 TB/s of peak theoretical m

    Inference
    5 min read

  15. Groq Secures $350M at $3.5B Valuation to Expand Nvidia-Powered AI Neocloud

    AI infrastructure provider Groq has raised $350 million in a Series A funding round at a $3.5 billion valuation, led by investment firm Disruptive with expected participation from Nvidia subject to customary closing conditions. The financing accelerates the company's structural pivot from developing custom inference silicon toward operating an enterprise-grade inference cloud powered by Nvidia accelerated computing systems. The round follows a $650 million capital raise completed in June 2026 a

    AI
    2 min read

  16. Nvidia Expands Nemotron Open-Weight Push to Counter Chinese Labs Under $6B Poolside Deal

    Nvidia Expands Nemotron Open-Weight Push to Counter Chinese Labs Under $6B Poolside Deal Nvidia plans to use the infrastructure and engineering team acquired through its $6 billion deal with AI startup Poolside to build frontier open-weight models under its Nemotron family, according to reporting from the Wall Street Journal. The initiative aims to counter the rapid global adoption of Chinese open-weight systems like DeepSeek-V3, Moonshot AI's Kimi K3, and Alibaba's Qwen series, while offering

    AI
    2 min read

  17. Token Healing and Partial Token Alignment in Production LLM Serving: Architecture, Prefix Trie Rollback, and Serving Trade-Offs

    Modern large language models operate on discrete subword tokens generated by greedy compression algorithms like Byte-Pair Encoding (BPE), WordPiece, or Unigram. While subword tokenization enables high compression rates and fixed vocabulary sizes, it introduces a structural defect at the interface between raw user text and autoregressive inference: the partial token problem, commonly known as the prompt boundary problem. When a user prompt terminates mid-token or at a punctuation boundary that c

    LLM
    7 min read

  18. Prefix-Tree KV Cache Management in Production: RadixAttention, Tree-Structured LRU Eviction, and Token-Level Sharing in SGLang and vLLM

    Prefix-Tree KV Cache Management in Production: RadixAttention, Tree-Structured LRU Eviction, and Token-Level Sharing in SGLang and vLLM Autoregressive large language model inference is heavily constrained by memory bandwidth and the computational overhead of the prefill phase. For workloads such as multi-turn conversations, autonomous agent tool loops, few-shot prompt evaluations, and tree-search decoding, consecutive requests often share substantial token prefixes. In a standard multi-turn ses

    AI
    7 min read

  19. CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination

    CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination In high-throughput large language model serving, the autoregressive generation (decode) phase presents an operational bottleneck distinct from prefill processing. While prefill computation is dominated by compute-bound General Matrix Multiplications (GEMMs) operating across extended sequence lengths, autoregressive decoding processes a single token per sequence at each ite

    Inference
    8 min read

  20. GPU Cluster Storage in Production: GPUDirect Storage, NVMe-oF, Parallel File Systems, and Checkpointing Throughput

    Training frontier large language models and serving hundred-billion parameter checkpoints places extreme demands on storage subsystems. While compute clusters frequently deploy thousands of GPUs connected via high-bandwidth interconnects like NVLink and InfiniBand, storage architectures often become severe bottlenecks during two critical operational phases: distributed checkpointing and cold-start model weight loading. A standard 70-billion parameter model in BF16 precision generates approximat

    AI
    7 min read