Subscribe
Category / page 3

Chips & Infrastructure

314 articles

Articles, page 3

41-60 of 314

  1. Post-Training Quantization Toolkits in Production: Comparing AutoAWQ, AutoGPTQ, bitsandbytes, and llm-compressor Architecture, Calibration Profiling, Kernel Execution, and Serving Economics

    Large language model deployment in high-concurrency production environments is governed primarily by memory bandwidth and hardware economics. In standard autoregressive decoding, memory traffic dominates computation: for every token generated, every parameter of the model must be loaded from GPU High Bandwidth Memory (HBM) into SRAM and tensor cores. Serving a 70-billion parameter model in 16-bit precision requires at least 140 GB of VRAM solely for model weights, necessitating multi-GPU tensor

    LLM
    9 min read

  2. Local LLM Inference Frameworks in Production: Comparing llama.cpp, Ollama, Apple MLX, and Exo Distributed Clusters

    The deployment landscape for large language models is bifurcating. While datacenter workloads rely on high-throughput continuous batching engines such as vLLM and TensorRT-LLM, local and edge deployments operate under fundamentally different physical constraints. On developer workstations, embedded hardware, and private office clusters, inference is rarely bound by compute saturation across thousands of concurrent requests. Instead, it is constrained by memory bandwidth, local VRAM capacity, hos

    AI
    6 min read

  3. Anthropic Previews Model Hardware Standard for AI Agent Control of Physical and Lab Equipment

    Anthropic has introduced the Model Hardware Standard (MHS), an open interface specification intended to let AI agents control physical machinery and scientific instrumentation. Released in a research preview on August 27, 2026, the standard extends the design principles of the Model Context Protocol (MCP) to physical actuators, automated laboratory equipment, and industrial hardware. Connecting autonomous software agents to physical hardware has historically required custom integration code for

    AI
    3 min read

  4. Nvidia Pauses Selected Revenue-Sharing Deals with AI Cloud Providers

    Nvidia has paused select partnership deals under its newly introduced AI cloud financing initiative, according to reporting by The Wall Street Journal. The program, unveiled in July 2026, was designed to provide credit support and hardware access to emerging cloud computing providers in exchange for a percentage of future recurring revenues. While Nvidia stated that the broader program remains operational, several individual agreements have been put on hold as the company reassesses customer te

    Nvidia
    2 min read

  5. Anthropic Discussed $7 Billion Acquisition of AI Chip Startup MatX Before Talks Stalled

    Anthropic held acquisition discussions to buy artificial intelligence semiconductor startup MatX for approximately $7 billion before talks stalled, according to reporting from Reuters. The negotiations, which have since transitioned into discussions surrounding a potential commercial partnership, highlight the increasing urgency among frontier AI laboratories to secure in-house silicon engineering capabilities. Following the breakdown of active acquisition talks, MatX is currently seeking to ra

    Anthropic
    3 min read

  6. llama.cpp Merges DFlash 2 Support for Up to 2x Faster Speculative Decoding Across Long Contexts

    The open-source llama.cpp project has merged native support for DFlash 2, bringing parallel speculative decoding and substantial inference throughput improvements to local LLM serving across CPU, Apple Silicon, and GPU backends. The implementation, integrated via Pull Request #27342, adds local convolution operators and candidate selector mechanics designed specifically for the DFlash 2 architecture. Non-Autoregressive Speculative Drafting Standard speculative decoding uses a smaller autoreg

    llama.cpp
    1 min read

  7. Synthetic Data Generation and Data Curation Pipelines in Production: Comparing Distilabel, NVIDIA NeMo Curator, Data-Juicer, and InstructLab

    As frontier language models exhaust human-authored web text, pretraining and post-training performance increasingly depends on automated data curation and synthetic data synthesis. Unfiltered public corpora introduce duplicate content, synthetic artifacts, low-reasoning text, and licensing liabilities. Modern foundation models, including Meta's Llama 3.1, NVIDIA's Nemotron-4 340B, Alibaba's Qwen 2.5, and IBM's Granite, are trained on datasets where synthetic generation and rigorous multi-stage f

    Data Curation
    9 min read

  8. Nvidia Supply Commitments Reach $279 Billion Across Multi-Year AI Hardware Pipeline

    Nvidia has significantly expanded its forward supply chain obligations, reporting total component and manufacturing capacity commitments of $279 billion in its fiscal second-quarter disclosures. The figure marks a 134 percent sequential increase from $119 billion reported in the preceding quarter and $95.2 billion at the close of fiscal 2026. The multi-year commitments reflect efforts to secure critical semiconductor fabrication, advanced packaging, and high-bandwidth memory (HBM) capacity nece

    AI
    2 min read

  9. Anthropic Hires Google TPU Architect Amir Salek for In-House Silicon Push

    Anthropic has hired Amir Salek, the engineer who established Google's custom silicon program and led the development of its first seven Tensor Processing Unit (TPU) generations, to expand the AI lab's in-house semiconductor engineering initiatives. Salek joins Anthropic's compute team and reports to James Bradbury, Bloomberg reported (via The Economic Times). Anthropic has indicated that it wants to build an in-house silicon business and has begun hiring and listing jobs for the effort. Backg

    AI
    2 min read

  10. AI Agent Sandboxes and Safe Code Execution Runtimes in Production: Comparing E2B, Modal, Daytona, and Docker/gVisor Architecture, Isolation Boundaries, Startup Latency, and Serving Economics

    Autonomous AI agents, evaluation harnesses, and code generation pipelines increasingly execute untrusted code generated directly by Large Language Models. When an agent runs dynamic Python scripts, executes terminal shell commands, installs arbitrary packages via package managers, or browses the web, it introduces severe security and operational risks. These risks include host privilege escalation, kernel exploitation, container breakout vulnerabilities such as CVE-2024-21626, credential theft v

    AI
    8 min read

  11. PagedAttention and Virtual Memory Management: Mathematical Foundations of Non-Contiguous KV-Cache Allocation, Dynamic Block Translation, and Copy-on-Write Forking Mechanics

    In autoregressive transformer serving, the generation phase is fundamentally constrained by GPU memory capacity and memory bandwidth rather than raw compute throughput. While the initial prefill phase processes the prompt in parallel with high arithmetic intensity, the subsequent token-by-token decoding phase computes attention for only a single new token per sequence at each step. At each decoding step, the model must read all historical Key (K) and Value (V) activation tensors from GPU High Ba

    AI
    13 min read

  12. OpenAI Researcher Warns Ultrafast Inference Demands Autonomous Cyber Defense

    Accelerating inference speeds in frontier artificial intelligence systems pose critical containment challenges that human security operators cannot manage in real time, according to OpenAI researcher roon. Commenting following the unveiling of custom inference hardware architectures and ultrafast model serving tiers, roon warned that unaligned or compromised agent systems running at 50 times baseline generation speeds could execute multi-stage network penetration and lateral movement before hum

    AI
    2 min read

  13. Google Sets Android Memory Limits as AI Data Centers Strain Chip Supply

    Google is implementing strict new memory performance thresholds for Android applications on the Google Play Store, directly citing component shortages driven by the artificial intelligence data center buildout. In an update on the Android Developers Blog, Google detailed new app quality requirements targeting dynamic memory allocation, bitmap memory usage, and execution efficiency. The policy changes reflect supply chain shifts in semiconductor manufacturing, where massive demand for high-bandw

    AI
    2 min read

  14. OpenAI Details Custom Inference Chip 'Jalapeño' at Hot Chips, Targeting 700W Efficiency Against Nvidia Blackwell

    OpenAI has revealed architectural specifications and benchmark data for its first in-house artificial intelligence accelerator, code-named Jalapeño. Presented by hardware lead Richard Ho at the Hot Chips conference at Stanford University, the application-specific integrated circuit (ASIC) is engineered specifically for large language model inference rather than model training. Developed over an 18-month partnership with Broadcom and manufactured by TSMC, the chip targets large-scale token gener

    AI
    2 min read

  15. StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics

    StreamingLLM and Attention Sinks: Mathematical Foundations of Softmax Normalization Artifacts, Initial Token Anchoring, and Infinite-Context Rolling KV-Cache Mechanics In autoregressive Large Language Models (LLMs), long-context deployment is constrained by the quadratic compute complexity of attention and the linear memory growth of the Key-Value (KV) cache. For an LLM processing a sequence of length $L$ with batch size $b$, hidden dimension $d$, and $n_{kv}$ key-value heads across $N_L$ trans

    AI
    11 min read

  16. Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled RoPE Mechanics, and Matrix Absorption

    Multi-Head Latent Attention (MLA): Mathematical Foundations, Low-Rank KV Compression, Decoupled RoPE Mechanics, and Matrix Absorption Autoregressive inference in large language models is fundamentally constrained by memory bandwidth rather than floating-point computation throughput. During the generation phase, the transformer must load the Key-Value (KV) cache of all prior tokens from high-bandwidth memory (HBM) to on-chip SRAM for every newly generated token. As context lengths scale to hundr

    AI
    12 min read

  17. SoftBank in Talks to Acquire Majority Stake in Humanoid Robotics Maker 1X at $6 Billion Valuation

    SoftBank Group is in advanced discussions to acquire a controlling majority stake in humanoid robotics manufacturer 1X Technologies in a deal valuing the startup at approximately $6 billion, according to reporting by The Information and Reuters. 1X Technologies, founded by Bernt Øivind Børnich and backed by the OpenAI Startup Fund, designs autonomous bipedal and wheeled robots for industrial logistics and home assistance. Deal Structure and Strategic Context The negotiations mark a valuation

    AI
    2 min read

  18. DeepSeek Prepares $7.4B Funding Round at $74B Valuation to Build Custom Silicon and Compute Infrastructure

    Chinese frontier artificial intelligence laboratory DeepSeek is preparing to secure approximately $7.4 billion (50 billion yuan) in fresh funding at a $74 billion (500 billion yuan) pre-money valuation, according to reporting by The Wall Street Journal and Reuters. The funding round follows a period of rapid financial and technical expansion for the Hangzhou-based research lab. DeepSeek completed its initial external capital round earlier this summer at a valuation exceeding $50 billion. The la

    AI
    2 min read

  19. Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration

    Post-Training Quantization (PTQ): Mathematical Foundations of Optimal Brain Surgeon, GPTQ Hessian Inversion, AWQ Salient Scaling, and SmoothQuant Outlier Migration Serving modern large language models at scale requires addressing severe hardware constraints. In autoregressive generation, decoding is fundamentally bounded by memory bandwidth rather than floating-point computation throughput. Each generated token requires reading every parameter from High Bandwidth Memory (HBM) into SRAM to perfo

    AI
    10 min read

  20. Speculative Decoding: Mathematical Foundations, Distribution Preservation Proofs, Tree-Structured Verification, and Memory-Bandwidth Amortization

    Autoregressive large language model (LLM) generation suffers from an acute hardware efficiency mismatch during inference. While the prefill phase (processing the input prompt) processes tokens in parallel and achieves high arithmetic intensity on modern matrix accelerators, the decode phase (generating text token-by-token) is fundamentally memory-bandwidth bound. At small batch sizes, each generated token requires transferring the model's entire multi-billion-parameter weight matrix from High-Ba

    AI
    13 min read