Subscribe
Category / page 15

Chips & Infrastructure

314 articles

Articles, page 15

281-300 of 314

  1. Continuous Batching in Production LLM Serving: Iteration-Level Scheduling, Chunked Prefills, and Throughput Trade-Offs

    Static batching served as the standard execution paradigm for deep learning inference across computer vision and traditional natural language processing for years. In those domains, incoming requests typically feature fixed input dimensions and deterministic execution graphs. Autoregressive large language model serving breaks every assumption underlying static batching. Input prompts vary widely in token length, output generations terminate nondeterministically upon emitting an end-of-sequence t

    LLM
    6 min read

  2. Token Sampling in Large Language Models: How Temperature, Top-k, Top-p, and Min-p Govern Generative Decoding

    Large language models are autoregressive next-token predictors. During inference, the neural network processes an input context and outputs a vector of unnormalized scores, known as logits, across every token in its vocabulary. The mathematical pipeline that converts those raw logits into a single chosen token is the sampling and decoding stage. While model architecture and pre-training weights determine what a model knows, sampling algorithms govern how that knowledge is extracted. Small chang

    AI
    7 min read

  3. Anthropic Rolls Out Global AI Text Watermarking for Claude to Comply with EU AI Act

    Anthropic has started deploying model-level text watermarking across Claude to meet the regulatory requirements of the European Union AI Act. The company confirmed that because it currently lacks infrastructure to reliably partition model inference behavior by geographic jurisdiction, watermarking is being applied globally across all Claude web products and API endpoints. The update follows the formalization of the EU Code of Practice on Transparency of AI-Generated Content, signed in July 2026

    Anthropic
    2 min read

  4. Grouped-Query Attention: How Head Sharing Slashed KV Cache Memory and Scaled LLM Serving

    Serving large language models at scale presents a fundamental hardware bottleneck: autoregressive decoding is bound by memory bandwidth rather than compute. While the initial prompt processing phase (prefill) operates as compute-bound matrix multiplications, token-by-token generation requires loading billions of cached attention states from GPU High-Bandwidth Memory (HBM) to on-chip SRAM for every single generated token. Multi-Head Attention (MHA), introduced in the foundational Vaswani et al.

    AI
    6 min read

  5. FlashAttention: How IO-Aware Tiling and Online Softmax Solved Transformer Memory Bottlenecks

    Standard multi-head attention is the fundamental computational primitive of modern autoregressive language models. While mathematically straightforward, the operation introduces a severe operational bottleneck as context windows scale. Naive implementations of scaled dot-product attention exhibit quadratic memory complexity $O(N^2)$ and quadratic memory access costs, bounding sequence lengths and leaving modern GPU tensor cores severely underutilized. FlashAttention, introduced by Tri Dao, Dani

    AI
    7 min read

  6. OpenAI Signs 20-Year Lease for Multi-Gigawatt Ohio AI Campus with Nvidia Backstop

    OpenAI has signed a 20-year lease agreement to anchor a multi-gigawatt AI data center campus in Pike County, Ohio, developed by SoftBank subsidiary SB Energy. Under the tripartite agreement, Nvidia will serve as the exclusive compute infrastructure provider and supply financial guarantees covering the site's land, power, and shell development. The facility, designated the PORTS-Pike Technology Campus, is designed to support an initial 4.25 gigawatts of information technology load (IT-GW) dedica

    AI
    2 min read

  7. LLM Model Routing and Cascades: Architecture, Economics, and Quality Trade-Offs

    In modern enterprise AI systems, uniform model dispatch, sending all incoming user traffic to a single frontier large language model, is one of the most common architectural inefficiencies. Frontier models like GPT-4o and Claude 3.5 Sonnet provide industry-leading reasoning and code generation capabilities, but their inference costs range between $2.50 and $15.00 per million tokens. Conversely, smaller open-weights or distilled models, such as Llama 3.1 8B or GPT-4o-mini, execute at a fraction o

    AI
    6 min read

  8. Speculative Decoding in Production: Architecture, Economics, and Serving Trade-Offs

    Standard autoregressive large language model (LLM) generation produces one token per forward pass. Because modern inference architectures must read tens of gigabytes of model weights from high-bandwidth memory (HBM) into SRAM to process each solitary token at low batch sizes, generation is strictly memory-bandwidth bound rather than compute bound. Speculative decoding alters this hardware equation. By pairing a fast draft mechanism with parallel verification by the primary target model, inferen

    LLM
    6 min read

  9. Prompt Caching in Production: Architecture, Economics, and Invalidation Pitfalls

    As multi-turn AI agents, code generation assistants, and retrieval-augmented generation (RAG) pipelines expand in production, prompt sizes routinely exceed tens of thousands of tokens per request. In these architectures, the vast majority of tokens across consecutive API calls are static: system prompts, OpenAPI tool specifications, reference documents, and accumulated conversation history. Without optimization, inference engines must re-evaluate the full attention matrix for every token on eac

    AI
    5 min read

  10. OpenAI signs an 8-gigawatt Ohio data center lease backed by up to $105B from Nvidia

    OpenAI has signed a 20-year lease for the PORTS-Pike data center campus in Ohio, its largest infrastructure commitment to date, with Nvidia agreeing to backstop up to $105 billion of the project's value. The deal, struck with SoftBank's SB Energy, gives OpenAI roughly 8 gigawatts of IT capacity, about 10 gigawatts once cooling and supporting infrastructure are included. The site sits partly on a former U.S. Department of Energy uranium enrichment facility and draws power from a 9.2-gigawatt gas

    AI
    2 min read

  11. AI and Data Centers Dominate U.S. Campaign Trail in 2026 Elections

    Artificial intelligence has become a major campaign issue across the United States, with data centers and their impact on local electricity costs, water use, and land use dominating the conversation. A Washington Post analysis of more than 1,200 websites from active candidates found AI appears in nearly 40 percent of all races for the House, Senate, and governor seats. Democratic candidates bring up AI twice as often as Republicans, focusing on AI risks, regulation, and child safety. Republican

    Policy
    2 min read

  12. Liquid AI ships LFM2.5-VL-3B, a 3B vision model built for the edge

    Liquid AI has released LFM2.5-VL-3B, a roughly three billion parameter vision-language model designed to run on phones, laptops, and other edge hardware instead of in a data center. The model pairs the LFM2.5-2.6B text backbone with a SigLIP2 NaFlex image encoder. Unlike many recent models, it does not reason step by step. It answers directly, which keeps latency low for real-time and on-device use, and it handles a 32,000 token context. What it is built to do Liquid says the release adds fo

    AI
    2 min read

  13. French startup Kog says smarter software can pull 30x faster inference out of stock GPUs

    The race for faster AI inference has pushed labs toward custom silicon, but a French startup argues the cheapest speedup is already sitting in servers companies bought. Kog says it can reach roughly thirty times faster LLM inference on standard data-center GPUs using software alone. Kog demonstrated three thousand tokens per second per request on AMD MI300X and NVIDIA H200 GPUs in May, running a small two-billion-parameter model it has since open-sourced as Laneformer 2B. Chief executive Gael D

    AI
    2 min read

  14. OpenAI's Ultrafast mode pushes GPT-5.6 Sol to 750 tokens per second

    OpenAI's Ultrafast mode pushes GPT-5.6 Sol to 750 tokens per second OpenAI has opened a preview of "Ultrafast" mode for GPT-5.6 Sol, its flagship reasoning model. The mode streams up to 750 output tokens per second, which OpenAI frames as roughly 14 times the speed of the standard API, by running inference on Cerebras hardware. Cerebras and OpenAI signed a ten billion dollar partnership earlier this year, and Ultrafast is the first major use of that capacity. Access is limited for now to

    AI
    1 min read

  15. Liquid AI ships LFM2-VL-3B, a 3B vision model built for the edge

    Liquid AI has released LFM2-VL-3B, a 3 billion parameter vision-language model designed to run on edge hardware rather than in a data center. It is the largest model in the company’s LFM2-VL family, which previously topped out at 1.6B parameters, and is available now under the LFM Open License on Hugging Face and through Liquid AI’s LEAP platform. The model pairs Liquid AI’s densest text backbone, LFM2-2.6B, with a SigLIP2 400M NaFlex vision encoder. That combination lets it take in images at t

    AI Models
    2 min read

  16. Nvidia's Nemotron 4 aims for a trillion parameters, but China is already there

    Nvidia is building Nemotron 4, a new family of open-weight AI models meant to challenge the strongest freely available models, according to reporting by The Information cited by The Decoder. The largest model in the family is planned to reach at least one trillion parameters, twice the size of Nvidia's current Nemotron 3 Ultra. To train it, Nvidia has tripled its cloud spending on in-house model development to 28 billion dollars through 2031. The earliest the models could ship is this fall. C

    Nvidia
    1 min read

  17. Nvidia's Nemotron 3.5 Lightning bets on speed over size

    Nvidia has released Nemotron 3.5 Lightning, the first model in a new Nemotron 3.5 family, and it is playing a different game than most open-weight releases this year. Instead of chasing the highest intelligence score, Nvidia built a compact model that runs fast and cheap while still matching much larger rivals on standard benchmarks. A small model that keeps up Lightning carries 31.6 billion total parameters, but only 3.6 billion are active at any moment. It keeps the hybrid Mamba-Transformer

    Nvidia
    2 min read

  18. Nvidia and Amazon channel billions into AI power infrastructure

    Nvidia and Amazon are putting serious money into the power generation needed to keep AI data centers running, with two separate deals announced this week that underscore how energy has become a bottleneck for the industry. Nvidia is investing up to $3 billion in Lancium, the power infrastructure developer behind the OpenAI and Oracle data center in Texas, according to The Information. A roughly $2 billion stake would give Nvidia about 20 percent of the company, which is valued at around $10 bil

    Infrastructure
    2 min read

  19. Amazon Data Center Could Be Powered by One of the Nation's Most Polluting Power Plants

    Amazon is investing in a new natural-gas power plant in Pecos County, Texas, to supply a West Texas data center, and the project holds a permit that would allow it to emit more carbon dioxide than any coal plant in the country, according to The Verge and the New York Times. The plant, tracked as GW Ranch by Cleanview, a service that monitors data center power projects, would deploy 35 natural-gas turbines generating about 7.65 gigawatts. At least initially, the plant would not connect to

    Amazon
    1 min read

  20. Why DeepMind Researchers Are Leaving: Limited TPU Access, Bureaucracy, and a Chips Conflict

    As senior AI researchers continue to leave Google DeepMind, reporting is surfacing reasons behind the departures. Two recurring themes: limited access to Google's TPU chips, and a conflict of interest where Google Cloud sells those same chips to rivals the researchers are expected to beat. Semafor reported that CEO Demis Hassabis stepped back from daily operations about a year ago, turning those duties over to DeepMind chief Koray Kavukcuoglu. Hassabis was not forced out, but reportedly found m

    DeepMind
    2 min read