Subscribe
Category / page 10

Chips & Infrastructure

314 articles

Articles, page 10

181-200 of 314

  1. Arithmetic Intensity and the Roofline Model: Why LLM Generation Is Memory-Bound and Prefill Is Compute-Bound

    Arithmetic Intensity and the Roofline Model in Large Language Models: Why Generation Is Memory-Bound and Prefill Is Compute-Bound Every optimization in modern large language model (LLM) serving exists to solve a single hardware reality: running a Transformer model is not one computational problem, but two completely distinct workloads operating on opposite sides of a physical performance barrier. When an LLM processes a prompt during the prefill phase, it saturates modern GPU Tensor Cores, ope

    AI
    9 min read

  2. Nscale Targets $3 Billion U.S. IPO as Soon as September

    Nscale Targets $3 Billion U.S. IPO as Soon as September Nscale signage during the Nvidia GTC conference in San Jose, California. London-based AI infrastructure startup Nscale is preparing for a U.S. initial public offering that could raise up to $3 billion, according to people familiar with the matter. The offering may launch as early as September, joining a wave of AI data center companies tapping public markets. Goldman Sachs and JPMorgan Chase are leading the IPO process, Bloomberg reporte

    AI
    2 min read

  3. NVIDIA Researchers Map Cross-Model KV Caches via Linear Ridge Regression to Accelerate Model Cascading Up to 25x

    In multi-agent systems and tiered model cascades, routing requests across different model sizes is a standard architecture for balancing inference cost and generation quality. However, switching models mid-session has historically imposed a severe compute tax: because each neural architecture maintains its own distinct internal representations, the receiving model cannot read the key-value (KV) states generated by the previous model. The target model must recompute the entire conversation histor

    AI
    4 min read

  4. Post-Training RL Frameworks in Production: Comparing verl, OpenRLHF, TRL, and DeepSpeed-Chat Architecture, Distributed Scheduling, and Serving Trade-Offs

    Post-training reinforcement learning (RL) has replaced standard supervised fine-tuning (SFT) as the primary mechanism for frontier model alignment and reasoning expansion. Whether running classic Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), or Reinforcement Learning with Verifiable Rewards (RLVR), modern post-training pipelines introduce an execution pattern distinct from standard pre-training: they alternate between high-th

    AI
    8 min read

  5. US Warns 35 Partner Countries to Choose Between Pax Silica and China's WAICO AI Coalition

    The U.S. Department of State is preparing formal diplomatic notices instructing 35 partner nations to select between Washington's AI alliance and Beijing's competing framework. According to a draft cable reviewed by Reuters and reported by The Decoder and CNBC, the U.S. warns that countries joining China's newly established AI initiative will be excluded from the U.S.-led Pax Silica coalition. The diplomatic draft states: "To be part of everything is to be part of nothing. Signature of the Pax

    AI
    2 min read

  6. Asynchronous Batch Inference in Production: Architecture, Queue Scheduling, and Cost Arbitrage

    Asynchronous Batch Inference in Production: Architecture, Queue Scheduling, and Cost Arbitrage Interactive AI applications require low Time-to-First-Token (TTFT) and high inter-token generation speed to maintain responsive user experiences. Achieving sub-second latency targets forces infrastructure teams to overprovision GPU capacity to absorb peak demand spikes. However, non-interactive production workloads (such as historical document processing, embedding generation, nightly model evaluation

    LLM
    7 min read

  7. Emergent Outlier Features in Large Language Models: Why Hidden Dimension Spikes Arise at Scale and How They Reshape Quantization

    Emergent Outlier Features in Large Language Models: Why Hidden Dimension Spikes Arise at Scale and How They Reshape Quantization When language models scale past approximately 6.7 billion parameters, their internal representations undergo a sharp qualitative phase transition. In smaller models (125M to 2.7B parameters), hidden state activations remain relatively compact, bounded within predictable normal distributions across all embedding dimensions. However, as demonstrated by Dettmers et al. (

    AI
    8 min read

  8. NVIDIA Takes Minority Stake in Cloverleaf Infrastructure to Accelerate AI Factory Site Development

    NVIDIA has taken a minority stake in Cloverleaf Infrastructure, a Houston-based developer of powered data-center sites, as part of a strategic partnership to accelerate the build-out of so-called AI factories. The companies did not disclose the size of the investment or its financial terms. Cloverleaf occupies the powered-land layer of the data-center supply chain. It identifies and prepares sites with grid-connected power, then delivers shovel-ready locations to infrastructure builders and ope

    AI
    2 min read

  9. Starcloud Raises $250M Series A Extension at $2.3B Valuation for Orbital AI Data Centers

    Orbital compute startup Starcloud has closed a $250 million Series A extension at a $2.3 billion post-money valuation, more than doubling its valuation from March 2026. The round brings the company's total funding to $450 million since its founding in 2024. The extension was led by Manhattan West, with partner Lauren Selig joining as a board observer. Participating investors include existing backers Benchmark, EQT, Soma, NFX, and 776, alongside new strategic investments from NVIDIA, Cisco Inves

    AI
    2 min read

  10. NVIDIA AVO Achieves 100% on ARC-AGI-3 with Claude Opus 5

    NVIDIA Research has published details on Agentic Variation Operators (AVO), a general-purpose autonomous agent architecture designed to sustain long-horizon engineering and reasoning workflows. In benchmark evaluations on ARC-AGI-3, an interactive reasoning suite that tests fluid intelligence without instructions or stated goals, AVO scored 100.00 Relative Human Action Efficiency (RHAE) across all 25 public-set environments and 183 levels using Anthropic's Claude Opus 5. The result highlights a

    AI
    3 min read

  11. Vision-Language Model Serving in Production: Visual Token Pruning, Encoder Caching, Dynamic Resolution, and Inference Economics

    Deploying Vision-Language Models (VLMs) into high-concurrency production environments introduces a distinct set of systems bottlenecks that text-only large language models do not exhibit. While text models ingest prompts with compact token densities, visual inputs require processing high-dimensional pixel arrays through vision encoders, expanding a single image into hundreds or thousands of visual tokens before autoregressive generation begins. In production architectures running models such as

    AI
    8 min read

  12. Confidential LLM Inference in Production: Hardware TEEs, GPU Enclaves, Attestation, and Serving Performance Trade-Offs

    Deploying large language models in multi-tenant cloud environments introduces a fundamental security boundary problem. Standard transport encryption (TLS) secures prompts in transit, and encryption-at-rest protects checkpoints on disk, but model weights, prompt tokens, and key-value (KV) caches exist in plaintext within system memory during active inference. For organizations processing protected health information (PHI), proprietary intellectual property, or regulated financial records, host op

    AI
    8 min read

  13. US Corporate AI Debt Surge Tests Investor Limits as Hyperscalers Issue $220B in 2026 Bonds

    The unprecedented capital requirements of frontier artificial intelligence infrastructure have triggered a record borrowing spree among US technology hyperscalers, testing the absorptive capacity of institutional credit markets. Total corporate debt issuance tied to AI hyperscalers has surpassed $220 billion in 2026 according to data from BNP Paribas, representing an eighteen-fold increase compared to the $12.5 billion issued during the same period in 2025. As mega-cap technology firms repeated

    AI
    2 min read

  14. Nvidia in Early Talks with South Korean AI Chip Designer Rebellions

    Nvidia is in early-stage discussions with South Korean AI semiconductor designer Rebellions regarding possible strategic tie-ups, including technology licensing partnerships, direct equity investments, or a full acquisition. Nvidia Chief Executive Officer Jensen Huang met with Rebellions co-founder and Chief Executive Officer Sunghyun Park at Nvidia headquarters in Santa Clara, California, according to reporting from Bloomberg citing people familiar with the matter. The discussions remain preli

    AI
    1 min read

  15. Speculative RAG in Production: Architecture, Multi-Perspective Sampling, and Draft Verification Trade-Offs

    Standard Retrieval-Augmented Generation (RAG) pipelines face an escalating trade-off between retrieval recall and inference latency. To ensure sufficient coverage for ambiguous or complex knowledge-intensive queries, production systems commonly ingest between 10 and 20 candidate passages per query. However, concatenating large document sets directly into the prompt context of a large language model introduces severe bottlenecks: quadratic attention scaling during the prefill phase, elevated Time

    Architecture
    7 min read

  16. Jacobi and Lookahead Decoding: How Parallel Fixed-Point Iteration Accelerates Autoregressive Inference Without Draft Models

    Standard autoregressive generation in large language models operates as a strictly sequential process. To generate a sequence of $K$ tokens, an inference engine must execute $K$ successive forward passes through the network. In single-request serving regimes (batch size 1), each forward pass is heavily memory-bandwidth bound: the GPU must stream billions of model parameters from High Bandwidth Memory (HBM) into on-chip SRAM to process a single token, leaving tensor compute cores severely underut

    AI
    7 min read

  17. LLM Output Calibration and Uncertainty Estimation in Production: Token Entropy, Semantic Clustering, and Risk-Controlled Abstention

    Production deployments of large language models frequently fail not because models lack capability, but because they lack reliable uncertainty estimation. Autoregressive language models generate hallucinations with the exact same fluent, assertive cadence as verified ground truth. When an enterprise application relies on downstream actions, database writes, or customer-facing advice, uncalibrated generations introduce severe operational risk. Treating raw token probabilities as calibrated confi

    AI
    6 min read

  18. Multi-Query Attention: How Single Key-Value Head Sharing Slashed Transformer Serving Bottlenecks

    Multi-query attention (MQA) is an architectural modification to the Transformer attention mechanism designed to resolve the memory bandwidth bottleneck during autoregressive token generation. First proposed by Noam Shazeer in the 2019 paper Fast Transformer Decoding: One Write-Head is All You Need, MQA alters the ratio of query, key, and value heads by sharing a single key head and a single value head across all query heads in each Transformer layer. While standard multi-head attention (MHA) pr

    AI
    7 min read

  19. Brazil Commits 44M to Sovereign AI, Splitting Compute Between Huawei, iFlytek, and US Suppliers

    Brazil has launched a national artificial intelligence infrastructure program totaling 2.3 billion reais ($444.2 million), dividing major compute and development contracts between Chinese and American technology providers to maintain technological sovereignty and avoid vendor lock-in. The initiative, announced by the administration of President Luiz Inacio Lula da Silva, allocates resources through the National Fund for Scientific and Technological Development (FNDCT). The funding structure est

    AI
    2 min read

  20. Dynamic Context Assembly and Token Budget Allocation in Production: Priority Packing, Prefix Cache Alignment, and Degradation Fallbacks

    In production large language model systems, prompt construction is frequently treated as simple string concatenation. Engineers assemble system instructions, tool schemas, retrieved document snippets, memory state, and multi-turn conversational history using template strings. In low-throughput prototypes, this approach functions adequately. In high-throughput production systems and multi-agent workflows, naive assembly introduces critical failure modes: abrupt context window overflows triggering

    LLM
    6 min read