Subscribe
Category / page 11

Chips & Infrastructure

314 articles

Articles, page 11

201-220 of 314

  1. Cross-Layer Attention: How Sharing Key-Value Tensors Across Transformer Layers Slashes KV Cache Memory

    In autoregressive large language models, the primary operational ceiling for serving long sequences and high batch concurrency is the Key-Value (KV) cache. During standard generation, every transformer layer computes and stores key and value activations for every token in the sequence to prevent quadratic recomputation during subsequent autoregressive decoding steps. While this mechanism transforms inference time complexity from $O(N^2)$ to $O(N)$ per generated token, it introduces a massive me

    AI
    9 min read

  2. Distributed Checkpointing in Production: PyTorch DCP, Asynchronous Staging, and Dynamic Topology Resharding

    Distributed Checkpointing in Production: PyTorch DCP, Asynchronous Staging, and Topology Resharding In multi-node distributed deep learning, checkpointing is often the largest single source of unforced downtime and degraded Model Flops Utilization (MFU). As models scale to tens or hundreds of billions of parameters across thousands of GPUs, saving model weights and optimizer states using traditional serialization primitives creates severe cluster stalls, memory exhaustion on root ranks, and rig

    AI
    6 min read

  3. GPU Cluster Scheduling in Production: Slurm vs. Kubernetes (Kueue/Volcano) vs. Ray

    Modern AI infrastructure represents a radical departure from traditional cloud computing. Standard cloud workloads (such as stateless microservices, web applications, and independent batch jobs) rely on fine-grained elasticity, independent container scheduling, and horizontal autoscaling. In contrast, distributed large language model (LLM) training and high-throughput inference pipelines violate virtually every assumption built into general-purpose cloud schedulers. A distributed 70B+ parameter

    AI
    9 min read

  4. Federal Judge Partially Overturns Conviction of Ex-Google Engineer Linwei Ding in AI Trade Secrets Case

    A federal judge in San Francisco has overturned the economic espionage convictions of former Google software engineer Linwei Ding, while upholding his conviction on seven counts of stealing proprietary artificial intelligence trade secrets. U.S. District Judge Vince Chhabria ruled on Thursday that federal prosecutors failed to present sufficient evidence demonstrating that Ding intended or knew his actions would benefit the government of China. Under federal statutes, establishing direct or ind

    AI
    2 min read

  5. TVA Board Approves Dedicated Data Center Rate Class to Shield Households from AI Compute Costs

    The Board of Directors of the Tennessee Valley Authority (TVA) voted on August 20, 2026, to establish a dedicated wholesale rate class for large data centers. The tariff restructuring is designed to insulate residential consumers and small commercial businesses from the escalating capital expenditures required to expand the power grid for artificial intelligence workloads. Approved during the board's quarterly meeting in Memphis, Tennessee, the package introduces targeted tariffs for facilities

    Infrastructure
    3 min read

  6. Fast Model Weight Loading in Production: Safetensors, Tensorizer, and Direct GPU Deserialization

    Fast Model Weight Loading in Production: Safetensors, Tensorizer, and Direct GPU Deserialization In modern large language model inference clusters, cold start latency is rarely bounded by GPU compute allocation. Instead, the operational bottleneck centers on storage I/O and weight deserialization. As foundation models scale from 70 billion to 405 billion parameters, raw weight footprints range from 140 GB to over 800 GB in standard 16-bit precision. On naive serving stacks, deserializing these

    AI
    6 min read

  7. Pre-Training Data Curation at Trillion-Token Scale: Comparing Datatrove, NeMo Curator, and Data-Juicer 2.0

    Preparing multi-trillion-token corpora for foundation model pre-training has become one of the most resource-intensive infrastructure challenges in modern artificial intelligence. While distributed training frameworks like Megatron-LM and TorchTitan have standardized model parallelism across GPU clusters, the upstream data engineering stack often remains fragmented across ad-hoc scripts, unoptimized Spark jobs, and brittle bash wrappers. Raw web dumps from Common Crawl contain massive quantitie

    AI
    8 min read

  8. Claude Multi-Model Outage Disrupts Claude.ai, Developer API, Claude Code, and Cowork

    Anthropic registered a multi-model service disruption on August 20, 2026, marking the provider's third recorded incident within a 24-hour window and continuing a sequence of elevated-error events across eight consecutive days. The incident, logged at 19:16 UTC on Anthropic's official status page, resulted in partial outages across the consumer web application (claude.ai), the core developer API (api.anthropic.com), and developer tooling including Claude Code and Claude Cowork. Affected Compon

    AI
    2 min read

  9. Broadcom Pursues $60B+ Debt Facility for Anthropic and Hyperscaler AI Chip Financing

    Broadcom is in advanced discussions with a syndicate of institutional lenders and private credit firms to raise more than $60 billion in debt to finance large-scale AI semiconductor infrastructure, according to reporting from Bloomberg. The capital package is structured to underwrite the manufacturing and deployment of custom AI processors, primarily benefiting frontier lab Anthropic alongside other enterprise compute customers. The financing structure highlights the growing role of complex pri

    AI
    2 min read

  10. Nvidia Strikes $6B Licensing Deal with Poolside and Backs Startup at $12B Valuation

    AI model developer Poolside has entered into a non-exclusive $6 billion licensing agreement with Nvidia, paired with a $1 billion direct equity investment that values the startup at $12 billion pre-money, according to an investor communication reported by Newcomer. As part of the transaction structure, Nvidia has extended job offers to 109 Poolside personnel, creating a hybrid licensing and talent transfer arrangement that mirrors recent structural deals across the frontier AI ecosystem. Deal

    AI
    2 min read

  11. Meta Emerges as Major Microsoft Azure AI Customer with Multi-Hundred-Million-Dollar Spend

    Meta Platforms has emerged as one of Microsoft Azure's largest artificial intelligence customers, spending hundreds of millions of dollars annually to access hosted AI models and inference compute, according to reporting by Bloomberg. The multi-hundred-million-dollar commitment underscores how current commercial demand for large-scale AI infrastructure remains intensely concentrated among frontier technology companies themselves. Bridging Internal Compute Gaps with Third-Party Infrastructure

    Meta
    2 min read

  12. Anthropic Modifies Enterprise Data Retention to Allow Customer Cloud Logging for Frontier Models

    Anthropic is preparing to revise the mandatory 30-day data retention requirement on its frontier models, allowing enterprise customers to retain logs on their own cloud infrastructure rather than storing conversation records on Anthropic servers. According to reporting from Bloomberg and Reuters, the upcoming safety architecture preserves the 30-day logging mandate for safety audits and abuse monitoring while shifting physical custody of the stored data into customer virtual private clouds. E

    Anthropic
    2 min read

  13. Weight Pruning in Large Language Models: How SparseGPT, Wanda, and Semi-Structured 2:4 Sparsity Compress Neural Networks Without Retraining

    Modern large language models require tens to hundreds of gigabytes of high-bandwidth memory to store billions of parameters. While post-training quantization compresses model footprints by reducing numerical precision from 16-bit floating point to 8-bit or 4-bit integers, weight pruning attacks model size along an orthogonal axis: setting redundant parameter values exactly to zero. Historically, pruning deep neural networks required iterative cycles of magnitude thresholding and compute-heavy r

    AI
    6 min read

  14. GPU Slicing in Production AI Systems: Comparing MIG, MPS, Time-Slicing, and Dynamic Partitioning

    GPU Slicing in Production AI Systems: Comparing MIG, MPS, Time-Slicing, and Dynamic Partitioning Modern production AI systems rarely deploy a single standalone large language model. Contemporary compound AI architectures rely on heterogeneous pipelines comprising embedding models (such as BGE or E5), cross-encoder rerankers, safety classifiers (such as Llama Guard), speculative decoding draft models, and vision-language encoders. While primary generation models typically require dedicated multi

    AI
    7 min read

  15. Liquid AI Ships LFM2.5-DSpark Draft Models for Up to 3.2x Faster Inference

    Liquid AI has released speculative decoding draft checkpoints for three models across its LFM2.5 series: LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and the mixture-of-experts model LFM2.5-8B-A1B. The release introduces small companion models designed to accelerate auto-regressive generation without altering final token distributions. The draft models are available in Safetensors and GGUF formats on Hugging Face, with immediate support implemented for SGLang and llama.cpp. Architecture and Draft Desig

    AI
    2 min read

  16. LLM Autoscaling and Cold Starts in Kubernetes: Architecture, KEDA Metrics, Model Weight Caching, and Ephemeral GPU Provisioning

    Autoscaling large language model workloads on Kubernetes presents a fundamentally different engineering problem than traditional stateless microservices. While web APIs scale on CPU utilization or request rate within seconds, LLM inference instances require specialized GPU accelerators, massive container images, multi-gigabyte weight tensors, and intensive runtime compilation before serving a single token. Without proactive architectural design, a cold-starting LLM pod on Kubernetes often requi

    Kubernetes
    6 min read

  17. Waymo Details Custom 5nm ASIC and Heterogeneous Compute Architecture for Robotaxis

    Alphabet subsidiary Waymo has disclosed the architectural specifications of the computing platform powering its sixth-generation autonomous driving system, revealing a custom 5-nanometer application-specific integrated circuit (ASIC) engineered specifically for front-end sensor ingestion and edge machine learning. The disclosure, authored by VP of Engineering Satish Jeyachandran and Compute Lead Daniel Rosenband, details how the company processes raw data from onboard sensor arrays in real time

    Autonomous Vehicles
    2 min read

  18. Speculative Tool Execution and Parallel Action Graphs in Production AI Agents: DAG Scheduling, Optimistic Concurrency, and Side-Effect Rollback

    Autonomous AI agents operating in complex environments (such as software engineering, scientific discovery, and automated workflow orchestration) face an operational bottleneck: execution latency. While foundation models have become faster at token generation, the standard agent interaction loop remains strictly serial. A model reasons, outputs a single tool call, halts generation, waits for the external environment to return a result, appends the output to context, and resumes reasoning. In mu

    AI
    5 min read

  19. Micron Launches Micron Research Labs with $10B Commitment for AI Memory Architecture

    Micron Technology announced on August 20, 2026, the creation of Micron Research Labs, a domestic long-horizon research institution headquartered in Boise, Idaho. Backed by a planned $10 billion investment across the next decade, the entity is designed to conduct precompetitive semiconductor and architecture research positioned upstream of commercial fabrication roadmaps. The funding operates independently from the more than $250 billion in domestic manufacturing and commercial development that

    Hardware
    2 min read

  20. Multi-Tenant LLM Serving in Production: Fair-Share Scheduling, Dynamic KV Cache Quotas, and Noisy Neighbor Isolation

    Operating a shared, multi-tenant large language model (LLM) serving cluster differs fundamentally from traditional stateless web tier hosting. In conventional microservices, tenants consume CPU cycles and static memory footprints in predictable, linear increments. In LLM serving, however, requests exhibit severe non-uniformity across multiple competing hardware dimensions: compute-bound prefill operations, memory-bandwidth-bound autoregressive decoding, and persistent High-Bandwidth Memory (HBM)

    Inference
    7 min read