Subscribe
Topic / page 4

Infrastructure

Articles, page 4

61-80
  1. AI Agent Sandboxes and Safe Code Execution Runtimes in Production: Comparing E2B, Modal, Daytona, and Docker/gVisor Architecture, Isolation Boundaries, Startup Latency, and Serving Economics

    Autonomous AI agents, evaluation harnesses, and code generation pipelines increasingly execute untrusted code generated directly by Large Language Models. When an agent runs dynamic Python scripts, executes terminal shell commands, installs arbitrary packages via package managers, or browses the web, it introduces severe security and operational risks. These risks include host privilege escalation, kernel exploitation, container breakout vulnerabilities such as CVE-2024-21626, credential theft v

    AI
    8 min read
  2. KV Cache Optimization and Prefix Caching in LLM Serving: Comparing RadixAttention, Automatic Prefix Caching, and Static Context Sharing

    Serving large language models in multi-turn conversational agents, complex retrieval-augmented generation (RAG) pipelines, and few-shot reasoning workflows presents a fundamental memory and compute asymmetry. During autoregressive decoding, every newly generated token must attend to all previous tokens in the sequence. To avoid recomputing Key and Value projection matrices at each decoding step, inference engines store intermediate activations in high-bandwidth GPU memory (HBM) as the KV cache.

    AI
    7 min read
  3. LLM Inference Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI Architecture, KV Cache Topologies, Kernel Optimizations, and Serving Economics

    LLM Inference Engines in Production: Comparing vLLM, SGLang, TensorRT-LLM, and TGI Architecture, KV Cache Topologies, Kernel Optimizations, and Serving Economics Serving large language models in enterprise production has evolved beyond naive execution runtimes. As context windows expand to 128k+ tokens and agentic workloads generate complex multi-turn execution graphs, the efficiency of the underlying inference engine dictates both latency Service Level Objectives (SLOs) and hardware infrastruc

    LLM
    7 min read
  4. Navitas to Acquire Claros for Up to $232.8M to Expand Grid-to-xPU AI Power Infrastructure

    Navitas Semiconductor has entered into a definitive merger agreement to acquire power-management startup Claros in a deal valued at up to $232.8 million. The transaction brings vertical power delivery (VPD) and integrated voltage regulator (IVR) technology under Navitas's portfolio, targeting the physical bottlenecks limiting power transmission in modern AI hardware accelerators. Under the agreed terms, Navitas will provide approximately $216.0 million at closing through a mix of cash and Class

    Hardware
    2 min read
  5. AM Intelligence Orders 9,000 Nvidia Vera Rubin Systems for $8B AI Infrastructure Project

    Indian AI infrastructure platform AM Intelligence (AMI) has placed a binding purchase order for 9,000 Nvidia Vera Rubin computing systems. The procurement represents one of the earliest hyperscale commitments for Nvidia's next-generation Rubin architecture across Asia and anchors an $8 billion capital expenditure initiative to build 1 gigawatt (GW) of dedicated AI computing capacity. The first phase of the deployment will take place at AMI's upcoming data center facility in Hyderabad, India. Th

    AI
    2 min read
  6. Amazon to Shut Down Mechanical Turk on September 30 After 21 Years

    Amazon Web Services has announced that it will permanently shut down Amazon Mechanical Turk on September 30, 2026. The retirement brings an end to a 21-year-old crowdsourcing platform that served as an early foundation for modern machine learning dataset annotation and human-in-the-loop computing. In a formal notice posted to the Mechanical Turk portal, AWS stated that the decision followed an internal assessment of its service portfolio. The platform had previously halted new customer registra

    Amazon
    3 min read
  7. Hierarchical KV Cache Offloading in Production LLM Serving: Host RAM, Local NVMe, Remote Storage, and LMCache Architecture

    High-concurrency large language model (LLM) serving faces an acute memory capacity bottleneck. While modern GPUs deliver high floating-point compute throughput, High-Bandwidth Memory (HBM) capacity remains severely constrained. In workloads involving multi-turn conversations, agentic coding loops, and long-document retrieval-augmented generation (RAG), Key-Value (KV) cache tensors rapidly consume available GPU memory, forcing engines to either reject requests, preempt running sequences, or recom

    AI
    9 min read
  8. OpenAI Data Center Lead Chris Malone Departs Amid Infrastructure Realignment

    OpenAI head of data centers Chris Malone has departed the company, marking another high-level transition within the artificial intelligence lab as it restructures its infrastructure divisions ahead of a planned initial public offering. Malone, who joined OpenAI in March 2025 after five years leading data center strategy at Meta and over a decade as an engineer at Google, oversaw technical deployment across OpenAI's compute expansion efforts. Infrastructure Realignment Malone's exit coincides

    AI
    1 min read
  9. Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules

    Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules Training modern large language models spanning hundreds of billions of parameters requires distributing model layers across multiple compute nodes. While Tensor Parallelism (TP) partitions individual matrix multiplications across GPUs within a single node, its reliance on high-frequency, all-reduce communications limits its practical scaling to the high-bandwidth domain of NVLi

    AI
    8 min read
  10. Emerald AI Secures $150M Series A at $1.05B Valuation for Grid-Responsive Data Center Software

    Emerald AI has raised $150 million in a Series A funding round co-led by Energize Capital and DCVC, valuing the energy technology startup at $1.05 billion. The round drew substantial participation from strategic and corporate venture arms across the semiconductor, utility, and industrial sectors, including NVIDIA, Samsung Ventures, Siemens, GE Vernova, Aramco Ventures, Salesforce Ventures, RWE, JERA Ventures, and In-Q-Tel. Additional participants include Radical Ventures, Energy Impact Partners

    AI
    2 min read
  11. Model Merging in Production: Comparing Task Arithmetic, TIES-Merging, DARE, and SLERP Architecture, Weight Interference Mitigation, and Serving Economics

    Model Merging in Production: Comparing Task Arithmetic, TIES-Merging, DARE, and SLERP Architecture, Weight Interference Mitigation, and Serving Economics In production language model deployment, teams frequently encounter a fundamental operational tension: specialized task performance versus serving cost. Fine-tuning distinct base model instances on specific internal domains (such as code generation, legal document analysis, mathematical reasoning, and safety alignment) produces high task accur

    AI
    9 min read
  12. Stripe to Acquire AI Gateway Startup OpenRouter for a Reported $7.5B

    Stripe said on August 19 that it has agreed to acquire OpenRouter, the AI model routing platform, and did not disclose terms. Bloomberg first reported on August 16 that the companies had finalized a deal worth more than $7 billion. The New York Times, citing a person familiar with the matter, put the price at about $7.5 billion, with $1.5 billion allocated to OpenRouter's founders, CNBC reported. OpenRouter announced in May that it had raised a $113 million Series B at a reported $1.3 billion v

    AI
    2 min read
  13. Disaggregated Prefill and Decode in Production LLM Serving: Architecture, Network KV Cache Migration, Chunked Prefill Trade-Offs, and Asymmetric Hardware Economics

    Large language model serving systems have historically treated transformer execution as a homogeneous sequence of forward passes over a single unified GPU pool. Under continuous batching engines, incoming requests execute their prompt evaluation (prefill) and autoregressive token generation (decode) on the exact same accelerators, co-locating both phases within shared iteration batches. While continuous batching improves GPU compute utilization compared to static batching, co-locating prefill a

    Inference
    11 min read
  14. AI Cloud Provider Lambda in Talks to Raise $3B at $12B Valuation Ahead of IPO

    AI cloud infrastructure provider Lambda Inc. is in negotiations to raise up to $3 billion in a pre-IPO funding round that could value the company at $12 billion or higher, according to people familiar with the discussions reported by Bloomberg. The round represents an eightfold valuation step-up from February 2024, when Lambda secured $320 million in Series C funding at a $1.5 billion valuation. The company's annualized revenue is projected to exceed $1.5 billion in 2026, driven by continuous e

    Lambda
    2 min read
  15. Quintessent Secures $40M Series A to Scale Quantum Dot Comb Lasers for AI Clusters

    Optical interconnect startup Quintessent has raised $40 million in an oversubscribed Series A funding round to commercialize quantum dot comb lasers for AI data center networking. The round was led by Cycle Capital, with participation from Goldman Sachs XIG-Industry Ventures, optical networking provider Ciena, Hina Liberty Capital, Susquehanna International Group, InterVest, Safar Partners, M Ventures, Osage University Partners, Foothill Ventures, and Sierra Ventures. The financing follows an $

    AI
    2 min read
  16. SCX.ai Partners With DDN to Scale Australian Sovereign AI Inference Infrastructure

    Australian sovereign AI infrastructure provider SCX.ai has entered a strategic partnership with data storage systems vendor DDN to expand its domestic AI inferencing cloud. The collaboration combines SCX's ASIC-based compute clusters with DDN's Infinia data management platform, targeting enterprise, government, and scientific research workloads that must remain strictly within Australian jurisdiction. The agreement comes days after SCX completed a $40 million initial public offering on the Aust

    AI
    2 min read
  17. Continuous LLM Performance Profiling in Production: Roofline Models, Model FLOPs Utilization, Model Bandwidth Utilization, and Hardware Bottleneck Diagnostics

    Evaluating the runtime performance of large language model serving infrastructures requires looking beyond raw GPU metrics. Standard operating system utilities such as nvidia-smi report high GPU utilization percentages whenever compute cores or memory controllers are active, masking critical inefficiencies in memory access, communication, and kernel scheduling. A serving node running single-stream autoregressive decoding can report 100% GPU utilization while operating at less than 2% of the hard

    AI
    7 min read
  18. River AI Secures $1.1B Led by General Catalyst to Build Open-Weight Model Infrastructure

    River AI, an artificial intelligence startup founded by former xAI co-founder Igor Babuschkin, has secured $1.1 billion in early-stage funding to build an open-weight model stack and decentralized AI infrastructure platform. The financing round was led jointly by General Catalyst and public benefit corporation AMP PBC, with strategic participation from NVIDIA, AMD Ventures, Y Combinator, and Singapore sovereign fund Temasek. Babuschkin, whose prior engineering background spans OpenAI, Google De

    AI
    2 min read