Subscribe
Category / page 2

Chips & Infrastructure

314 articles

Articles, page 2

21-40 of 314

  1. Intel Unveils Data Center GPU D-Series with 480GB LPDDR5X Memory for AI Inference

    Intel has introduced the Intel GPU D-series, a data center accelerator engineered specifically for large language model inference workloads. Unveiled at the Open Compute Project (OCP) Global Summit, the accelerator (previously codenamed Crescent Island) integrates up to 480GB of LPDDR5X memory, prioritizing raw memory capacity and power efficiency over ultra-high-bandwidth memory tiers. By equipping a single PCIe accelerator with 480GB of memory, Intel surpasses the onboard capacity of Nvidia's

    Hardware
    2 min read

  2. Open-Source Video Diffusion Models in Production: Comparing HunyuanVideo, CogVideoX, Mochi 1, and LTX-Video

    Generating temporally coherent, high-resolution video from natural language prompts has transitioned from UNet-based frame interpolation pipelines to scalable Diffusion Transformers (DiT) operating over compressed spatiotemporal latent spaces. The release of open-weight video foundation models has established production-viable alternatives to proprietary commercial APIs. However, engineering teams deploying video generation pipelines face stark trade-offs across transformer parameter scaling, 3D

    AI
    7 min read

  3. Open-Source Function-Calling Models in Production: Comparing Gorilla OpenFunctions-v2, Hermes 3, Functionary, and xLAM

    Deploying autonomous AI agents and automated API workflows requires reliable structured output generation and tool invocation. While general-purpose language models can be prompted to output JSON tool invocations, specialized function-calling fine-tunes provide higher schema compliance, lower parameter hallucination rates, and support for parallel tool invocation. Four open-source model families have emerged as prominent architectures for local and private function calling: Gorilla OpenFunction

    AI
    4 min read

  4. Microsoft Cancels 23,000 GPU Deal at Nscale UK AI Data Center Over Grid Delays

    Microsoft has canceled its contract to lease 23,000 Nvidia AI chips at Nscale's planned data center facility in Loughton, Essex, following mounting delays in securing local electrical grid power. The decision removes the sole anchor tenant from Nscale's flagship British campus just weeks before the infrastructure provider's planned initial public offering in New York. Grid Connection Bottlenecks The Loughton facility was announced as the site of the UK's largest AI supercomputer, intended to

    AI
    4 min read

  5. Open-Source Sub-3B Small Language Models in Production: Comparing SmolLM2, Qwen2.5, Llama 3.2, and Gemma 2

    Deploying language models directly on client hardware, edge gateways, and low-latency routing layers requires balancing parameter count, memory footprint, and task-specific reasoning capability. The sub-3B parameter class has shifted from stripped-down toy models to highly capable engines trained on tens of trillions of tokens or distilled from frontier teachers. In production architectures, models in the 1B to 3B range serve three primary roles: speculative decoding draft models that accelerat

    AI
    6 min read

  6. Self-Hosted AI Code Completion Engines in Production: Comparing Tabby, Continue, Refact.ai, and Twinny Architecture, Fill-in-the-Middle Pipelines, AST Context Retrieval, and Serving Latency

    Deploying AI-assisted code completion within air-gapped environments, regulated enterprises, and privacy-conscious engineering teams requires decoupling developer tooling from third-party cloud APIs. While agentic coding systems focus on multi-turn planning and repository-wide refactoring with relaxed latency budgets, inline code completion operates under strict real-time constraints. Developers expect ghost-text suggestions to appear within 100 to 200 milliseconds of pausing keystrokes. Delive

    AI
    7 min read

  7. Embedding and Reranker Serving Engines in Production: Comparing Hugging Face TEI, Infinity, FastEmbed, and vLLM

    Serving dense embedding and reranking models in production involves fundamentally different hardware and execution bottlenecks than autoregressive LLM text generation. While generative decoder models are memory-bandwidth-bound during single-token autoregressive decoding, embedding models (such as BGE-M3, ModernBERT, GTE-Qwen2, and Snowflake Arctic) and cross-encoder rerankers operate purely in encoder or forward-pass pooling mode. They are compute-bound and latency-critical. In retrieval-augmen

    AI
    6 min read

  8. Open-Source Text-to-Speech (TTS) Serving in Production: Comparing Kokoro-82M, F5-TTS, StyleTTS 2, and Fish Speech

    Deploying open-source text-to-speech (TTS) in production has shifted rapidly over the past year. Where teams previously defaulted to managed cloud APIs like ElevenLabs or OpenAI Audio due to the complexity and computational weight of self-hosted acoustic models, modern open-weight architectures offer distinct trade-offs across latency, zero-shot cloning fidelity, and inference cost. Selecting a self-hosted TTS engine requires navigating four fundamentally different architectural paradigms: ligh

    AI Voice
    6 min read

  9. Flash-Decoding: Mathematical Foundations, Split-KV Parallelism, Online Softmax Reduction, and Serving Latency Optimization

    Large language model inference consists of two distinct computational regimes: the compute-bound prefill phase and the memory-bandwidth-bound autoregressive decoding phase. While the prefill phase processes prompt tokens in parallel across dense matrix multiplication units, generation proceeds token by token. At each generation step, a single newly generated query vector must attend to the entire historical Key-Value (KV) cache stored in High-Bandwidth Memory (HBM). As context windows scale fro

    AI
    8 min read

  10. KV Cache Quantization in Production LLM Serving: Comparing FP8, INT8, INT4, KIVI, and KVQuant

    In high-throughput large language model serving, the primary hardware bottleneck during autoregressive token generation is memory bandwidth and GPU High Bandwidth Memory (HBM) capacity rather than raw compute FLOPS. While model weights remain static and are loaded once per forward pass, the Key-Value (KV) cache grows dynamically with sequence length and batch size. For a standard model in FP16 or BF16 precision, storing the KV cache requires 4 bytes per token across all layers and heads: Memor

    Inference
    6 min read

  11. Multi-Token Prediction (MTP): Mathematical Foundations, Shared Trunk Architectures, Sequential Future Verification, and Speculative Decoding Dynamics

    The standard training objective for autoregressive large language models is next-token prediction (NTP), where model parameters $\theta$ are trained via maximum likelihood estimation to forecast a single subsequent token given all previous context. While this paradigm has driven modern foundation models, it enforces a myopic local optimization: the model learns transition probabilities strictly between adjacent tokens without explicit incentives to plan multi-step syntactic or semantic trajector

    AI
    10 min read

  12. KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation

    KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation In autoregressive transformer generation, memory consumption and serving throughput are dominated by the Key-Value (KV) cache. For long-context generation and continuous multi-turn dialogue, the linear growth of the KV cache with sequence length imposes an unsustainable memory footprint and saturates high-bandwidth GPU memory channels. Standar

    AI
    9 min read

  13. Blue Owl Leads $2.4B AI Infrastructure Financing for IREN's Blackwell Ultra Campus

    Funds managed by alternative asset manager Blue Owl Capital have led a $2.4 billion compute equipment financing facility for IREN Limited. The capital is designated to fund purchases of Nvidia Accelerated Computing Infrastructure, including Blackwell Ultra GPUs, for IREN's Mackenzie data center campus in British Columbia, Canada. The financing package consists of a $1.2 billion senior secured term loan and $1.2 billion in senior secured notes. The facility was arranged at a 9.0% fixed interest

    AI
    2 min read

  14. Lambda Secures $1B in Private Debt to Finance Nvidia GPUs Leased to Microsoft

    AI cloud infrastructure provider Lambda has secured $1 billion in private, short-dated debt to purchase Nvidia GPU clusters that will be leased directly to Microsoft. The transaction, arranged by JPMorgan Chase, highlights an accelerating shift across the AI infrastructure sector toward structured debt and asset-backed financing to fund capital-intensive data center hardware. The short-dated maturity structure indicates that Lambda plans to amortize the debt rapidly using contracted lease reven

    AI
    2 min read

  15. Andreessen Horowitz Launches $1.1B Machine Age Fund for Physical AI Infrastructure

    Venture capital firm Andreessen Horowitz (a16z) has announced the close of a $1.1 billion dedicated fund named "Machine Age." The vehicle is structured to invest specifically in the physical infrastructure and hardware supply chains required to scale modern artificial intelligence workloads. The launch marks a significant capital allocation into capital-intensive physical systems by a firm historically recognized for software-centric investments. Target Hardware and Infrastructure Domains Ac

    AI
    1 min read

  16. Meta Deploys ABB and Kinova Robots to Automate Physical Hardware Tasks in AI Data Centers

    Meta has begun testing specialized robotic systems from industrial automation manufacturers to perform physical maintenance tasks across its AI data center fleet, according to internal reporting confirmed by current and former staff. The automation program, deployed across facilities including the company's Prometheus data center campus in New Albany, Ohio, uses robots to replace networking cables, power-cycle servers, reseat components, and perform routine hardware maintenance without human in

    AI
    2 min read

  17. Google DeepMind Pilots Cryptographic Double-Blind AI Evaluations to Prevent Benchmark Contamination

    Google DeepMind, in collaboration with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, has piloted a cryptographic framework for double-blind evaluations of proprietary frontier language models. The pilot, conducted on Gemini 2.5 Flash Lite, uses hardware-isolated confidential computing to ensure that model developers cannot see evaluation prompts while evaluators cannot inspect proprietary weights or inference code. The project addresses benchmark contamination and intellec

    AI
    3 min read

  18. USD.AI Secures $100M Stablecoin Debt Facility from Bullish to Fund GPU-Backed Infrastructure Loans

    Digital asset exchange operator Bullish has committed a $100 million stablecoin-denominated debt facility to USD.AI, an on-chain lending protocol that issues non-recourse debt collateralized by data center GPU hardware. The financing establishes an institutional credit channel connecting liquid stablecoin capital with the expanding compute infrastructure requirements of modern AI training and inference providers. Developed by Permian Labs, USD.AI structures asset-backed credit facilities specif

    Hardware
    2 min read

  19. South Korea Launches AI for All Initiative Treating Frontier Models as Public Utilities

    South Korea's Ministry of Science and ICT (MSIT) has launched the "AI for All" initiative, a government-sponsored project aimed at deploying nationwide, free-tier access to artificial intelligence chatbots and public administrative agents. The initiative structures generative AI capabilities and agent workflows as public utilities, establishing a subsidized access model for South Korean citizens. Consortia and Public Bidding Six major commercial consortia submitted bids to operate the public

    AI
    2 min read

  20. Open-Source Embedding Models and Serving Frameworks in Production: Comparing BGE-M3, NV-Embed-v2, GTE-Qwen2, and ModernBERT-Embed Architecture, Matryoshka Projections, Context Scaling, and Serving Economics

    Open-Source Embedding Models and Serving Frameworks in Production: Comparing BGE-M3, NV-Embed-v2, GTE-Qwen2, and ModernBERT-Embed Architecture, Matryoshka Projections, Context Scaling, and Serving Economics (TEI vs. Triton vs. vLLM) Text embeddings form the indexing and retrieval foundation for production Retrieval-Augmented Generation (RAG), semantic search, and agentic memory systems. While early production architectures relied almost exclusively on closed commercial APIs (such as OpenAI's te

    AI
    8 min read