Subscribe
Topic

Inference

How LLM inference works and how to make it faster and cheaper, from serving engines to decoding tricks and inference hardware. Recent posts compare vLLM, SGLang and TensorRT-LLM, explain speculative decoding, and cover KV cache quantization in production.

Articles

1-20
  1. Intel Unveils Data Center GPU D-Series with 480GB LPDDR5X Memory for AI Inference

    Intel has introduced the Intel GPU D-series, a data center accelerator engineered specifically for large language model inference workloads. Unveiled at the Open Compute Project (OCP) Global Summit, the accelerator (previously codenamed Crescent Island) integrates up to 480GB of LPDDR5X memory, prioritizing raw memory capacity and power efficiency over ultra-high-bandwidth memory tiers. By equipping a single PCIe accelerator with 480GB of memory, Intel surpasses the onboard capacity of Nvidia's

    Hardware
    2 min read
  2. Open-Source Function-Calling Models in Production: Comparing Gorilla OpenFunctions-v2, Hermes 3, Functionary, and xLAM

    Deploying autonomous AI agents and automated API workflows requires reliable structured output generation and tool invocation. While general-purpose language models can be prompted to output JSON tool invocations, specialized function-calling fine-tunes provide higher schema compliance, lower parameter hallucination rates, and support for parallel tool invocation. Four open-source model families have emerged as prominent architectures for local and private function calling: Gorilla OpenFunction

    AI
    4 min read
  3. Open-Source Text-to-Speech (TTS) Serving in Production: Comparing Kokoro-82M, F5-TTS, StyleTTS 2, and Fish Speech

    Deploying open-source text-to-speech (TTS) in production has shifted rapidly over the past year. Where teams previously defaulted to managed cloud APIs like ElevenLabs or OpenAI Audio due to the complexity and computational weight of self-hosted acoustic models, modern open-weight architectures offer distinct trade-offs across latency, zero-shot cloning fidelity, and inference cost. Selecting a self-hosted TTS engine requires navigating four fundamentally different architectural paradigms: ligh

    AI Voice
    6 min read
  4. Flash-Decoding: Mathematical Foundations, Split-KV Parallelism, Online Softmax Reduction, and Serving Latency Optimization

    Large language model inference consists of two distinct computational regimes: the compute-bound prefill phase and the memory-bandwidth-bound autoregressive decoding phase. While the prefill phase processes prompt tokens in parallel across dense matrix multiplication units, generation proceeds token by token. At each generation step, a single newly generated query vector must attend to the entire historical Key-Value (KV) cache stored in High-Bandwidth Memory (HBM). As context windows scale fro

    AI
    8 min read