Subscribe
Topic / page 21

LLM

Articles, page 21

401-420
  1. The QK and OV Circuits in Transformers: How Bilinear Attention Routing and Subspace Projections Move Information

    The QK and OV Circuits in Transformers: How Bilinear Attention Routing and Subspace Projections Move Information The standard mathematical presentation of multi-head self-attention, introduced in Vaswani et al. (2017), describes the layer as a sequence of matrix projections followed by scaled dot-product operations, concatenation, and an output projection. While computationally efficient for parallel GPU hardware, this formulation obscures the fundamental linear mechanics governing how transfor

    AI
    9 min read
  2. Decentralized and Peer-to-Peer LLM Inference in Production: Architecture, Ring Memory Partitioning, and Network Latency

    Decentralized and Peer-to-Peer LLM Inference in Production: Architecture, Ring Memory Partitioning, and Network Latency Frontier open-weight models such as Llama 3.1 405B, DeepSeek-V3, and Command R+ have expanded model capabilities, but their parameter scales exceed the physical memory limits of individual consumer and edge workstations. Running a 405-billion parameter model in 16-bit precision requires over 810 GB of memory, and even 4-bit quantized variants require roughly 230 GB of contiguo

    AI
    7 min read
  3. Discrete Diffusion in Large Language Models: How Continuous-Time Markov Chains, Absorbing States, and Score Entropy Challenge Autoregressive Generation

    The dominance of autoregressive architectures in large language models rests on a fundamental mathematical formulation: the chain rule of probability. By factoring the joint distribution of a sequence into a product of conditional probabilities, $p(x) = \prod_{i=1}^N p(x_i \mid x_{<i})$, autoregressive models reduce text generation to sequential next-token prediction. While this left-to-right causal factorization has scaled effectively across compute regimes, it imposes rigid operational constr

    AI
    8 min read
  4. AI Agents Surpass Humans on OpenRouter as Agentic Token Usage Jumps 14x

    Autonomous AI agents have overtaken human users as the primary consumers of language model compute on OpenRouter, with agentic token volume surging fourteenfold over the past six months. Data published by OpenRouter analyst Peter Walker indicates that February 6 marked the permanent inflection point where token consumption by automated agents exceeded direct human API traffic. Since that threshold, agentic token volume on the multi-model gateway has climbed from 0.51 trillion to 7.3 trillion to

    AI
    2 min read
  5. Continuous Pre-Training in Production: Domain Adaptation, Replay Buffers, Learning Rate Restarts, and Catastrophic Forgetting Mitigation

    Continuous Pre-Training in Production: Domain Adaptation, Replay Buffers, Learning Rate Restarts, and Catastrophic Forgetting Mitigation Adapting general-purpose foundation models to specialized enterprise domains (such as clinical medicine, corporate law, quantitative finance, and proprietary software codebases) presents a fundamental architectural challenge. While Retrieval-Augmented Generation (RAG) and Supervised Fine-Tuning (SFT) remain standard first-line approaches, both exhibit severe s

    LLM
    7 min read
  6. Hybrid SSM-Transformer Architectures: How Interleaving Attention and Recurrence Solves the State-Retrieval Trade-Off

    Hybrid SSM-Transformer Architectures: How Interleaving Attention and Recurrence Solves the State-Retrieval Trade-Off Autoregressive language models face a fundamental tension between inference efficiency and long-context retrieval capacity. Pure Transformer architectures scale quadratic computational complexity during sequence prefill and linear key-value (KV) cache memory consumption during autoregressive token generation. Conversely, pure State Space Models (SSMs) and linear recurrent neural

    AI
    8 min read
  7. Study: Why Labor-Saving LLMs Incline Scientists to Do More Work Less Well

    A theoretical study published by researchers from Princeton University, the University of Washington, and collaborating institutions models how large language models alter researchers' time allocation across projects. The authors find that by reducing time friction across different stages of the research lifecycle, AI assistants increase the opportunity cost of researcher time, creating economic incentives to publish a higher volume of less thoroughly refined papers. The paper, titled The unint

    AI
    2 min read
  8. Hallucination Detection and Faithfulness Verification in Production RAG: Architecture, NLI Claim Decomposition, and Runtime Guardrail Economics

    Retrieval-Augmented Generation (RAG) is commonly deployed under the assumption that grounding generation in retrieved passages eliminates factual inaccuracies. In practice, grounding provides an evidence boundary but does not guarantee factual fidelity. For why models hallucinate in the first place, see our AI hallucinations explainer. Production language models regularly synthesize claims absent from the retrieved context (extrinsic hallucinations) or directly assert statements conflicting with

    AI
    6 min read
  9. Sparse Attention and BigBird: How Window, Global, and Random Graphs Preserve Turing Completeness in Linear Time

    Standard self-attention in transformer architectures scales quadratically with sequence length. Computing full pairwise interactions between n tokens requires evaluating an n x n attention matrix, yielding O(n^2) computational complexity and memory consumption. While hardware accelerators and IO-aware tiling algorithms like FlashAttention optimize memory traffic, the quadratic compute and KV footprint remains a barrier for processing long contexts, document-level summarization, and genomic seque

    AI
    7 min read
  10. Oxford Study Details Chinese Gray-Market Proxies Reselling Claude Tokens at 90% Discounts

    An investigation by the Oxford China Policy Lab reveals that Chinese developers routinely access Anthropic's frontier Claude models at discounts between 70% and 90% below list price, bypassing geographical blocks, payment filters, and biometric identity verification through a decentralized network of API proxies known locally as "transfer stations" (中转站). The analysis, authored by Oxford researcher Zilan Qian and published via ChinaTalk, outlines the modular supply chain and economic mechanics

    AI
    3 min read
  11. Low-Precision Quantization Kernels in Production: Comparing Marlin, ExLlamaV2, FlashInfer, and BitBLAS

    Low-Precision Quantization Kernels in Production: Comparing Marlin, ExLlamaV2, FlashInfer, and BitBLAS Architecture, Memory Bandwidth, and Decoding Throughput Autoregressive large language model (LLM) serving operates under two distinct compute regimes: a compute-bound prefill phase and a memory-bandwidth-bound decode phase. While processing the initial prompt involves matrix-matrix multiplications (GEMM) with high arithmetic intensity, generating tokens one by one requires matrix-vector multip

    AI
    9 min read
  12. xLSTM: How Exponential Gating and Matrix Memory Scale Recurrent Neural Networks

    xLSTM: How Exponential Gating and Matrix Memory Scale Recurrent Neural Networks For over two decades following its introduction by Hochreiter and Schmidhuber (1997), the Long Short-Term Memory (LSTM) network served as the dominant architecture for sequence modeling. By introducing the constant error carousel and multiplicative gating, LSTMs mitigated the vanishing gradient problem that plagued vanilla recurrent neural networks. However, the emergence of the Transformer architecture (Vaswani et

    AI
    10 min read
  13. LLM Inference on AMD ROCm in Production: MI300X Architecture, Triton Kernel Parity, and vLLM Serving Benchmarks

    LLM Inference on AMD ROCm in Production: MI300X Architecture, Triton Kernel Parity, and vLLM Serving Benchmarks Serving frontier large language models in enterprise production has historically been synonymous with NVIDIA CUDA infrastructure. However, the deployment of AMD Instinct MI300X accelerators across tier-one hyperscalers and neoclouds has established a viable alternative for high-throughput inference fleets. With 192 GB of high-bandwidth memory (HBM3) and 5.3 TB/s of peak theoretical m

    Inference
    5 min read
  14. Identity Preference Optimization (IPO): How Exact Loss Inversion Prevents Overfitting in Direct Alignment

    The post-training alignment of large language models underwent a structural shift with the introduction of Direct Preference Optimization (Rafailov et al., 2023). By reparameterizing the closed-form optimal policy under a Kullback-Leibler (KL) constrained Reinforcement Learning from Human Feedback (RLHF) objective, DPO eliminated the need to fit an explicit reward model or maintain complex actor-critic rollout loops. However, standard DPO introduces a distinct mathematical pathology: under dete

    AI
    7 min read
  15. Infinite Agentic Loops in Production: Architecture, Feedback Topologies, and Bound Verification

    Autonomous AI agents have transitioned software architectures from static, single-turn request-response patterns into stateful, iterative execution loops. Built around foundational paradigms such as ReAct (Yao et al., 2022) and implemented across frameworks including LangGraph, CrewAI, AutoGen, and the OpenAI Agents SDK, agents repeatedly perceive environmental state, reason over intermediate goals, dispatch tool invocations, observe execution outputs, and append new observations back into their

    AI
    8 min read
  16. Noise-Contrastive Estimation and InfoNCE: How Partition Function Estimation and Mutual Information Lower Bounds Power Modern AI

    Noise-Contrastive Estimation and InfoNCE: How Partition Function Elimination and Mutual Information Lower Bounds Power Modern AI In statistical machine learning and generative modeling, evaluating the exact probability of an observed event frequently requires calculating a normalizing constant known as the partition function. For continuous spaces or discrete spaces with high cardinality, such as a natural language vocabulary spanning over one hundred thousand tokens or high-dimensional pixel d

    AI
    10 min read
  17. Agent Task Planning and Decomposition in Production: Plan-and-Solve vs. ReAct, Hierarchical Task Graphs, and Dynamic Replanning Architectures

    Autonomous AI agents deployed in production environments frequently fail when tasks require long-horizon reasoning across dozens of sequential tool calls. While single-turn tool calling is well-handled by modern frontier models, multi-step workflows introduce compounding failure modes: plan drift, unrecoverable tool exceptions, context window saturation, and premature task termination. Building resilient agent systems requires moving beyond simple prompt-driven loops. Production engineering has

    AI
    5 min read
  18. Performers and FAVOR+: How Positive Orthogonal Random Features Linearize Transformer Attention

    The quadratic complexity of standard self-attention has remained a central computational ceiling in Transformer architectures. Because standard attention computes pairwise similarity across all token pairs in a sequence of length $L$, memory consumption and compute scale as $O(L^2)$. For long contexts, high-resolution visual tokens, and biological sequence modeling, this quadratic bottleneck forces strict sequence truncation or aggressive hardware partitioning. In Rethinking Attention with Perf

    AI
    9 min read
  19. Nvidia Expands Nemotron Open-Weight Push to Counter Chinese Labs Under $6B Poolside Deal

    Nvidia Expands Nemotron Open-Weight Push to Counter Chinese Labs Under $6B Poolside Deal Nvidia plans to use the infrastructure and engineering team acquired through its $6 billion deal with AI startup Poolside to build frontier open-weight models under its Nemotron family, according to reporting from the Wall Street Journal. The initiative aims to counter the rapid global adoption of Chinese open-weight systems like DeepSeek-V3, Moonshot AI's Kimi K3, and Alibaba's Qwen series, while offering

    AI
    2 min read