Subscribe
Topic / page 34

LLM

Articles, page 34

661-680
  1. Grouped-Query Attention: How Head Sharing Slashed KV Cache Memory and Scaled LLM Serving

    Serving large language models at scale presents a fundamental hardware bottleneck: autoregressive decoding is bound by memory bandwidth rather than compute. While the initial prompt processing phase (prefill) operates as compute-bound matrix multiplications, token-by-token generation requires loading billions of cached attention states from GPU High-Bandwidth Memory (HBM) to on-chip SRAM for every single generated token. Multi-Head Attention (MHA), introduced in the foundational Vaswani et al.

    AI
    6 min read
  2. FlashAttention: How IO-Aware Tiling and Online Softmax Solved Transformer Memory Bottlenecks

    Standard multi-head attention is the fundamental computational primitive of modern autoregressive language models. While mathematically straightforward, the operation introduces a severe operational bottleneck as context windows scale. Naive implementations of scaled dot-product attention exhibit quadratic memory complexity $O(N^2)$ and quadratic memory access costs, bounding sequence lengths and leaving modern GPU tensor cores severely underutilized. FlashAttention, introduced by Tri Dao, Dani

    AI
    7 min read
  3. LLM Model Routing and Cascades: Architecture, Economics, and Quality Trade-Offs

    In modern enterprise AI systems, uniform model dispatch, sending all incoming user traffic to a single frontier large language model, is one of the most common architectural inefficiencies. Frontier models like GPT-4o and Claude 3.5 Sonnet provide industry-leading reasoning and code generation capabilities, but their inference costs range between $2.50 and $15.00 per million tokens. Conversely, smaller open-weights or distilled models, such as Llama 3.1 8B or GPT-4o-mini, execute at a fraction o

    AI
    6 min read
  4. Low-Rank Adaptation (LoRA): How Matrix Decomposition Made LLM Fine-Tuning Accessible

    Full-parameter fine-tuning of modern foundation models requires substantial compute and memory infrastructure. Adapting an open-weight 70-billion-parameter model using standard 16-bit precision and first-order adaptive optimizers like AdamW demands well over 1 terabyte of GPU memory. Low-Rank Adaptation (LoRA) bypassed this hardware bottleneck by framing task-specific weight updates as low-rank matrix decompositions. By freezing pre-trained weights and training small auxiliary rank decompositio

    AI
    6 min read
  5. Direct Preference Optimization: How Implicit Reward Modeling Simplified LLM Alignment

    Direct Preference Optimization (DPO) has emerged as a cornerstone of post-training alignment across modern open and proprietary language models, including Meta's Llama series, Mistral, and Qwen. While early alignment pipelines relied on complex reinforcement learning loops with separate reward models, DPO reformulates the preference optimization problem into a single, closed-form classification objective. By demonstrating that a language model implicitly defines its own reward function, DPO eli

    AI
    7 min read
  6. Speculative Decoding in Production: Architecture, Economics, and Serving Trade-Offs

    Standard autoregressive large language model (LLM) generation produces one token per forward pass. Because modern inference architectures must read tens of gigabytes of model weights from high-bandwidth memory (HBM) into SRAM to process each solitary token at low batch sizes, generation is strictly memory-bandwidth bound rather than compute bound. Speculative decoding alters this hardware equation. By pairing a fast draft mechanism with parallel verification by the primary target model, inferen

    LLM
    6 min read
  7. Rotary Position Embeddings: How Geometry Solved Long Context in Modern LLMs

    Rotary Position Embedding (RoPE) has become the standard positional encoding mechanism across modern large language models, including Meta's Llama series, Mistral, Qwen, and DeepSeek. Unlike earlier techniques that added positional vectors directly to token representations or modified attention matrices with relative distance penalties, RoPE encodes position through geometric rotations in the complex plane. This design enables models to compute relative token distances while processing individu

    AI
    6 min read
  8. Prompt Caching in Production: Architecture, Economics, and Invalidation Pitfalls

    As multi-turn AI agents, code generation assistants, and retrieval-augmented generation (RAG) pipelines expand in production, prompt sizes routinely exceed tens of thousands of tokens per request. In these architectures, the vast majority of tokens across consecutive API calls are static: system prompts, OpenAPI tool specifications, reference documents, and accumulated conversation history. Without optimization, inference engines must re-evaluate the full attention matrix for every token on eac

    AI
    5 min read
  9. How Mixture of Experts Works: Sparse Gating, Token Routing, and Load Balancing

    Large language models face a fundamental scaling trade-off: as parameter counts grow to absorb more world knowledge, the floating-point operations (FLOPs) required per token increase proportionally. In a standard dense Transformer, every single parameter participates in processing every incoming token. Mixture of Experts (MoE) decouples total model capacity from per-token compute cost. By replacing dense feed-forward layers with a dynamic pool of specialized sub-networks ("experts") and routing

    AI
    6 min read
  10. African Defense-Tech Startup Terra Industries Raises $52M Seed

    African defense-tech startup Terra Industries, founded in 2024, has raised a $52 million seed round from 8VC and other investors, and says it is on track to book $100 million in contracts. The company builds AI-driven security and surveillance systems for the African continent, focusing on drone-based monitoring, threat detection, and infrastructure protection. Terra's defense technology integrates computer vision, sensor fusion, and autonomous aerial platforms to provide real-time situational

    AI
    1 min read
  11. Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things

    Alibaba's Qwen research lab released Qwen 3.8 27B, a 27-billion-parameter Apache 2.0 licensed AI model available in 17GB GGUF format that can run on high-end consumer laptops. The model supports vision, long contexts, and tool calling. Our Qwen3.8 explainer shows where the 27B sits in the rest of the family. The over-thinking problem Qwen's documentation describes the model as defaulting to xhigh for reasoning effort, a setting designed for complex tasks demanding thorough analysis. In practi

    AI
    1 min read
  12. Cardano founder ships free code that strips Anthropic Claude watermarks

    Charles Hoskinson, the founder of the Cardano blockchain, has published a free open-source release that removes the watermarks Anthropic now attaches to Claude output. He calls it Anthropies. The release lands five days after Anthropic switched on invisible marking across every Claude product. The code appeared on GitHub under an Apache 2.0 license, so anyone can copy, modify, or sell it. Anthropic's mark is not a hidden character. A secret key steers tournament sampling, the step that breaks

    AI
    2 min read
  13. Study finds RL for reasoning changes only a few tokens, and you can copy the effect

    A new analysis of reinforcement learning for LLM reasoning suggests the field's biggest gains come from a surprisingly small change. RL is not rewriting how models think. It is nudging a tiny slice of the output. What the researchers measured Across several model families and common methods like GRPO and PPO, reinforcement learning reranks only 1.0 to 4.1 percent of tokens. The token it promotes is almost always already one of the base model's top five choices. Training, in other words, pushe

    AI
    1 min read
  14. DeepSeek V4 Flash tops charts but fails half its real agent tasks

    DeepSeek's V4 Flash has become the most-used AI model on OpenRouter and one of the highest-rated open-weight models available. In real agent tests, though, it finished barely more than half of the jobs it was given. The gap between leaderboard and workplace is the story. What Composio found The integration company Composio ran V4 Flash through eight agent harnesses, including Claude Code, Codex, and OpenCode, on 30 deliberately hard multi-step tasks. The tasks used live tools: Gmail, GitHub,

    AI
    2 min read
  15. Anthropic Investors Eye a $2 Trillion IPO

    Anthropic is preparing for a stock market debut that could value the five-year-old maker of Claude at more than two trillion dollars, according to investors cited by the Financial Times. Six investors in the company told the FT that rapidly rising revenue could support a valuation above two trillion dollars in a potential October listing, with some scenarios stretching toward three trillion. That would eclipse SpaceX and rank as the largest initial public offering on record. From a fraction o

    AI
    2 min read
  16. OpenAI Dissolves Preparedness Team, Reassigns Catastrophic Risk Work

    OpenAI shut down its "Preparedness" team at the end of July 2026, the team responsible for evaluating whether the company's AI models could pose severe or catastrophic risks. Its work on biological and cyber risks has been parceled out to existing teams, the Financial Times reported. The team, led by Dylan Scandinaro, assessed risks including biological weapons development, cyberattacks, and recursively self-improving AI systems. Scandinaro now focuses on safety risks from "recursively self-imp

    AI
    1 min read
  17. Lenovo's Q1 revenue jumps 43% on AI-driven demand

    Lenovo's Q1 revenue jumps 43% on AI-driven demand Lenovo reported first-quarter revenue of $26.9 billion, marking a 43% year-on-year increase and surpassing the $22.3 billion estimate. The PC maker attributed the growth to strong demand for artificial intelligence infrastructure and a resilient personal-computer business, alongside benefits from a global memory chip shortage. AI-related revenue specifically rose 60% year-on-year to reach $9.3 billion, highlighting the accelerating contribution

    AI
    1 min read
  18. AI is now embedded across the cyber intrusion lifecycle

    A new threat-intelligence report says AI is no longer just a tool attackers use to write malware. According to Gambit Security's "AI Across the Intrusion Lifecycle," large language models are now folded into the day-to-day mechanics of real intrusions, helping operators adapt as they move through compromised networks. The report, written by Gambit Director of Threat Intelligence Eyal Sela and researcher Nir Varon, examines three unrelated threat actors. Across the cases, the models built script

    AI
    2 min read
  19. Alibaba open-weights Qwen 3.8 with a 27B model that tops its bigger predecessor

    Alibaba's Qwen team announced the open weights for Qwen3.8 on August 14. The Qwen3.8-27B model card lists the Apache 2.0 license, and Hugging Face shows its weight files uploaded on August 13 (UTC). The move puts a capable new open model directly into the hands of researchers and companies that want to fine-tune and self-host it. The centerpiece is Qwen3.8-27B, a 27 billion parameter multimodal dense model. Qwen says it outperforms the larger Qwen3.7-Plus on coding and office tasks. The team al

    AI
    3 min read