Subscribe
Topic / page 15

LLM

Articles, page 15

281-300
  1. Beyond Naive RAG: Production Comparison of Self-RAG, CRAG, and Adaptive-RAG for Enterprise LLMs

    Beyond Naive RAG: Production Comparison of Self-RAG, CRAG, and Adaptive-RAG for Enterprise LLMs As retrieval-augmented generation (RAG) matures from prototype to production, teams face a critical choice: which advanced RAG variant best balances accuracy, latency, and operational complexity? Three leading approaches—Self-RAG, Corrective RAG (CRAG), and Adaptive-RAG—offer distinct trade-offs for enterprise deployment. Architectural Overview Self-RAG: Learning to Reflect Self-RAG (Asai et al.

    AI
    4 min read
  2. Matryoshka Representation Learning: Mathematical Foundations, Nested Subspace Optimization, and Adaptive Dimension Truncation in Modern Embedding Systems

    In modern deep learning systems, embedding vectors serve as the foundational intermediate representation for search, recommendation, retrieval-augmented generation (RAG), and zero-shot classification. Historically, these representations have been rigid: a model trained to produce 1536-dimensional or 2048-dimensional vectors outputs a fixed-width array for every input. Downstream systems must ingest, store, index, and compute distance metrics across the entire dimensional width, regardless of whe

    AI
    8 min read
  3. DeepSeek Generates $70.7M in Revenue with $106M Net Loss in First Seven Months of 2026

    Hangzhou-based artificial intelligence laboratory DeepSeek generated approximately 475 million yuan ($70.7 million) in revenue and recorded a net loss of $106 million during the first seven months of 2026, according to financial figures reported by The Information. The performance marks a roughly tenfold revenue surge compared to the lab's full-year 2025 revenue, alongside a modest contraction in net burn from the $139 million net loss reported for all of 2025. The disclosures provide a rare ac

    AI
    3 min read
  4. Activation-Aware Weight Quantization (AWQ): Mathematical Foundations, Salient Weight Protection, and Hardware-Efficient Low-Bit Inference

    Large language model inference during autoregressive generation is overwhelmingly memory bandwidth bound. While the prefill phase processes multiple prompt tokens in parallel with high arithmetic intensity, the token generation phase computes matrix-vector multiplications ($M=1$) for each sequential token. In this regime, the GPU spends the vast majority of its cycle budget streaming model parameters from High Bandwidth Memory (HBM) or GDDR into SRAM rather than performing floating-point arithme

    AI
    7 min read
  5. GPTQ: Mathematical Foundations, Optimal Brain Surgeon Inversion, and Second-Order Error Minimization in LLM Quantization

    Large language model inference during autoregressive generation is overwhelmingly memory-bandwidth bound. For batch size 1 decoding, each generated token requires streaming every parameter of a model from High Bandwidth Memory (HBM) into GPU SRAM and Tensor Cores. A 70-billion parameter model in 16-bit precision (FP16 or BF16) requires roughly 140 GB of VRAM, exceeding the capacity of a single 80 GB NVIDIA A100 or H100 GPU and demanding multi-GPU tensor parallelism solely to hold the model weigh

    AI
    14 min read
  6. SmoothQuant: Mathematical Foundations, Per-Channel Outlier Migration, and Hardware-Efficient W8A8 Inference in Large Language Models

    SmoothQuant: Mathematical Foundations, Per-Channel Outlier Migration, and Hardware-Efficient W8A8 Inference in Large Language Models Serving large language models (LLMs) in production environments presents two distinct hardware bottlenecks. During the autoregressive generation (decode) phase with small batch sizes, inference is memory-bandwidth bound, as billions of parameters must be streamed from High Bandwidth Memory (HBM) to on-chip SRAM for every generated token. Conversely, during the pro

    Quantization
    12 min read
  7. Odds Ratio Preference Optimization (ORPO): Mathematical Foundations, Monolithic SFT-Alignment, and Reference-Free Odds Penalties

    Post-training large language models typically relies on a decoupled, two-stage pipeline: Supervised Fine-Tuning (SFT) on curated instruction-response pairs, followed by Preference Alignment using algorithms such as Reinforcement Learning from Human Feedback (PPO) or Direct Preference Optimization (DPO). While effective, this multi-stage paradigm introduces substantial operational and computational friction. First, DPO and PPO require maintaining auxiliary reference models in GPU memory to preve

    AI
    9 min read
  8. QLoRA: Mathematical Foundations of 4-Bit NormalFloat Quantization, Double Quantization, and Paged Optimizers

    Fine-tuning large language models in full 16-bit precision requires substantial hardware infrastructure. For a 65-billion parameter model, storing weights in 16-bit brain floating point (BF16) requires 130 GB of VRAM. When optimizing those parameters with 32-bit AdamW, storing gradients (130 GB), first moments (260 GB), second moments (260 GB), and master weights (260 GB) drives the minimum memory footprint beyond 780 GB of VRAM, even before accounting for sequence activations. While Low-Rank A

    AI
    7 min read
  9. Projecting Conflicting Gradients (PCGrad): Mathematical Foundations, Orthogonal Projections, and Multi-Task Optimization in Deep Learning

    Projecting Conflicting Gradients (PCGrad): Mathematical Foundations, Orthogonal Projections, and Multi-Task Optimization in Deep Learning In modern machine learning systems, models rarely optimize for a single objective. Foundation models are trained simultaneously on diverse data distributions spanning natural language, source code, mathematical reasoning, and multimodal inputs. Similarly, post-training alignment pipelines must simultaneously optimize for helpfulness, factual accuracy, harmles

    AI
    11 min read
  10. Hierarchical KV Cache Offloading in Production LLM Serving: Host RAM, Local NVMe, Remote Storage, and LMCache Architecture

    High-concurrency large language model (LLM) serving faces an acute memory capacity bottleneck. While modern GPUs deliver high floating-point compute throughput, High-Bandwidth Memory (HBM) capacity remains severely constrained. In workloads involving multi-turn conversations, agentic coding loops, and long-document retrieval-augmented generation (RAG), Key-Value (KV) cache tensors rapidly consume available GPU memory, forcing engines to either reject requests, preempt running sequences, or recom

    AI
    9 min read
  11. LLM Gateways in Production: Comparing LiteLLM, Portkey, Kong AI Gateway, and Cloudflare AI Gateway Architecture, Fallback Cascades, and Serving Economics

    As enterprise production architectures scale from single prototype endpoints to multi-model agentic pipelines, coupling application code directly to foundation model provider APIs creates severe operational bottlenecks. Direct client integration leads to fractured telemetry, credential sprawl across microservices, unhandled upstream rate limits, and vendor lock-in. To isolate application logic from downstream provider volatility, engineering teams deploy dedicated LLM gateways. These specialize

    AI
    8 min read
  12. Synthetic Data Pipelines in Production LLM Post-Training: Architecture, Prompt Evolution, Quality Filtering, and Contamination Control

    Scaling supervised fine-tuning (SFT) and preference alignment (DPO, PPO, GRPO) through human annotation faces severe economic and operational constraints. Human annotation costs between $5.00 and $50.00 per complex instruction-response trajectory, exhibits significant variance across labeler cohorts, and scales linearly with dataset volume. The LIMA study by Zhou et al. (2023) showed that post-training requires strict data quality rather than raw volume, yet modern frontier models demand million

    AI
    6 min read
  13. Continuous Batching and Request Scheduling in Production LLM Serving: Comparing Orca, FastServe, Sarathi-Serve, and vLLM Architecture, Preemption Policies, Chunked Prefill Interleaving, and TTFT-TBT Trade-Offs

    Autoregressive large language model serving exhibits a fundamental architectural tension between compute utilization and latency guarantees. Standard deep learning inference pipelines rely on static request-level batching, where incoming queries are grouped into a fixed tensor, executed across forward passes until all sequences finish, and evicted simultaneously. In transformer-based text generation, static batching collapses serving efficiency. Because sequence lengths vary widely and token gen

    AI
    7 min read
  14. Anthropic Claude Autonomously Designs Validated Protein Binders Across 14 Targets

    Anthropic has released results from an autonomous de novo protein binder design campaign run by its Claude models, with wet-lab validation from two outside evaluators. Claude designed binders against 15 targets and succeeded against 14 of them, according to Anthropic. Across all campaigns, 27% of designs bound, and 49% of the designs Claude ranked first for each target bound, according to Anthropic's technical report. The campaign used Claude Opus 4.8 and Claude Mythos Preview inside Claude Sci

    AI
    3 min read
  15. Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules

    Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules Training modern large language models spanning hundreds of billions of parameters requires distributing model layers across multiple compute nodes. While Tensor Parallelism (TP) partitions individual matrix multiplications across GPUs within a single node, its reliance on high-frequency, all-reduce communications limits its practical scaling to the high-bandwidth domain of NVLi

    AI
    8 min read
  16. BitNet b1.58: Mathematical Foundations of 1.58-Bit Ternary Quantization, Absmean Scaling, and Multiplication-Free LLM Architectures

    Standard Large Language Models (LLMs) rely on 16-bit floating-point (FP16 or BF16) matrix multiplications for both pre-training and autoregressive inference. While post-training quantization techniques (such as GPTQ, AWQ, and dynamic FP8/INT4 scaling) compress weights after training, they often introduce perplexity degradation at ultra-low bitwidths ($<4$ bits) or still require floating-point dequantization at runtime. Quantization-aware pre-training architectures directly tackle this computati

    AI
    6 min read
  17. Test-Time Compute and Inference Search in Production: Comparing Best-of-N, Monte Carlo Tree Search, PRM-Guided Beam Search, and Dynamic Budget Allocation Architecture, Token Economics, and Serving Latency

    In standard production Large Language Model (LLM) deployments, generation compute is fixed: a prompt is processed during prefill, and the model generates an answer via greedy autoregressive decoding or low-temperature sampling. Under this paradigm, accuracy on complex reasoning, mathematical derivation, theorem proving, and code synthesis is bounded by the model parameter count and training data distributions. Recent empirical work demonstrates that scaling inference-time compute can achieve pe

    AI
    11 min read
  18. Context Parallelism in Production: Comparing DeepSpeed Ulysses, Megatron Sequence Parallelism, Ring Attention, and Unified Sequence Parallelism (USP) Architecture, Communication Overhead, and Scale-Out Economics

    Scaling context windows in Large Language Models (LLMs) from 8,192 tokens to 128k, 256k, and 1M tokens introduces severe memory and compute constraints. Standard Data Parallelism (DP) duplicates entire model activations per sequence, failing when a single sequence exceeds GPU VRAM. Tensor Parallelism (TP) splits weight matrices across GPUs, but scaling TP beyond a single 8-GPU node degrades throughput due to high-frequency, low-latency cross-GPU collective communications. Pipeline Parallelism (P

    AI
    10 min read
  19. Anthropic Unifies Memory Across Claude Chat and Cowork Sessions

    Anthropic has updated its architecture to unify the memory systems powering Claude chat and Claude Cowork, allowing context and user preferences established during conversational sessions to persist directly into workspace tasks. The update addresses a common friction point in AI workflow orchestration, where users previously had to manually rebrief agentic workspaces on project context already established in conversational threads. Real-Time In-Session Memory Extraction Prior iterations of C

    AI
    3 min read