Subscribe
Research / page 2

Research

470 articles

Articles, page 2

21-40 of 470

  1. Titans: Learning to Memorize at Test Time — Mathematical Foundations of Neural Long-Term Memory, Surprise-Driven Gradient Updates, Adaptive Forgetting, and Context Scaling

    Sequence modeling in large language models has long been caught in a fundamental trade-off between attention mechanisms and recurrent architectures. Softmax attention provides precise associative recall over all tokens in the active context window, but incurs quadratic computational and memory complexity, $O(N^2)$, with respect to sequence length $N$. Conversely, linear recurrent models, such as State Space Models (Mamba, Mamba-2) and linear Transformers (GLA, RetNet, DeltaNet), compress sequenc

    AI
    8 min read

  2. Open-Source Video Diffusion Models in Production: Comparing HunyuanVideo, CogVideoX, Mochi 1, and LTX-Video

    Generating temporally coherent, high-resolution video from natural language prompts has transitioned from UNet-based frame interpolation pipelines to scalable Diffusion Transformers (DiT) operating over compressed spatiotemporal latent spaces. The release of open-weight video foundation models has established production-viable alternatives to proprietary commercial APIs. However, engineering teams deploying video generation pipelines face stark trade-offs across transformer parameter scaling, 3D

    AI
    7 min read

  3. Gated Linear Attention (GLA): Mathematical Foundations, Data-Dependent Gating, Chunkwise Parallel Forms, and Sub-Quadratic Sequence Modeling

    Standard Transformer architectures rely on softmax self-attention, which exhibits quadratic time and memory complexity $O(N^2)$ with respect to sequence length $N$. While FlashAttention and its successors optimize memory access patterns to execute exact softmax attention efficiently within GPU SRAM, the fundamental quadratic compute barrier remains. Moreover, autoregressive generation requires maintaining a Key-Value (KV) cache that grows linearly with sequence length, creating a memory capacity

    AI
    9 min read

  4. Open-Source Text-to-Image Diffusion Models in Production: Comparing FLUX.1, Stable Diffusion 3.5, PixArt-Sigma, and Sana

    Production deployments of text-to-image generative systems have undergone a structural transition over the past two years. The classical convolutional U-Net architectures that powered Stable Diffusion 1.5 and Stable Diffusion XL have largely been superseded by transformer-based backbones. Modern open-weight models rely on Diffusion Transformers (DiT), Multimodal Diffusion Transformers (MMDiT), Linear DiTs, and Flow Matching formulations. Choosing an open-source text-to-image foundation model fo

    AI
    8 min read

  5. Self-Taught Reasoner (STaR) and Quiet-STaR: Mathematical Foundations, Rationalization Dynamics, and Bootstrapped Reasoning in Large Language Models

    Self-Taught Reasoner (STaR) and Quiet-STaR: Mathematical Foundations, Rationalization Dynamics, and Bootstrapped Reasoning in Large Language Models Chain-of-thought prompting enables autoregressive transformers to solve multi-step arithmetic, symbolic logic, and commonsense reasoning tasks by emitting intermediate tokens before generating a final answer. However, scaling chain-of-thought capabilities faces a fundamental data bottleneck: supervised datasets of high-quality, human-annotated reaso

    AI
    9 min read

  6. Prompt Injection Detection and Jailbreak Defense Frameworks in Production: Comparing Rebuff, Vigil, LLM Guard, and Meta Prompt Guard

    Prompt injection remains one of the primary operational vulnerabilities facing production Large Language Model (LLM) deployments. As systems expand from isolated chat interfaces into autonomous agents with access to databases, web browsing, APIs, and file systems, the blast radius of an unmitigated injection attack shifts from offensive output generation to unauthorized tool invocation and systemic data exfiltration. Standard input sanitization methods developed for traditional software enginee

    Security
    6 min read

  7. Chinese Mathematicians Prove 3D Ghys Conjecture Using Shanghai AI Lab's Shusheng-S2 and Lean Verification

    A research team led by mathematicians from Great Bay University and the Shanghai Artificial Intelligence Laboratory has published a complete proof of the three-dimensional Ghys conjecture, resolving a 31-year-old open problem in dynamical systems. The proof relied on Shanghai AI Lab's Shusheng-S2 multimodal scientific foundation model to explore deduction pathways, followed by formal computer verification in the Lean proof assistant conducted by researchers at Peking University. The achievement

    AI
    2 min read

  8. Differential Transformer (Diff Transformer): Mathematical Foundations of Differential Attention, Noise Cancellation, and Hallucination Suppression

    Differential Transformer (Diff Transformer): Mathematical Foundations of Differential Attention, Noise Cancellation, and Hallucination Suppression Standard Transformer architectures rely on softmax attention to route information between tokens in an autoregressive sequence. While effective across scale, softmax attention suffers from a structural tendency to allocate non-trivial probability mass to irrelevant context tokens. In long-context retrieval, multi-hop reasoning, and document summariza

    Education
    6 min read

  9. Diffusion Transformers (DiT): Mathematical Foundations, Patchification, Adaptive LayerNorm (adaLN-Zero), and Latent Diffusion Scaling

    Diffusion Transformers (DiT), introduced by William Peebles and Saining Xie in their 2023 paper Scalable Diffusion Models with Transformers, established a structural departure in generative visual modeling. For years, diffusion pipelines relied almost exclusively on convolutional U-Net backbones derived from Denoising Diffusion Probabilistic Models (DDPM) and Latent Diffusion Models (LDM). DiT replaced these multi-scale convolutional networks with standard isotropic Transformer architectures ope

    AI
    5 min read

  10. Open-Source Safety and Moderation Classifiers in Production: Comparing Llama Guard 3, ShieldGemma, WildGuard, and Prompt-Guard

    Production large language model (LLM) applications require structured defenses against adversarial jailbreaks, prompt injections, and policy-violating content. While guardrail orchestration frameworks handle workflow logic, the actual enforcement decisions rely on specialized safety classifier models. Open-source safety classifiers have diverged into distinct architectural paradigms, ranging from lightweight bidirectional encoders to full autoregressive decoders and multi-task refusal evaluator

    AI
    5 min read

  11. Reflection AI Releases Beam 1.0: 501B Open-Weight MoE Under Apache 2.0

    US AI research lab Reflection AI has released Beam 1.0, a 501-billion-parameter open-weight Mixture-of-Experts (MoE) foundation model licensed under Apache 2.0. The release includes weights in BF16, FP8, and NVFP4 precision formats alongside training harnesses and inference integration for vLLM, SGLang, and Ollama. Architecture and Sparse Compute Profile Beam is built across 52 layers with an interleaved local and global attention structure, maintaining a 3:1 ratio between sliding-window and

    Open Source
    2 min read

  12. Sigmoid Loss for Language Image Pre-Training (SigLIP): Mathematical Foundations, Pairwise Binary Classification, Softmax Normalization Removal, and Batch Size Scaling in Multimodal Vision-Language Models

    Executive Summary Multimodal foundation models rely heavily on contrastive representation learning to map visual features and natural language tokens into a unified embedding space. For years, the dominant paradigm was the InfoNCE objective popularized by CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which uses categorical cross-entropy with a softmax normalization over all paired elements in a mini-batch. While effective, the softmax normalization imposes a severe distributed syste

    AI
    8 min read

  13. Tree of Thoughts (ToT): Mathematical Foundations, Thought Decomposition, State Evaluation, and Tree Search Algorithms in Large Language Models

    Autoregressive large language models generate text left-to-right via token-level greedy or stochastic decoding. While Chain-of-Thought (CoT) prompting enables models to generate intermediate reasoning steps, standard inference remains constrained to a single linear trajectory. In complex problem-solving environments requiring combinatorial planning, heuristic evaluation, and lookahead, single-trajectory generation suffers from compounding errors and an inability to backtrack from early dead ends

    AI
    6 min read

  14. Open-Source LLM Benchmark Evaluation Harnesses in Production: Comparing LM Evaluation Harness, Lighteval, AlpacaEval 2, and Arena-Hard-Auto

    Standardized evaluation is the bedrock of LLM development, yet production evaluation pipelines face a split between two distinct paradigms: static, deterministic multi-task benchmarks evaluated via log-likelihood or exact string matching, and dynamic, open-ended instruction-following benchmarks evaluated via LLM-as-a-Judge pairwise win rates. Engineering teams building pre-training runs, fine-tuning recipes, and model alignment loops require distinct tooling for each paradigm. Four open-source

    AI
    5 min read

  15. Stochastic Decoding Strategies in Large Language Models: Mathematical Foundations of Temperature Scaling, Top-k, Nucleus (Top-p), Typical Sampling, and Min-p

    Autoregressive language models generate text sequentially by estimating a conditional probability distribution over a vocabulary at each decoding step. While model training optimizes cross-entropy loss over large corpora, the quality, diversity, and coherence of generated sequences depend directly on the decoding strategy used to select tokens from the output logits. Deterministic algorithms such as greedy search and beam search systematically fail in open-ended generation tasks, collapsing int

    AI
    8 min read

  16. Open-Source Sub-3B Small Language Models in Production: Comparing SmolLM2, Qwen2.5, Llama 3.2, and Gemma 2

    Deploying language models directly on client hardware, edge gateways, and low-latency routing layers requires balancing parameter count, memory footprint, and task-specific reasoning capability. The sub-3B parameter class has shifted from stripped-down toy models to highly capable engines trained on tens of trillions of tokens or distilled from frontier teachers. In production architectures, models in the 1B to 3B range serve three primary roles: speculative decoding draft models that accelerat

    AI
    6 min read

  17. Machine Unlearning in Large Language Models: Mathematical Foundations of Gradient Ascent, Negative Preference Optimization, and Representation Misdirection

    Machine Unlearning in Large Language Models: Mathematical Foundations of Gradient Ascent, Negative Preference Optimization, and Representation Misdirection Machine unlearning in large language models (LLMs) addresses the challenge of removing specific training data, copyrighted material, private identifiers, or hazardous capabilities from a pre-trained model without retraining the network from scratch. Because training frontier language models requires millions of GPU-hours and petabyte-scale c

    AI
    8 min read

  18. The Logit Lens and Tuned Lens: Mathematical Foundations, Residual Stream Projections, and Representation Dynamics in Transformers

    Large language models based on decoder-only transformer architectures process information through a sequence of stacked attention and multi-layer perceptron (MLP) blocks connected by a residual stream. While the final output distribution over the vocabulary is produced by projecting the final hidden state through an unembedding matrix, understanding how intermediate representations evolve across depth remains a central problem in mechanistic interpretability. The logit lens and the tuned lens p

    AI
    5 min read

  19. Open-Source Reasoning Models in Production: Comparing DeepSeek-R1 Distillations, QwQ-32B, Sky-T1, and Marco-o1

    Open-Source Reasoning Models in Production: Comparing DeepSeek-R1 Distillations, QwQ-32B, Sky-T1, and Marco-o1 The release of frontier reasoning models demonstrated that allocating test-time compute to explicit chain-of-thought generation significantly elevates problem-solving accuracy in mathematics, software engineering, and scientific reasoning. While proprietary systems conceal their training pipelines and reasoning traces, the open-source ecosystem has produced diverse approaches to replic

    Reasoning
    7 min read

  20. Query-Key Normalization (QK-Norm): Mathematical Foundations, Attention Logit Drift, Entropy Collapse, and Large-Scale Transformer Training Stability

    Large-scale Transformer pre-training at trillion-token scale frequently encounters sudden, irrecoverable training instabilities, often manifesting as gradient norm spikes and divergence in training loss. While gradient clipping, learning rate warmups, and pre-layer normalization (Pre-LN) mitigate optimization bottlenecks in shallow architectures, deeper and wider models exhibit an intrinsic failure mode in the standard scaled dot-product attention mechanism: unconstrained attention logit growth.

    AI
    6 min read