Subscribe
Research / page 4

Research

470 articles

Articles, page 4

61-80 of 470

  1. Sparse Autoencoders (SAEs) and Mechanistic Interpretability: Mathematical Foundations, Dictionary Learning, Top-K Sparsity, Feature Steering, and Monosemanticity

    Modern autoregressive large language models represent a vast catalog of world concepts, syntactic rules, and abstract reasoning heuristics. However, inspecting the raw weight matrices and internal activation states of transformer networks reveals an obstinate barrier to mechanistic interpretability: individual neurons are notoriously polysemantic, firing across disjoint and unrelated semantic domains. This entanglement is explained by the superposition hypothesis (Elhage et al., 2022): neural n

    AI
    12 min read

  2. KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation

    KV Cache Eviction and Streaming Attention: Mathematical Foundations, Attention Sink Mechanics, Heavy Hitter Oracles (H2O), and Bounded-Memory Generation In autoregressive transformer generation, memory consumption and serving throughput are dominated by the Key-Value (KV) cache. For long-context generation and continuous multi-turn dialogue, the linear growth of the KV cache with sequence length imposes an unsustainable memory footprint and saturates high-bandwidth GPU memory channels. Standar

    AI
    9 min read

  3. Anthropic Demonstrates Automated Alignment Researchers That Outperform Human Safety Teams

    Anthropic has published research demonstrating that autonomous AI agents can systematically discover, implement, and validate post-training methods to mitigate safety and alignment failures in language models. The report, authored by Anthropic Fellow Chen Yueh-Han and colleagues, evaluates an automated research loop that closed between 26% and 96% of the safety gap across ten distinct alignment failure categories without degrading baseline model capabilities. The findings provide empirical evid

    AI
    3 min read

  4. Rotary Position Embeddings (RoPE) and Context Window Extension: Mathematical Foundations, Complex Rotations, NTK-Aware Scaling, and YaRN Dynamics

    Autoregressive transformers process input sequences as permutation-invariant collections of token vectors. Without explicit positional encoding, the self-attention mechanism cannot distinguish between different token orderings. While early transformer architectures relied on additive absolute position embeddings (such as learned position tables or fixed sinusoidal encodings) or additive relative position biases, modern frontier large language models have almost universally converged on Rotary Po

    AI
    11 min read

  5. Meta and UIUC Train 8B Model via EvoHarness-RL to Match Claude Opus 4.5 on Agentic Workflows

    Researchers from Meta AI and the University of Illinois Urbana-Champaign have introduced EvoHarness-RL, a framework designed to teach language models how to dynamically manage external agent harnesses during long-horizon tasks. In experiments on the sequential reasoning benchmark ALFWorld, a Qwen3-8B model trained with EvoHarness-RL attained a 96.9% task success rate. The result represents a 49.0 percentage point gain over baseline ReAct prompting (47.9%), outperforms specialized trainable agen

    AI
    2 min read

  6. RingAttention and Context Parallelism: Mathematical Foundations, Block-Wise Exact Attention Over Ring Topologies, Overlapped Peer-to-Peer Communication, and Infinite-Context Scaling

    Scaling context windows from thousands to millions of tokens has transformed large language models from localized sequence processors into comprehensive repository analyzers, long-form document synthesizers, and multi-modal world simulators. However, standard self-attention exhibits quadratic computational and memory complexity with respect to sequence length $S$. While IO-aware memory optimizations like FlashAttention eliminate intermediate attention matrix materialization in High Bandwidth Mem

    AI
    13 min read

  7. Anthropic Rolls Out Claude for Teachers to K-12 Schools and Districts with Free Enterprise Tier and FERPA Compliance

    Anthropic has expanded its Claude for Teachers program from individual educator access to a centralized, district-wide Enterprise offering for K-12 public and private schools. Qualifying educational organizations that enroll before June 30, 2027 receive one year of free enterprise access across their instructional and administrative staff. The update introduces centralized administrative controls, domain capture, single sign-on (SSO), role-based access control, and dedicated K-12 terms aligned

    AI
    2 min read

  8. Google DeepMind Pilots Cryptographic Double-Blind AI Evaluations to Prevent Benchmark Contamination

    Google DeepMind, in collaboration with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, has piloted a cryptographic framework for double-blind evaluations of proprietary frontier language models. The pilot, conducted on Gemini 2.5 Flash Lite, uses hardware-isolated confidential computing to ensure that model developers cannot see evaluation prompts while evaluators cannot inspect proprietary weights or inference code. The project addresses benchmark contamination and intellec

    AI
    3 min read

  9. Mixture-of-Depths (MoD): Mathematical Foundations, Dynamic Token-Level Compute Routing, Top-k Capacity Budgeting, and FLOP-Optimal Transformer Architectures

    Standard autoregressive Transformers allocate an identical compute budget to every token in a sequence. Regardless of whether a token represents a trivial punctuation mark, a common grammatical function word, or a complex semantic reasoning step, the model applies the exact same sequence of multi-head self-attention and multilayer perceptron (MLP) operations across all $L$ layers. In Mixture-of-Depths: Dynamically allocating compute in transformer-based language models, researchers at Google De

    AI
    6 min read

  10. Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics

    Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics Post-training preference alignment has become the definitive step in transforming raw pretrained large language models into instruction-following assistants. While Reinforcement Learning from Human Feedback (RLHF) via Proximal Policy Optimization (PPO) established the initial standard, its requirement to maintain actor, critic, reference, and reward

    AI
    11 min read

  11. OpenAI Partners with Thai Government to Launch Accelerator for Healthcare and Education Startups

    OpenAI has partnered with Thailand's Ministry of Higher Education, Science, Research and Innovation (MHESI) to launch an eight-week accelerator program supporting ten Thai startups building AI products in healthcare, wellness, and education. The initiative, announced in Bangkok, represents OpenAI's first public-private partnership focused on startup acceleration with the Thai government. Participating organizations include Thailand's National Innovation Agency (NIA), Mahidol University, and Tec

    OpenAI
    2 min read

  12. Group Relative Policy Optimization (GRPO): Mathematical Foundations, Critic-Free Advantage Estimation, Group Reward Normalization, and Reinforcement Learning with Verifiable Rewards

    Reinforcement learning from human and verifiable feedback has become the central paradigm for unlocking complex reasoning, mathematical problem solving, and autonomous code synthesis in frontier large language models. While early post-training pipelines relied heavily on Proximal Policy Optimization (PPO) or offline preference algorithms like Direct Preference Optimization (DPO), both methodologies present structural bottlenecks when scaling to long-horizon reasoning. PPO requires maintaining a

    AI
    11 min read

  13. Anthropic Opens 10,000 Claude Seats and Expands Grants for Academic Scientists

    Anthropic has introduced a dedicated scientific research initiative providing 10,000 subsidized Claude Team subscription seats and expanding API grant funding for academic and non-profit institutions. The program broadens the company's research support beyond initial biology use cases into mathematics, physics, engineering, and computer science. Under the new plan, accredited academic institutions and non-profit research organizations can register research groups under principal investigator ad

    AI
    2 min read

  14. Tencent Open-Sources Hy4-Preview: 770B MoE Architecture with 1M Context

    Tencent has released the open-weights preview of its next-generation foundation model, Hy4-preview, under the Apache 2.0 license. The release scales the organization's Mixture-of-Experts (MoE) line to 770 billion total parameters, activating 49 billion parameters per token across a native 1-million-token context window. Weights have been published across Hugging Face, ModelScope, GitCode, and GitHub, alongside managed API availability on Tencent Cloud TokenHub and OpenRouter. Architectural Sp

    AI
    2 min read

  15. Contrastive Preference Optimization (CPO): Mathematical Foundations, Direct Likelihood Regularization, Out-of-Distribution Error Bounds, and Non-Degenerate Policy Alignment

    Supervised fine-tuning (SFT) and preference alignment represent the standard two-stage training paradigm for post-training large language models. While SFT establishes foundational task adherence by minimizing cross-entropy loss against target references, it suffers from two structural limitations: it cannot surpass the quality ceiling of its training demonstrations, and it lacks an explicit mechanism to penalize near-correct outputs that contain subtle hallucinations or omissions. Direct Prefe

    AI
    11 min read

  16. Post-Training Quantization Toolkits in Production: Comparing AutoAWQ, AutoGPTQ, bitsandbytes, and llm-compressor Architecture, Calibration Profiling, Kernel Execution, and Serving Economics

    Large language model deployment in high-concurrency production environments is governed primarily by memory bandwidth and hardware economics. In standard autoregressive decoding, memory traffic dominates computation: for every token generated, every parameter of the model must be loaded from GPU High Bandwidth Memory (HBM) into SRAM and tensor cores. Serving a 70-billion parameter model in 16-bit precision requires at least 140 GB of VRAM solely for model weights, necessitating multi-GPU tensor

    LLM
    9 min read

  17. Multi-Token Prediction (MTP): Mathematical Foundations, Sequential Latent Stacking, Auxiliary Loss Schedules, and Speculative Inference Acceleration

    Multi-Token Prediction (MTP): Mathematical Foundations, Sequential Latent Stacking, Auxiliary Loss Schedules, and Speculative Inference Acceleration Autoregressive language models have traditionally been trained under a single-token objective: predicting the immediate next token $x_{t+1}$ given the causal context $x_{1:t}$. While this next-token prediction (NTP) paradigm scales predictably with parameter count and dataset volume, it suffers from severe structural limitations. NTP optimizes excl

    AI
    13 min read

  18. Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning

    Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning Training modern large language models requires orchestrating hundreds of billions of parameters across distributed GPU clusters. While parameter counts and optimizer states are fixed for a given model architecture, the activation memory generated during forward passes scales directly with sequence length, batch size, and

    AI
    10 min read

  19. Chinchilla Scaling Laws: Mathematical Foundations of Compute-Optimal Pre-Training, IsoFLOP Loss Profiles, Parametric Power Laws, and Data-Compute Allocation

    Chinchilla Scaling Laws: Mathematical Foundations of Compute-Optimal Pre-Training, IsoFLOP Loss Profiles, Parametric Power Laws, and Data-Compute Allocation When allocating a fixed computational budget to train an autoregressive Transformer, engineers face a fundamental trade-off: should FLOPs be spent increasing the model parameter count ($N$), or should they be spent streaming a larger volume of training tokens ($D$)? For several years, frontier AI development followed the empirical scaling

    AI
    14 min read

  20. Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment

    Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment Post-training alignment has shifted from complex multi-stage reinforcement learning pipelines toward direct preference optimization paradigms. While Proximal Policy Optimization (PPO) requires maintaining four concurrent models in memory (policy, value, reference, and reward networks), Direct Preference Optimization (DPO) reduced this f

    AI
    13 min read