Subscribe
Subscribe
Archive / page 17 of 49

Latest articles, page 17

  1. Continuous Batching and Request Scheduling in Production LLM Serving: Comparing Orca, FastServe, Sarathi-Serve, and vLLM Architecture, Preemption Policies, Chunked Prefill Interleaving, and TTFT-TBT Trade-Offs

    Autoregressive large language model serving exhibits a fundamental architectural tension between compute utilization and latency guarantees. Standard deep learning inference pipelines rely on static request-level batching, where incoming queries are grouped into a fixed tensor, executed across forward passes until all sequences finish, and evicted simultaneously. In transformer-based text generation, static batching collapses serving efficiency. Because sequence lengths vary widely and token gen

    AI
    7 min read

  2. Figure AI Unveils Index Platform with 16 Million Crowdsourced Videos for Robot Foundation Models

    Humanoid robotics startup Figure AI has launched Index, a global crowdsourced data collection platform engineered to capture real-world human task demonstrations at scale. Operating in stealth for four months prior to its public unveiling, the platform has compiled 16 million video demonstrations from contributors across 108 countries, generating embodied training data for Figure's physical AI foundation models. The initiative directly targets the primary bottleneck in scaling embodied AI: the

    AI
    2 min read

  3. Anthropic Claude Autonomously Designs Validated Protein Binders Across 14 Targets

    Anthropic has released results from an autonomous de novo protein binder design campaign run by its Claude models, with wet-lab validation from two outside evaluators. Claude designed binders against 15 targets and succeeded against 14 of them, according to Anthropic. Across all campaigns, 27% of designs bound, and 49% of the designs Claude ranked first for each target bound, according to Anthropic's technical report. The campaign used Claude Opus 4.8 and Claude Mythos Preview inside Claude Sci

    AI
    3 min read

  4. Model Routing and Cascades in Production: Comparing RouteLLM, FrugalGPT, Embedding Classifiers, and Verifier Cascades

    Model Routing and Cascades in Production: Comparing RouteLLM, FrugalGPT, Embedding Classifiers, and Verifier Cascades Enterprise LLM deployments face a persistent structural inefficiency: the uniform routing of all incoming queries to flagship frontier models. Commercial API pricing and self-hosted GPU infrastructure costs span two orders of magnitude between lightweight models (such as Llama 3.1 8B, GPT-4o-mini, and Claude 3.5 Haiku at $0.15 to $0.30 per million tokens) and frontier reasoning

    Inference
    9 min read

  5. Skild AI Introduces S1 Robotics Foundation Model with In-Context Video Prompting

    Robotics foundation model startup Skild AI has unveiled S1, a foundation model capable of learning physical manipulation tasks unseen during pretraining directly from a single video demonstration prompt without fine-tuning. Traditional robotic adaptation typically requires extensive task-specific teleoperation data, domain randomization, and model fine-tuning before a system can reliably execute novel actions. S1 employs in-context prompting to translate visual demonstrations directly into real

    Robotics
    1 min read

  6. Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules

    Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules Training modern large language models spanning hundreds of billions of parameters requires distributing model layers across multiple compute nodes. While Tensor Parallelism (TP) partitions individual matrix multiplications across GPUs within a single node, its reliance on high-frequency, all-reduce communications limits its practical scaling to the high-bandwidth domain of NVLi

    AI
    8 min read

  7. Anthropic Launches $5M Grant Program to Fund AI Wellbeing Evaluations and Benchmarks

    Anthropic has launched a $5 million grant initiative to support independent development of open-source benchmarks and evaluation harnesses measuring the impact of artificial intelligence systems on user wellbeing. The program will supply research teams with direct financial grants, subsidized API access to Claude models, and technical support from Anthropic's Safeguards team. All evaluation frameworks, datasets, and grading methodology developed under the grant program will be released publicly

    Anthropic
    2 min read

  8. OpenAI Introduces Admin Plugin for ChatGPT Work and Codex to Automate Workspace Governance

    OpenAI has released the Admin plugin for ChatGPT Work and Codex, embedding enterprise workspace management, user provisioning, permission audits, and spending controls directly into chat conversations. The plugin exposes Admin Console functionality through permission-scoped tools, allowing administrators to execute diagnostic queries and administrative actions without toggling across external consoles or building bespoke scripting pipelines. Conversational Workspace Administration Managing en

    OpenAI
    2 min read

  9. BitNet b1.58: Mathematical Foundations of 1.58-Bit Ternary Quantization, Absmean Scaling, and Multiplication-Free LLM Architectures

    Standard Large Language Models (LLMs) rely on 16-bit floating-point (FP16 or BF16) matrix multiplications for both pre-training and autoregressive inference. While post-training quantization techniques (such as GPTQ, AWQ, and dynamic FP8/INT4 scaling) compress weights after training, they often introduce perplexity degradation at ultra-low bitwidths ($<4$ bits) or still require floating-point dequantization at runtime. Quantization-aware pre-training architectures directly tackle this computati

    AI
    6 min read

  10. Test-Time Compute and Inference Search in Production: Comparing Best-of-N, Monte Carlo Tree Search, PRM-Guided Beam Search, and Dynamic Budget Allocation Architecture, Token Economics, and Serving Latency

    In standard production Large Language Model (LLM) deployments, generation compute is fixed: a prompt is processed during prefill, and the model generates an answer via greedy autoregressive decoding or low-temperature sampling. Under this paradigm, accuracy on complex reasoning, mathematical derivation, theorem proving, and code synthesis is bounded by the model parameter count and training data distributions. Recent empirical work demonstrates that scaling inference-time compute can achieve pe

    AI
    11 min read