Subscribe
Archive / page 98 of 123

Latest articles, page 98

  1. Self-Hosted Embedding and Reranking Serving in Production: TEI vs. Infinity vs. vLLM Architecture, Dynamic Batching, and Serving Economics

    While generative large language models dominate inference infrastructure discussions, vector embeddings and cross-encoder rerankers handle order-of-magnitude higher request volumes in production retrieval-augmented generation (RAG) and search pipelines. Serving embedding and reranking models presents fundamentally different computational characteristics than auto-regressive text generation. Without auto-regressive token generation loops or key-value (KV) cache state management, the primary engin

    AI
    5 min read

  2. Pipeline Parallelism in Large Language Models: How GPipe, 1F1B Scheduling, and Interleaving Tame Memory and Bubbles

    Training frontier large language models with tens or hundreds of billions of parameters exceeds the physical memory capacity of any individual GPU. While intra-node sharding strategies such as Tensor Parallelism partition individual matrix multiplications across accelerators over high-speed NVLink interconnects, scaling across multi-node clusters encounters strict hardware boundaries. Tensor Parallelism requires multiple collective All-Reduce communications per transformer layer. Across standar

    AI
    6 min read

  3. Modular Open-Sources Mojo Language Compiler and Toolchain Under Apache 2.0

    Modular Open-Sources Mojo Language Compiler and Toolchain Under Apache 2.0 Modular has released the complete source code for the Mojo programming language compiler, standard tooling, and runtime infrastructure under the Apache 2.0 license with LLVM exceptions. The announcement, delivered on August 18, 2026 during the company's ModCon developer conference, fulfills a multi-year roadmap commitment to transition the systems programming language to a fully open development model. The compiler sour

    AI
    2 min read

  4. AI Agent Evaluation in Production: Trajectory Benchmarks, Sandbox Harnesses, and Flakiness Mitigation

    Evaluating standard large language models relies on static input-output pairs: a fixed prompt produces a completion that an automated script compares against reference strings or grades with a calibrated judge. Autonomous AI agents break this paradigm completely. An agent executes a multi-step trajectory consisting of planning, tool invocation, environment state observation, error recovery, and variable-length decision loops. Evaluating an agent requires testing not just the final string output,

    Agents
    5 min read

  5. Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training

    Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training Training large language models across distributed GPU clusters introduces a fundamental memory bottleneck. In traditional Distributed Data Parallel (DDP) setups, every GPU maintains an identical copy of model weights, optimizer states, and gradients while processing independent data batches. As models scale from billions to hundreds of billions of parameters, static model s

    AI
    6 min read

  6. Anthropic Prepares Dual-Class Super-Voting Shares for Co-Founders Ahead of Planned IPO

    Anthropic is preparing to implement a dual-class share structure that grants super-voting equity to its co-founders ahead of a planned initial public offering, according to a report from The Information. The mechanism is designed to concentrate long-term operational voting control with executive leadership and insulate decision-making from external market and investor pressures. The structure comes as the maker of the Claude model family scales enterprise commercialization, with annual revenue

    Anthropic
    2 min read

  7. Alibaba Demonstrates Native Qwen 3.8 27B Inference on XuanTie C950 RISC-V CPU at 30 Tokens per Second

    Alibaba's semiconductor division, T-Head, announced day-zero native inference support for its latest open-weight model, Qwen 3.8 27B, running directly on the XuanTie C950 RISC-V server processor. Operating without discrete graphics processing units, the 64-core RISC-V chip delivered sustained decode throughput of 30 tokens per second alongside a time-to-first-token latency of 1.9 seconds. The benchmark demonstrates how architectural extensions on general-purpose open instruction sets can handle

    Hardware
    2 min read

  8. Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads

    Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads Training and serving modern large language models requires navigating severe hardware memory and compute constraints. While standard Distributed Data Parallelism (DDP) replicates the entire model across multiple accelerators, modern frontier architectures containing tens or hundreds of billions of parameters exceed the physical memory capacity of any single GPU. Even with 80 GB o

    AI
    9 min read

  9. Local LLM Inference on Apple Silicon: Architecture, Unified Memory, and Serving Benchmarks for MLX, llama.cpp, and Ollama

    Local large language model (LLM) serving on consumer hardware has historically faced a hard trade-off between memory capacity and execution bandwidth. Discrete consumer GPUs offer high memory bandwidth (up to 1,008 GB/s on an Nvidia RTX 4090) but are capped at 24 GB of VRAM, requiring model sharding or quantization to fit models beyond 14 billion parameters. Apple Silicon platforms bypass this capacity ceiling through a Unified Memory Architecture (UMA), where the CPU, GPU, and Apple Neural Eng

    LLM
    5 min read

  10. Mistral Expands Platform to Host Third-Party Open Weights Starting with GLM-5.2

    Mistral AI has broadened its API platform to host external open-weight foundation models, beginning with Zhipu AI's GLM-5.2. The move marks a strategic shift for the Paris-based AI company from serving only in-house architectures (such as Mistral Small, Mistral Medium, Mistral Large, and Voxtral) toward operating as a sovereign managed inference hub for third-party open weights. The integration introduces GLM-5.2 under the model identifier zai-glm-5-2 in public preview. The model is hosted with

    Mistral
    2 min read