llms.blog
axc

Author

axc

386 articles

Articles

  • Defending AI Agents Against Indirect Prompt Injection: Dual-LLM Architectures, Privilege Boundaries, and Information Flow Control

    Autonomous AI agents are increasingly entrusted with system privileges, including terminal execution, API invocation, internal database queries, and automated communications. As agents transition from isolated conversational sandboxes to interconnected tools, they encounter an inherent architectural vulnerability: indirect prompt injection (IPI). When an agent reads untrusted data from the web, an inbound email, an enterprise ticketing system, or a database record, any instructions embedded wit

    1 min

  • Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving

    Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving Large language model inference is fundamentally constrained by memory bandwidth during the auto-regressive decoding phase. Because each token generation step requires loading billions of model parameters from high-bandwidth memory (HBM) to compute units for a single token, standard auto-regressive generation operates at low arithmetic intensity. Speculative decoding addresses this bottlene

    1 min

  • Beijing Clears Initial Shipments of 10,000 Nvidia H200 Chips to ByteDance and Tencent

    Chinese regulators have authorized the delivery of initial batches of Nvidia H200 artificial intelligence processors to mainland tech giants, marking a pivotal development in Beijing's management of high-performance compute access. According to reporting from the Financial Times, ByteDance and Tencent have each received roughly 10,000 H200 accelerators at their mainland data center facilities in recent weeks. Several additional domestic technology companies are currently awaiting clearance for

    1 min

  • Durable Execution for AI Agents: Architecture, State Checkpointing, and Failure Recovery

    Autonomous AI agents deployed in production environments frequently fail due to infrastructural instability rather than model reasoning flaws. Standard agent control loops, often structured as in-memory while-loops operating on transient servers or containerized pods, lack persistence across network blips, pod evictions, process restarts, or rate-limit timeouts. When an unhandled process failure occurs mid-task, standard agent architectures restart from scratch. This introduces three severe oper

    1 min

  • FlashDecoding: How Sequence Partitioning Solved the Memory Bandwidth Bottleneck in LLM Generation

    FlashDecoding: How Sequence Partitioning Solved the Memory Bandwidth Bottleneck in LLM Generation In large language model serving, execution divides into two distinct operational regimes: prompt prefill and autoregressive token generation (decoding). While FlashAttention transformed prefill throughput by eliminating High Bandwidth Memory (HBM) round-trips for intermediate attention matrices, standard FlashAttention algorithms encounter a severe hardware utilization bottleneck during decoding.

    1 min

  • Z.ai Opens GLM-5.3 API Access at .40/.40 per Million Tokens with Prompt Caching

    Chinese foundation model developer Z.ai (Zhipu AI) has opened public API access to GLM-5.3, offering developers direct endpoint integration following the model's initial release. The company kept base token rates aligned with the prior generation while introducing discounted prompt caching. GLM-5.3 is priced at $1.40 per million input tokens and $4.40 per million output tokens on the Z.ai platform. For workloads utilizing prompt caching, cached input tokens are billed at $0.26 per million, an 8

    1 min

  • Chinese Humanoid Robot Maker Unitree Surges 629% in 04M Shanghai IPO Debut

    Chinese humanoid and quadruped robotics manufacturer Unitree Robotics made its public debut on the Shanghai Stock Exchange STAR Market on Wednesday, August 19, 2026, with shares surging 629% in early morning trading. The listing marks the first pure-play embodied artificial intelligence and humanoid robotics IPO on China's mainland A-share exchange. Unitree priced its initial public offering at 150.80 yuan ($22.36) per share, issuing 40.4 million shares to raise approximately $904 million (4.2

    1 min

  • Codebase Context Engineering for AI Agents: Repo Maps, AST Indexing, and Retrieval Architectures

    Modern software repositories routinely contain hundreds of thousands or millions of lines of source code. A medium-sized production project with 500 files and 100,000 lines of code typically spans 3 million to 5 million tokens. While frontier models have expanded context windows to 200,000 tokens or more, stuffing an entire repository into a prompt remains fundamentally impractical. Beyond exponential inference costs and KV cache memory bloat, long-context attention suffers from severe retrieval

    1 min

  • FP8 Mixed-Precision Training: Formats, Scaling Recipes, and Fine-Grained GEMMs

    FP8 Mixed-Precision Training: Formats, Scaling Recipes, and Fine-Grained GEMMs Training frontier large language models requires navigating harsh memory bandwidth and compute boundaries. For years, 16-bit formats such as IEEE FP16 and Brain Floating Point (BF16) served as the standard arithmetic precision for deep learning workloads. While BF16 provided sufficient dynamic range to prevent gradient underflow without manual loss scaling, training clusters still spent substantial time bounded by me

    1 min

  • Z Lab Releases DFlash 2 for Qwen 3.8 27B: Block Diffusion Speculative Decoding with Target KV Injection

    Z Lab has released DFlash 2 checkpoints for Alibaba's Qwen 3.8 27B model family, advancing block-diffusion speculative decoding for open-weights LLM serving. By replacing conventional autoregressive draft models with a non-causal diffusion mechanism paired with direct target key-value (KV) cache injection, the framework achieves up to 3x to 4.3x throughput speedups in production inference engines like SGLang and vLLM without altering output token distributions. Speculative decoding conventional

    1 min

  • Pennsylvania Restricts Speculative AI Data Centers in Executive Order 2026-05

    Pennsylvania Governor Josh Shapiro has signed Executive Order 2026-05, introducing strict regulatory standards on high-capacity data center construction and ending the state's expedited permitting program for computing facilities. The directive requires prospective developers of large-scale facilities to enter legally binding commitments with the Commonwealth to safeguard local power grids, protect municipal water supplies, and secure approval from local governments before receiving state enviro

    1 min

  • Automated LLM Red Teaming in Production: Comparing Garak, PyRIT, and Promptfoo

    Static penetration testing and manual prompt probing cannot secure non-deterministic language models or agentic systems. Manual testing provides anecdotal security at best: the attack surface of large language models spans thousands of adversarial permutations, multi-turn conversational steering, payload encoding, and indirect prompt injections introduced through external retrieval. To systematically identify failure modes before deployment, engineering teams rely on automated red teaming frame

    1 min

  • Activation Checkpointing in Large Language Models: How Selective Recomputation Eliminates Memory Bottlenecks

    Large language model pre-training and fine-tuning are fundamentally constrained by GPU memory (VRAM). While distributed techniques such as Fully Sharded Data Parallel (FSDP), ZeRO, and Tensor Parallelism successfully shard model parameters, optimizer states, and gradients across hundreds or thousands of GPUs, activation memory presents a distinct scaling bottleneck. During the forward pass of a transformer model, intermediate tensor outputs must be preserved in GPU memory so that backpropagatio

    1 min

  • Anthropic Demonstrates Autonomous De Novo Protein Design and Chemical Analysis with Claude

    Anthropic Demonstrates Autonomous De Novo Protein Design and Chemical Analysis with Claude Anthropic has published experimental results demonstrating Claude's ability to autonomously design de novo protein binders with physical wet-lab validation and automate complex analytical chemistry workflows. The findings show frontier LLMs acting as autonomous agents across computational biology and molecular characterization pipelines. In the primary experiment, Anthropic evaluated Claude Mythos Previe

    1 min

  • Cerebras Unveils CS-4 Rack-Scale System Powered by Three WSE-3 Turbo Chips and Nexus Architecture

    Cerebras Unveils CS-4 Rack-Scale System Powered by Three WSE-3 Turbo Chips and Nexus Architecture Cerebras Systems has announced the CS-4, a rack-scale AI accelerator system designed around three of its next-generation Wafer Scale Engine 3 Turbo (WSE-3 Turbo) chips and a modular hardware architecture dubbed Nexus. Cerebras confirmed that initial customer shipments for the CS-4 are scheduled to begin in the current quarter. The new system marks a structural shift from Cerebras's single-wafer CS

    1 min

  • AI FinOps: Cutting LLM Inference Costs by 30-60% Through Model Tiering, Caching, and GPU Optimization

    AI FinOps: Cutting LLM Inference Costs by 30-60% Through Model Tiering, Caching, and GPU Optimization Inference costs have become the second-largest line item in enterprise AI budgets, trailing only talent spend according to RapidData's State of Enterprise AI 2026. This shift represents a fundamental inversion from the 2021-2023 era when training dominated AI expenditure. The compounding nature of serving costs—accumulating every hour as long as users hit the API—means that even modest producti

    1 min

  • Sequence Parallelism in Large Language Models: How Megatron-SP, DeepSpeed Ulysses, and RingAttention Distribute Long Contexts

    Sequence Parallelism in Large Language Models: How Megatron-SP, DeepSpeed Ulysses, and RingAttention Distribute Long Contexts Training and serving frontier large language models on context windows spanning hundreds of thousands to millions of tokens introduces a fundamental memory barrier. While model parameters can be distributed across GPUs using Tensor Parallelism (TP) or Fully Sharded Data Parallelism (FSDP / ZeRO), activation memory scales directly with sequence length $S$. For sequence le

    1 min

  • GLM-5.3 Scores 60 on Artificial Analysis Intelligence Index, Matching Kimi K3

    Independent AI evaluation platform Artificial Analysis has published its benchmark results for Z.ai's GLM-5.3, awarding the reasoning model a score of 60 on its Intelligence Index v4.1.1. The result places GLM-5.3 level with Moonshot AI's Kimi K3 and three points behind frontier leader Claude Opus 5 (63). The evaluation tested GLM-5.3 at its maximum reasoning effort configuration across a nine-part battery that measures agentic tool execution, terminal coding, graduate-level scientific problem-

    1 min

  • Block Open-Sources Berd: Apache 2.0 Desktop Workspace for Multi-Model AI Agents

    Block has open-sourced Berd, an Apache 2.0-licensed desktop application designed to serve as a unified workspace for managing AI agents across different foundation models, toolsets, and execution harnesses. Originally built for internal use across Square, Cash App, and Tidal, the desktop client reached version 0.6.2 on August 18, 2026, with builds available for macOS, Windows, and Linux. The release addresses growing operational fragmentation as developers juggle specialized agent environments

    1 min

  • Self-Hosted Embedding and Reranking Serving in Production: TEI vs. Infinity vs. vLLM Architecture, Dynamic Batching, and Serving Economics

    While generative large language models dominate inference infrastructure discussions, vector embeddings and cross-encoder rerankers handle order-of-magnitude higher request volumes in production retrieval-augmented generation (RAG) and search pipelines. Serving embedding and reranking models presents fundamentally different computational characteristics than auto-regressive text generation. Without auto-regressive token generation loops or key-value (KV) cache state management, the primary engin

    1 min