Subscribe
Topic

System Design

System design for production LLM apps and agents. Posts explain multi-tenant LLM serving with fair-share scheduling and KV cache quotas, and context compaction and session pruning for long-running AI agents.

Articles

1-2
  1. Multi-Tenant LLM Serving in Production: Fair-Share Scheduling, Dynamic KV Cache Quotas, and Noisy Neighbor Isolation

    Operating a shared, multi-tenant large language model (LLM) serving cluster differs fundamentally from traditional stateless web tier hosting. In conventional microservices, tenants consume CPU cycles and static memory footprints in predictable, linear increments. In LLM serving, however, requests exhibit severe non-uniformity across multiple competing hardware dimensions: compute-bound prefill operations, memory-bandwidth-bound autoregressive decoding, and persistent High-Bandwidth Memory (HBM)

    Inference
    7 min read
  2. Context Compaction and Session Pruning in Production AI Agents: Architecture, Hierarchical Summarization, and Constraint Preservation

    Context Compaction and Session Pruning in Production AI Agents: Architecture, Hierarchical Summarization, and Constraint Preservation Long-running autonomous agents executing multi-step workflows (codebase refactoring, recursive research, multi-turn debugging, and system operations) face an inevitable physical ceiling: working context exhaustion. As an agent executes shell commands, inspects repository trees, parses LSP diagnostics, and consumes tool outputs, the active context window fills rap

    Agents
    7 min read