Subscribe
Category / page 14

Coding & Agents

352 articles

Articles, page 14

261-280 of 352

  1. Multi-Tenant LLM Serving in Production: Fair-Share Scheduling, Dynamic KV Cache Quotas, and Noisy Neighbor Isolation

    Operating a shared, multi-tenant large language model (LLM) serving cluster differs fundamentally from traditional stateless web tier hosting. In conventional microservices, tenants consume CPU cycles and static memory footprints in predictable, linear increments. In LLM serving, however, requests exhibit severe non-uniformity across multiple competing hardware dimensions: compute-bound prefill operations, memory-bandwidth-bound autoregressive decoding, and persistent High-Bandwidth Memory (HBM)

    Inference
    7 min read

  2. ReAct in Large Language Models: How Interleaving Reasoning and Action Traces Built the Foundation of AI Agents

    ReAct in Large Language Models: How Interleaving Reasoning and Action Traces Built the Foundation of AI Agents Before autonomous agents could interact reliably with APIs, search engines, and bash environments, large language models (LLMs) operated in one of two disconnected paradigms: internal reasoning without external interaction, or external action generation without internal deliberation. In pure reasoning paradigms such as Chain-of-Thought (CoT) prompting, models generate intermediate natu

    AI
    10 min read

  3. Contextual Retrieval in Production RAG: Architecture, Prompt Caching Economics, Hybrid Fusion, and Reranking Pipelines

    Contextual Retrieval in Production RAG: Architecture, Prompt Caching Economics, Hybrid Fusion, and Reranking Pipelines Standard Retrieval-Augmented Generation (RAG) architectures suffer from an inherent design flaw at the preprocessing stage: chunking destroys document hierarchy. When a system divides a large document corpus into fixed-size passages (such as 300 to 800 tokens) or applies semantic boundaries, the resulting chunks lose their surrounding narrative, parent headings, entity definiti

    AI
    7 min read

  4. Traffic Shadowing and Canary Deployments in Production: Architecture, Semantic Divergence, and Automated Rollbacks for LLM Upgrades

    Upgrading large language model pipelines in production presents an operational challenge distinct from traditional software deployments. In microservice architectures, canary deployments rely on deterministic signals: HTTP 5xx error rates, unhandled runtime exceptions, CPU saturation, and latency percentiles. When updating an LLM system (whether swapping a base model checkpoint, updating a system prompt, tweaking sampling hyperparameters, or attaching a newly fine-tuned LoRA adapter), the servic

    AI
    6 min read

  5. Late Interaction and ColBERT: How Multi-Vector Embeddings and the MaxSim Operator Transform Neural Retrieval

    Information retrieval systems have long wrestled with a fundamental tension between computational efficiency and semantic expressiveness. Traditional dense bi-encoders like DPR compress an entire passage into a single dense vector, allowing sub-linear approximate nearest neighbor (ANN) search over millions of documents. However, forcing multi-sentence passages into a single vector representation creates an information bottleneck that discards fine-grained token-level nuances, entities, and keywo

    AI
    6 min read

  6. Binance Launches Agent OS with MCP Support for Autonomous AI Trading

    Binance has released Agent OS, an infrastructure layer designed to connect autonomous artificial intelligence agents directly to its spot, derivatives, and decentralized finance services. The release introduces official Model Context Protocol (MCP) support alongside dedicated sub-account sandboxes, allowing client-side agents to execute trades, query order books, and interact with on-chain protocols. The integration enables developers using developer tools and agent runtimes, including Anthropi

    Agents
    2 min read

  7. Data Ingestion and Incremental Sync for Production RAG: CDC Streams, Content Hashing, Backpressure, and Zero-Downtime Indexing

    Maintaining retrieval-augmented generation (RAG) systems in production introduces a fundamental distributed systems challenge that rarely surfaces in proof-of-concept architectures: state synchronization. While initial ingestion across a static document corpus is straightforward, production data sources (PostgreSQL databases, transactional stores, object storage, and enterprise knowledge hubs) undergo continuous mutation. Records are inserted, updated, soft-deleted, and reassigned new access per

    RAG
    8 min read

  8. Unsloth Releases Dynamic V3.0 GGUFs for Qwen 3.8 27B with 1-Bit Mode and MTP

    Unsloth AI has published its Dynamic V3.0 quantization suite for Alibaba's Qwen 3.8 27B model family, releasing optimized GGUF and NVFP4 checkpoints alongside public calibration matrices. The release claims a greater than 10 percent increase in top-1 percent accuracy at identical file sizes compared to standard baseline quantizations, while introducing an ultra-low-bit dynamic tier that operates within 8GB of memory. Qwen 3.8 27B is a dense vision-language model utilizing hybrid attention layer

    Open Weights
    2 min read

  9. HoneyBook Launches Claude MCP Connector for Autonomous Small Business CRM

    Small-business CRM platform HoneyBook has rolled out an official integration for Anthropic's Claude built on the Model Context Protocol (MCP). The connector exposes structured customer records, project timelines, and billing systems to conversational AI agents, allowing service professionals to run client workflows through natural language interfaces. While large enterprises have increasingly deployed autonomous AI agents into core enterprise resource planning systems, smaller operators face st

    Anthropic
    2 min read

  10. Agent Egress Security in Production: Network Sandboxing, Secretless Token Rewriting, and DNS Exfiltration Defenses

    Agent Egress Security in Production: Network Sandboxing, Secretless Token Rewriting, and DNS Exfiltration Defenses Autonomous AI agents with tool execution, code execution environments, and Model Context Protocol (MCP) servers present a fundamental shift in network security architecture. Traditional web application security treats outbound traffic from backend services as trusted or semi-trusted, focusing defense mechanisms on inbound traffic via Web Application Firewalls (WAFs) and API gateway

    AI
    7 min read

  11. Custom LLM Kernel Optimization in Production: Triton vs. CUDA C++ vs. torch.compile vs. CUTLASS

    Serving large language models at scale requires extracting maximum performance from modern GPU architectures like NVIDIA Ampere, Hopper, and Blackwell. While early production deployments relied on standard PyTorch eager execution and standard cuBLAS calls, high-throughput serving systems such as vLLM, SGLang, and TensorRT-LLM depend on specialized fused GPU kernels to eliminate memory bandwidth bottlenecks and saturate Tensor Cores. Engineering teams face four primary paradigms for kernel devel

    AI
    7 min read

  12. DeepSeek Open-Sources DeepSeek Harness: Modular Agent Runtime Built on Cordis Plugin Framework

    DeepSeek AI has released DeepSeek Harness (dsh), an open-source agent harness built on a modular "everything is a plugin" design. Published under the MIT license on GitHub, the project provides a runtime framework where all core agent functions—including model adapters, session storage, tool registries, sandboxes, and the execution loop itself—are implemented as interchangeable, hot-swappable plugins. Rather than developing a monolithic agent runner, DeepSeek structured the system around Cordis

    AI
    2 min read

  13. Vector Indexing in Production: HNSW vs. DiskANN vs. IVF-PQ Architecture, Memory Footprint, and Search Economics

    Vector Indexing in Production: HNSW vs. DiskANN vs. IVF-PQ Architecture, Memory Footprint, and Search Economics Scaling vector search beyond prototype deployments exposes a fundamental tension across three competing constraints: retrieval recall, query latency, and memory footprint. In high-dimensional representation spaces, exact k-nearest neighbor search via brute-force flat scans requires $O(N \cdot d)$ floating-point operations per query. For a corpus of 100 million 1536-dimensional FP32 em

    AI
    7 min read

  14. Multi-LoRA Serving in Production: Architecture, Dynamic Adapter Swapping, and GPU Memory Management

    Deploying hundreds or thousands of fine-tuned language models across enterprise workflows presents a fundamental infrastructure dilemma. While parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) reduce training compute by freezing base model weights and training compact low-rank matrices, naive deployment strategies fail at scale. Merging adapter weights directly into the base model creates a distinct 7B to 70B parameter checkpoint for every specialized task, requiring

    AI
    8 min read

  15. Small Language Models in Production: Task Specialization, Serving Economics, and the Frontier Offloading Pattern

    The default architecture for first-generation enterprise AI agents routed every prompt, tool selection, and intermediate evaluation step to a single frontier large language model. While this monolithic approach simplified initial orchestration, it introduced severe latency bottlenecks and unsustainable inference unit economics in high-throughput production environments. In production agentic loops, between 40% and 70% of model invocations are narrow, highly structured operations: classifying in

    AI
    6 min read

  16. TrueFoundry Open-Sources TrueForge Agent Harness to Cut Managed Inference Spend

    Enterprise ML infrastructure platform TrueFoundry has open-sourced its custom agent runtime, TrueForge, under the permissive MIT license. The framework is designed to orchestrate autonomous reasoning loops, tool interactions, and long-running execution across vendor-neutral LLM backends while systematically reducing token overhead and operational compute costs. According to benchmark data published by TrueFoundry, pairing TrueForge with the open-weight GLM-5.2 model completed 11 of 14 enterpris

    AI
    2 min read

  17. Context Compaction and Session Pruning in Production AI Agents: Architecture, Hierarchical Summarization, and Constraint Preservation

    Context Compaction and Session Pruning in Production AI Agents: Architecture, Hierarchical Summarization, and Constraint Preservation Long-running autonomous agents executing multi-step workflows (codebase refactoring, recursive research, multi-turn debugging, and system operations) face an inevitable physical ceiling: working context exhaustion. As an agent executes shell commands, inspects repository trees, parses LSP diagnostics, and consumes tool outputs, the active context window fills rap

    Agents
    7 min read

  18. Human-in-the-Loop Architectures for Production AI Agents: Interrupt Patterns, State Resumption, and Permission Escalation

    Deploying autonomous AI agents into production environments exposes a fundamental tension between system velocity and operational safety. While read-only tasks such as data extraction and document summarization carry minimal operational blast radius, agents equipped with write-access tools (database mutations, API transactions, cloud infrastructure provisioning, and outbound communications) introduce severe operational risks. Hallucinations, prompt injections, and logical drift can trigger irrev

    AI
    6 min read

  19. Semantic Routers in Production: Architecture, Embedding Classification vs. LLM Triage, and Cascade Topologies

    Routing user requests to specialized tools, APIs, or tiered foundation models is central to production AI architectures. However, relying on large language models (LLMs) or native JSON tool-calling mechanisms to perform initial intent classification introduces significant latency and cost overheads. A single LLM-based classification pass typically adds between 200 milliseconds and 1.5 seconds of time-to-first-token (TTFT) latency, alongside linear token billing for routine triage queries. To el

    AI
    5 min read

  20. Self-Correction and Reflection Loops in Production AI Agents: Architecture, Verification Oracles, and the Over-Correction Trap

    Autonomous AI agents frequently fail on initial generation when solving multi-step reasoning, code generation, and complex API orchestration tasks. To address initial execution failures, system architects widely deploy self-correction and reflection loops. However, the mechanism through which reflection operates determines whether a system converges on a valid solution or degrades into hallucinations and infinite loops. Recent research demonstrates a sharp division in reflection paradigms: whil

    AI
    6 min read