Subscribe
Topic

Production AI

Production patterns for AI systems that have to work every day. Posts cover document chunking for RAG, disaggregated prefill and decode for serving, embedding fine-tuning with hard negative mining, and automated prompt optimization.

Articles

1-4
  1. Document Chunking Strategies in Production RAG: Comparing Recursive Splitting, Semantic Clustering, Late Chunking, and Contextual Retrieval Architecture, Boundary Preservation, and Retrieval Precision

    In retrieval-augmented generation (RAG) systems, document chunking dictates the upper bound of downstream retrieval quality and generation fidelity. Naive document fragmentation breaks cross-sentence references, severs entity co-occurrences, and discards global discourse structure. Conversely, oversized chunks inject extraneous noise into context windows, diluting the dense embedding representation and wasting generation budget. Production architectures have moved beyond static character counti

    RAG
    7 min read
  2. Disaggregated Prefill and Decode in Production LLM Serving: Architecture, Network KV Cache Migration, Chunked Prefill Trade-Offs, and Asymmetric Hardware Economics

    Large language model serving systems have historically treated transformer execution as a homogeneous sequence of forward passes over a single unified GPU pool. Under continuous batching engines, incoming requests execute their prompt evaluation (prefill) and autoregressive token generation (decode) on the exact same accelerators, co-locating both phases within shared iteration batches. While continuous batching improves GPU compute utilization compared to static batching, co-locating prefill a

    Inference
    11 min read
  3. Embedding Model Fine-Tuning in Production: Hard Negative Mining, Synthetic Data Pipelines, and Contrastive Distillation

    Production retrieval-augmented generation (RAG) and semantic search architectures frequently suffer from domain mismatch when relying on general-purpose embedding models. Off-the-shelf bi-encoders trained on broad web corpora often experience a 15% to 30% degradation in retrieval metrics such as NDCG@10 and MRR@10 when deployed on specialized enterprise corpora, including proprietary codebases, internal API schemas, clinical trials, and technical documentation. While downstream cross-encoder re

    RAG
    8 min read
  4. Automated Prompt Optimization in Production: Signatures, Teleprompters, and Metric-Driven Compilation with DSPy

    Manual prompt engineering remains one of the largest sources of technical debt in modern LLM applications. Teams routinely spend weeks hand-crafting multi-paragraph system prompts, hardcoding few-shot examples, and tweaking phrasing to extract reliable outputs from specific model checkpoints. When the underlying model is upgraded, migrated to an open-weight alternative, or integrated into a multi-step pipeline, these hand-crafted strings break, requiring another cycle of trial-and-error adjustme

    AI
    5 min read