Subscribe
Topic / page 28

LLM

Articles, page 28

541-560
  1. Slack Launches Slack Code to Host AI Coding Agents in Dedicated Project Channels

    Slack has introduced Slack Code, a native environment that shifts AI-assisted software development out of private chat windows and into dedicated, collaborative workspace channels. Available immediately across all Slack subscription tiers, including free accounts, the feature enables engineering and product teams to summon autonomous coding agents directly into project-scoped spaces where teammates can monitor implementation progress, inspect code diffs, review live web previews, and control pro

    AI
    2 min read
  2. Multi-Tenant LLM Serving in Production: Fair-Share Scheduling, Dynamic KV Cache Quotas, and Noisy Neighbor Isolation

    Operating a shared, multi-tenant large language model (LLM) serving cluster differs fundamentally from traditional stateless web tier hosting. In conventional microservices, tenants consume CPU cycles and static memory footprints in predictable, linear increments. In LLM serving, however, requests exhibit severe non-uniformity across multiple competing hardware dimensions: compute-bound prefill operations, memory-bandwidth-bound autoregressive decoding, and persistent High-Bandwidth Memory (HBM)

    Inference
    7 min read
  3. ReAct in Large Language Models: How Interleaving Reasoning and Action Traces Built the Foundation of AI Agents

    ReAct in Large Language Models: How Interleaving Reasoning and Action Traces Built the Foundation of AI Agents Before autonomous agents could interact reliably with APIs, search engines, and bash environments, large language models (LLMs) operated in one of two disconnected paradigms: internal reasoning without external interaction, or external action generation without internal deliberation. In pure reasoning paradigms such as Chain-of-Thought (CoT) prompting, models generate intermediate natu

    AI
    10 min read
  4. Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs

    Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs Standard inference pipelines in large language models rely on greedy decoding or low-temperature sampling to generate answers. For tasks requiring direct factual recall or basic classification, greedy token selection generates fluent and accurate outputs. However, when large models are tasked with complex multi-step reasoning, greedy generation exhibits severe structural fragility. In

    AI
    6 min read
  5. Traffic Shadowing and Canary Deployments in Production: Architecture, Semantic Divergence, and Automated Rollbacks for LLM Upgrades

    Upgrading large language model pipelines in production presents an operational challenge distinct from traditional software deployments. In microservice architectures, canary deployments rely on deterministic signals: HTTP 5xx error rates, unhandled runtime exceptions, CPU saturation, and latency percentiles. When updating an LLM system (whether swapping a base model checkpoint, updating a system prompt, tweaking sampling hyperparameters, or attaching a newly fine-tuned LoRA adapter), the servic

    AI
    6 min read
  6. Late Interaction and ColBERT: How Multi-Vector Embeddings and the MaxSim Operator Transform Neural Retrieval

    Information retrieval systems have long wrestled with a fundamental tension between computational efficiency and semantic expressiveness. Traditional dense bi-encoders like DPR compress an entire passage into a single dense vector, allowing sub-linear approximate nearest neighbor (ANN) search over millions of documents. However, forcing multi-sentence passages into a single vector representation creates an information bottleneck that discards fine-grained token-level nuances, entities, and keywo

    AI
    6 min read
  7. State Space Duality and Mamba-2: How Semiseparable Matrices Unify Attention and SSMs

    For years, deep sequence modeling operated under a structural divide. On one side stood the Transformer architecture, anchored by softmax attention. Softmax attention scales quadratically in compute and memory with sequence length (O(T^2)), but its operations map cleanly to dense General Matrix Multiplications (GEMMs), maximizing utilization of GPU Tensor Cores. On the other side stood Structured State Space Models (SSMs), such as S4 and Mamba-1. Structured SSMs scale linearly in time (O(T · N))

    AI
    6 min read
  8. Diffusion Transformers (DiT): How Patchification and adaLN-Zero Replaced U-Nets in Generative AI

    Generative visual models relied for years on convolutional U-Net architectures to execute iterative denoising. From Denoising Diffusion Probabilistic Models (DDPM) and Ablated Diffusion Models (ADM) to Latent Diffusion Models (LDMs) behind Stable Diffusion, convolutional backbones served as the default engine for image synthesis. While convolutional inductive biases provided translation equivariance and local spatial hierarchies, they imposed architectural rigidities that resisted compute scalin

    AI
    8 min read
  9. Agent Egress Security in Production: Network Sandboxing, Secretless Token Rewriting, and DNS Exfiltration Defenses

    Agent Egress Security in Production: Network Sandboxing, Secretless Token Rewriting, and DNS Exfiltration Defenses Autonomous AI agents with tool execution, code execution environments, and Model Context Protocol (MCP) servers present a fundamental shift in network security architecture. Traditional web application security treats outbound traffic from backend services as trusted or semi-trusted, focusing defense mechanisms on inbound traffic via Web Application Firewalls (WAFs) and API gateway

    AI
    7 min read
  10. Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts

    Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at

    AI
    8 min read
  11. Custom LLM Kernel Optimization in Production: Triton vs. CUDA C++ vs. torch.compile vs. CUTLASS

    Serving large language models at scale requires extracting maximum performance from modern GPU architectures like NVIDIA Ampere, Hopper, and Blackwell. While early production deployments relied on standard PyTorch eager execution and standard cuBLAS calls, high-throughput serving systems such as vLLM, SGLang, and TensorRT-LLM depend on specialized fused GPU kernels to eliminate memory bandwidth bottlenecks and saturate Tensor Cores. Engineering teams face four primary paradigms for kernel devel

    AI
    7 min read
  12. DeepSeek Open-Sources DeepSeek Harness: Modular Agent Runtime Built on Cordis Plugin Framework

    DeepSeek AI has released DeepSeek Harness (dsh), an open-source agent harness built on a modular "everything is a plugin" design. Published under the MIT license on GitHub, the project provides a runtime framework where all core agent functions, including model adapters, session storage, tool registries, sandboxes, and the execution loop itself, are implemented as interchangeable, hot-swappable plugins. Rather than developing a monolithic agent runner, DeepSeek structured the system around Cord

    AI
    2 min read
  13. DeepReinforce Releases Ornith-1.5: MIT-Licensed 9B, 35B, and 397B Open Weights Trained on Autonomous Self-Improvement Loops

    DeepReinforce and the Ornith team have released Ornith-1.5, a family of open-weight foundation models trained using autonomous self-improvement loops. Released under the permissive MIT license, the model family spans three architectures: a 9B dense model, a 35B mixture-of-experts model activating approximately 3B parameters per token (35B-A3B), and a flagship 397B mixture-of-experts model. The release expands on the team's earlier Ornith-1.0 release by automating the entire post-training optimi

    AI
    2 min read
  14. GPU Cluster Networking in Production: InfiniBand vs. RoCEv2 vs. Ultra Ethernet Architecture, Congestion Control, and NCCL Collective Latency

    Distributed training and high-throughput inference workloads are fundamentally bound by the network fabric. While traditional cloud applications rely on asynchronous request-response cycles that absorb latency jitter, distributed deep learning relies on synchronous collective communication. Operations such as All-Reduce, All-Gather, and All-to-All require hundreds or thousands of GPUs to exchange tensors and synchronize at strict barrier points before execution can proceed. In this execution mo

    AI
    8 min read
  15. OKX and Goldman Sachs Cut Off Hong Kong Staff From Anthropic's Claude AI

    Crypto exchange OKX has restricted access to Anthropic's Claude model for employees based in Hong Kong or traveling through China after its corporate account was briefly suspended, according to Bloomberg. The restriction comes despite OKX making AI proficiency part of employee performance evaluations and encouraging workers to integrate AI tools into daily workflows. OKX stopped providing Claude access to staff in the region in early August following the suspension, Bloomberg reported. Internal

    Policy
    2 min read
  16. Vector Indexing in Production: HNSW vs. DiskANN vs. IVF-PQ Architecture, Memory Footprint, and Search Economics

    Vector Indexing in Production: HNSW vs. DiskANN vs. IVF-PQ Architecture, Memory Footprint, and Search Economics Scaling vector search beyond prototype deployments exposes a fundamental tension across three competing constraints: retrieval recall, query latency, and memory footprint. In high-dimensional representation spaces, exact k-nearest neighbor search via brute-force flat scans requires $O(N \cdot d)$ floating-point operations per query. For a corpus of 100 million 1536-dimensional FP32 em

    AI
    7 min read
  17. Rotational Quantization: How Orthogonal Transforms and Hadamard Incoherence Enable Outlier-Free Low-Bit Inference

    Large language model serving is governed by two physical bottlenecks: memory bandwidth during autoregressive decoding and compute throughput during prompt prefill. While weight-only post-training quantization (such as GPTQ or AWQ) reduces parameter footprint to 4 bits, it leaves activations in 16-bit floating-point formats. As a result, inference engines cannot utilize high-throughput INT4 tensor cores, and the key-value (KV) cache continues to consume massive memory pools. Attempting to quanti

    AI
    8 min read
  18. Multi-LoRA Serving in Production: Architecture, Dynamic Adapter Swapping, and GPU Memory Management

    Deploying hundreds or thousands of fine-tuned language models across enterprise workflows presents a fundamental infrastructure dilemma. While parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) reduce training compute by freezing base model weights and training compact low-rank matrices, naive deployment strategies fail at scale. Merging adapter weights directly into the base model creates a distinct 7B to 70B parameter checkpoint for every specialized task, requiring

    AI
    8 min read
  19. Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap

    Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap Parameter-efficient fine-tuning (PEFT) has become the standard operational paradigm for adapting large language models to domain-specific downstream tasks. Among existing PEFT methodologies, Low-Rank Adaptation (LoRA) remains the default implementation across industry and academia due to its minimal parameter footprint and zero inference overhead. However, empirical studies consistently reve

    AI
    8 min read