llms.blog
LLM135 articles

LLM

Articles

  • LLM Fine-Tuning Frameworks in Production: Unsloth vs. Axolotl vs. LLaMA-Factory vs. Torchtune Architecture, Throughput, and Distributed Scaling

    Modern post-training pipelines have moved beyond basic training scripts. As model parameter counts, context windows, and alignment techniques expand, the choice of fine-tuning framework directly dictates GPU memory overhead, token throughput, and developer iteration speed. Four open-source frameworks dominate the enterprise fine-tuning landscape: Unsloth, Axolotl, LLaMA-Factory, and Meta's Torchtune. While all four orchestrate parameter-efficient fine-tuning (PEFT) and full parameter adaptation

    1 min

  • Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads

    Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads Training and serving modern large language models requires navigating severe hardware memory and compute constraints. While standard Distributed Data Parallelism (DDP) replicates the entire model across multiple accelerators, modern frontier architectures containing tens or hundreds of billions of parameters exceed the physical memory capacity of any single GPU. Even with 80 GB o

    1 min

  • Local LLM Inference on Apple Silicon: Architecture, Unified Memory, and Serving Benchmarks for MLX, llama.cpp, and Ollama

    Local large language model (LLM) serving on consumer hardware has historically faced a hard trade-off between memory capacity and execution bandwidth. Discrete consumer GPUs offer high memory bandwidth (up to 1,008 GB/s on an Nvidia RTX 4090) but are capped at 24 GB of VRAM, requiring model sharding or quantization to fit models beyond 14 billion parameters. Apple Silicon platforms bypass this capacity ceiling through a Unified Memory Architecture (UMA), where the CPU, GPU, and Apple Neural Eng

    1 min

  • Synthetic Data Pipelines for LLM Post-Training: Generation, Quality Filtering, Deduplication, and Contamination Auditing

    As frontier model post-training expands beyond the limits of human-annotated datasets, synthetic data generation (SDG) has become the core driver of alignment. Public disclosures from major research labs confirm that synthetic data now comprises the vast majority of tokens used in supervised fine-tuning (SFT) and preference alignment. For example, NVIDIA reported that over 98% of the data used in the alignment pipeline for Nemotron-4 340B was synthetically generated. Similarly, models across the

    1 min

  • IBM Research Evaluates Agentic Memory Sizing Across 8 Models: Dosage Calibrations, Ceiling Effects, and Token Efficiency

    In a technical report published on August 18, 2026, researchers at IBM Research detailed empirical evaluations on sizing and calibrating agentic memory across eight large language models. The study, conducted using the open-source ALTK-Evolve framework across the AppWorld benchmark, demonstrates that agentic memory performance is governed by capability-dependent dosage rather than uniform prompt accumulation. Agentic memory architectures typically extract procedural guidelines from prior execut

    1 min

  • Multi-Token Prediction: How Future Token Supervision Densifies Representations and Speeds Up LLM Serving

    Standard autoregressive language models are trained under a strict next-token prediction objective. At every sequence position, the model consumes a prefix of tokens and predicts the single immediate successor token using a cross-entropy loss. While this paradigm has scaled language modeling across orders of magnitude, it suffers from an architectural limitation: myopic optimization. By evaluating loss exclusively on the immediate next step, standard training fails to reward representations that

    1 min

  • Executable Code Actions vs. JSON Tool Calling: Architecture, Token Economics, Sandboxing, and Expressivity in Production AI Agents

    Executable Code Actions vs. JSON Tool Calling: Architecture, Token Economics, Sandboxing, and Expressivity in Production AI Agents The dominant paradigm for connecting large language models to external tools has relied on structured JSON function calling. First standardized across commercial APIs via JSON Schema manifests and constrained decoding, this approach frames agent interaction as remote procedure calls (RPC): the model outputs a JSON object specifying a tool name and parameters, the ho

    1 min

  • Group Relative Policy Optimization (GRPO): How Eliminating Value Models Scaled LLM Reasoning

    Post-training reinforcement learning (RL) has become the primary mechanism for scaling reasoning capabilities in large language models. While early reinforcement learning from human feedback (RLHF) focused on conversational style and safety alignment, extending RL to multi-step reasoning domains such as mathematics, algorithmic coding, and formal logic exposed critical limitations in classical algorithms. Standard Proximal Policy Optimization (PPO), long the foundational algorithm for instructi

    1 min

  • Context Window Extension in Large Language Models: How Position Interpolation, YaRN, and LongRoPE Scale Sequence Lengths

    Large language models are bounded during pretraining by a fixed sequence length, typically between 2,048 and 8,192 tokens. When standard autoregressive transformers attempt to process sequences beyond this pretraining context window, performance degrades immediately. Perplexity rises sharply and the model loses coherence within a few dozen tokens past the training boundary. Extending this context window by training from scratch on long sequences is computationally prohibitive due to the quadrat

    1 min

  • Automated Prompt Optimization in Production: Signatures, Teleprompters, and Metric-Driven Compilation with DSPy

    Manual prompt engineering remains one of the largest sources of technical debt in modern LLM applications. Teams routinely spend weeks hand-crafting multi-paragraph system prompts, hardcoding few-shot examples, and tweaking phrasing to extract reliable outputs from specific model checkpoints. When the underlying model is upgraded, migrated to an open-weight alternative, or integrated into a multi-step pipeline, these hand-crafted strings break, requiring another cycle of trial-and-error adjustme

    1 min

  • Model Merging in Large Language Models: How Task Arithmetic, TIES, and DARE Combine Checkpoints Without Training

    Fine-tuning foundation models for specialized tasks typically produces isolated checkpoints. A model adapted for mathematical reasoning retains high numerical precision but often degrades in general dialogue or code generation. Traditionally, unifying these capabilities required multi-task training: gathering mixed datasets, re-running optimization across multiple GPUs, and managing gradient conflicts during backpropagation. Model merging provides an alternative paradigm. By operating directly

    1 min

  • Harvey Introduces Tenet, Its First In-House Legal LLM Trained on Moonshot's Kimi K3

    Legal AI startup Harvey has announced Harvey Tenet, its first proprietary, in-house foundation model tailored for legal workflows. The release marks a strategic shift for the $11 billion legal tech company, which has historically relied on API access to third-party frontier models from OpenAI and Anthropic. Tenet is post-trained on top of Kimi K3, an open-weights model released in July 2026 by Chinese AI lab Moonshot AI. The initiative is part of a broader platform update titled Harvey II, whic

    1 min

  • Document Chunking Strategies for Production RAG: Fixed-Size, Semantic, Hierarchical, and Late Chunking Trade-Offs

    Document Chunking Strategies for Production RAG: Fixed-Size, Semantic, Hierarchical, and Late Chunking Trade-Offs In production retrieval-augmented generation (RAG), document chunking is often treated as a trivial preprocessing step. In practice, the method used to partition raw text directly dictates the upper bound of retrieval recall, embedding representation quality, and downstream generation accuracy. Retrieval systems face a fundamental tension. Dense vector search models perform best wh

    1 min

  • Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    1 min

  • Warp Launches Warp Factories to Automate Multi-Agent Software Development Lifecycles

    Terminal and developer tools maker Warp has introduced Warp Factories, a turnkey infrastructure system designed to manage and orchestrate autonomous AI coding agents across the software development lifecycle. The platform aims to lower the barrier for engineering teams implementing multi-agent workflows by providing preconfigured orchestration pipelines, evaluation harnesses, and runtime observability. Software Factory Architecture The "software factory" model structures development into five

    1 min

  • Sentence Transformers v6.0 Adds Native Multi-Vector Late Interaction for ColBERT and ColPali

    Hugging Face has released Sentence Transformers v6.0, adding native multi-vector late-interaction retrieval to the library through a new MultiVectorEncoder interface. The update integrates ColBERT-style models and vision-language document retrieval systems directly into the standard Sentence Transformers workflow alongside dense bi-encoders, sparse models, and cross-encoder rerankers. Mechanics of Late Interaction and MaxSim Standard dense embedding models compress an entire passage or query

    1 min

  • Agent Memory Architectures in Production: Working Context, Episodic Buffers, Semantic Graphs, and State Serialization

    Large Language Models operate as stateless prediction engines: every API call processes an input prompt independently, without retaining memory of previous turns, decisions, or external interactions. While extending context windows to 1 million or 2 million tokens provides temporary capacity for long transcripts, treating raw context windows as long-term memory introduces severe engineering bottlenecks. Unbounded context growth dramatically inflates time-to-first-token (TTFT) latency, increases

    1 min

  • Constitutional AI and RLAIF: How Natural Language Principles and Automated Critiques Scale LLM Alignment

    Early alignment frameworks for large language models relied almost entirely on Reinforcement Learning from Human Feedback (RLHF). While RLHF transformed raw base models into usable assistants, the approach faces structural scaling bottlenecks. Collecting tens of thousands of high-quality human preference annotations is slow, expensive, and logistically complex. Furthermore, human annotators frequently disagree on nuanced safety boundaries, suffer psychological fatigue when reviewing harmful outp

    1 min

  • Snowflake Adds Dynamic Model Routing to Cortex AI Gateway to Cut Inference Spend

    Snowflake has introduced dynamic model routing into its Cortex AI Gateway, allowing enterprise developers to automate large language model selection instead of binding agentic workloads to static endpoints. According to internal benchmarks reported by the company, automated routing can reduce token expenditure by up to 3x across mixed workloads by directing routine queries away from expensive frontier models toward lightweight architectures. The Inference Cost Bottleneck in Agentic Workflows

    1 min

  • GraphRAG vs. Vector RAG in Production: Architecture, Community Summaries, and Cost-Latency Trade-Offs

    Retrieval-Augmented Generation (RAG) has become the standard architecture for grounding Large Language Models in external knowledge bases. However, production implementations frequently encounter structural limits when relying entirely on naive vector search. Standard Vector RAG fragments documents into arbitrary chunks and retrieves top-k passages via cosine similarity in embedding space. While effective for localized fact retrieval, this approach struggles with global, corpus-wide synthesis an

    1 min