Subscribe
Topic

llama.cpp

llama.cpp runs LLMs locally on everyday hardware. Coverage includes llama.cpp merging DFlash 2 support for up to 2x faster speculative decoding on long contexts, and benchmarks of local inference on Apple Silicon.

Articles

1-2
  1. llama.cpp Merges DFlash 2 for Up to 2x Faster Decoding

    The open-source llama.cpp project has merged native support for DFlash 2, bringing parallel speculative decoding and substantial inference throughput improvements to local LLM serving across CPU, Apple Silicon, and GPU backends. The implementation, integrated via Pull Request #27342, adds local convolution operators and candidate selector mechanics designed specifically for the DFlash 2 architecture. Non-Autoregressive Speculative Drafting Standard speculative decoding uses a smaller autoreg

    llama.cpp
    4 min read
  2. Local LLM Inference on Apple Silicon: Architecture, Unified Memory, and Serving Benchmarks for MLX, llama.cpp, and Ollama

    Local large language model (LLM) serving on consumer hardware has historically faced a hard trade-off between memory capacity and execution bandwidth. Discrete consumer GPUs offer high memory bandwidth (up to 1,008 GB/s on an Nvidia RTX 4090) but are capped at 24 GB of VRAM, requiring model sharding or quantization to fit models beyond 14 billion parameters. Apple Silicon platforms bypass this capacity ceiling through a Unified Memory Architecture (UMA), where the CPU, GPU, and Apple Neural Eng

    LLM
    5 min read