Subscribe
Topic

Edge AI

Running AI on phones, laptops and embedded devices. Posts compare small language models like SmolLM2 and Llama 3.2, on-device runtimes such as ExecuTorch and MLC-LLM, and local tools like llama.cpp and Ollama, plus edge hardware like Nvidia Jetson.

Articles

1-20
  1. Open-Source Sub-3B Small Language Models in Production: Comparing SmolLM2, Qwen2.5, Llama 3.2, and Gemma 2

    Deploying language models directly on client hardware, edge gateways, and low-latency routing layers requires balancing parameter count, memory footprint, and task-specific reasoning capability. The sub-3B parameter class has shifted from stripped-down toy models to highly capable engines trained on tens of trillions of tokens or distilled from frontier teachers. In production architectures, models in the 1B to 3B range serve three primary roles: speculative decoding draft models that accelerat

    AI
    6 min read
  2. Local LLM Inference Frameworks in Production: Comparing llama.cpp, Ollama, Apple MLX, and Exo Distributed Clusters

    The deployment landscape for large language models is bifurcating. While datacenter workloads rely on high-throughput continuous batching engines such as vLLM and TensorRT-LLM, local and edge deployments operate under fundamentally different physical constraints. On developer workstations, embedded hardware, and private office clusters, inference is rarely bound by compute saturation across thousands of concurrent requests. Instead, it is constrained by memory bandwidth, local VRAM capacity, hos

    AI
    6 min read
  3. Activation-Aware Weight Quantization (AWQ): Mathematical Foundations, Salient Weight Protection, and INT4 Tensor Core Execution

    Activation-Aware Weight Quantization (AWQ): Mathematical Foundations, Salient Weight Protection, and INT4 Tensor Core Execution Large language models have transformed AI applications, but their deployment remains constrained by memory and compute barriers. A 70B parameter model in FP16 occupies ~140 GB of VRAM — exceeding even the 192 GB of NVIDIA's flagship B200 GPU, let alone edge devices. Quantization addresses this by reducing weight precision from 16-bit floats to 4-bit integers, shrinking

    AI
    6 min read
  4. NVIDIA Announces Jetson Orin Nano 2 with 78 TOPS AI Compute and 40% Power Cut

    NVIDIA has announced the Jetson Orin Nano 2, an updated entry-level robotics and edge AI computer designed to double inference throughput over the Jetson Orin Nano Super while maintaining the identical physical form factor. The module delivers up to 78 trillion operations per second (TOPS) of AI compute and reduces power consumption by 40% when matched against its predecessor's performance baseline. Targeted at robotics, autonomous delivery drones, and edge computer vision deployments, the hard

    Hardware
    2 min read
  5. Xiaomi Unveils Xring O100 AI Chip and D100 Driving Chip

    Xiaomi has unveiled details of its custom semiconductor roadmap, introducing two specialized AI processors alongside its next-generation mobile system-on-chip: the 6-nanometer Xring O100 near-memory AI accelerator and the 3-nanometer Xring D100 autonomous driving chip. Both processors are manufactured by TSMC and have completed hardware validation ahead of planned commercial rollouts. Xiaomi has invested more than 20 billion yuan (about $3 billion) in Xring chip development, Reuters reported,

    AI
    4 min read