llms.blog
Edge AI11 articles

Edge AI

Articles

  • Local LLM Inference Frameworks in Production: Comparing llama.cpp, Ollama, Apple MLX, and Exo Distributed Clusters

    The deployment landscape for large language models is bifurcating. While datacenter workloads rely on high-throughput continuous batching engines such as vLLM and TensorRT-LLM, local and edge deployments operate under fundamentally different physical constraints. On developer workstations, embedded hardware, and private office clusters, inference is rarely bound by compute saturation across thousands of concurrent requests. Instead, it is constrained by memory bandwidth, local VRAM capacity, hos

    1 min

  • Activation-Aware Weight Quantization (AWQ): Mathematical Foundations, Salient Weight Protection, and INT4 Tensor Core Execution

    Activation-Aware Weight Quantization (AWQ): Mathematical Foundations, Salient Weight Protection, and INT4 Tensor Core Execution Large language models have transformed AI applications, but their deployment remains constrained by memory and compute barriers. A 70B parameter model in FP16 occupies ~140 GB of VRAM — exceeding even the 192 GB of NVIDIA's flagship B200 GPU, let alone edge devices. Quantization addresses this by reducing weight precision from 16-bit floats to 4-bit integers, shrinking

    1 min

  • NVIDIA Announces Jetson Orin Nano 2 with 78 TOPS AI Compute and 40% Power Cut

    NVIDIA has announced the Jetson Orin Nano 2, an updated entry-level robotics and edge AI computer designed to double inference throughput over the Jetson Orin Nano Super while maintaining the identical physical form factor. The module delivers up to 78 trillion operations per second (TOPS) of AI compute and reduces power consumption by 40% when matched against its predecessor's performance baseline. Targeted at robotics, autonomous delivery drones, and edge computer vision deployments, the hard

    1 min

  • Xiaomi Unveils Custom Silicon Roadmap with 6nm Xring O100 AI Accelerator and 3nm D100 Smart-Driving Processor

    Xiaomi has unveiled details of its custom semiconductor roadmap, introducing two specialized AI processors alongside its next-generation mobile system-on-chip: the 6-nanometer Xring O100 near-memory AI accelerator and the 3-nanometer Xring D100 autonomous driving chip. Both processors are manufactured by TSMC and have completed hardware validation ahead of planned commercial rollouts. The announcements follow a reported investment of more than 21 billion yuan ($3.1 billion) by Xiaomi into in-ho

    1 min

  • UK and Ukraine Sign AI Defense Pact to Share Battlefield Sensor Data and Target Detection Models

    The United Kingdom and Ukraine have signed a bilateral artificial intelligence defense partnership, granting British researchers and defense contractors access to Ukraine's battlefield data platform, Avengers AI Labs. The agreement was signed in Kyiv by British Prime Minister Andy Burnham and Ukrainian President Volodymyr Zelenskyy during Burnham's first official overseas visit. Under the framework, Britain becomes the first international partner permitted to access Ukraine's operational datase

    1 min

  • Ukrainian Officials Identify Nvidia Jetson Orin Modules in Autonomous Russian Strike Drones

    Ukrainian forensic examiners inspecting the wreckage of Russian strike drones have recovered Nvidia Jetson Orin microcomputers, confirming that Moscow is deploying commercial edge-computing hardware to run autonomous targeting systems on the battlefield. The findings, detailed in investigations by Ukrainian military specialists and reported by The New York Times, indicate that Russian engineers have integrated off-the-shelf edge AI accelerators into modified uncrewed aerial vehicles, including

    1 min

  • Decentralized and Peer-to-Peer LLM Inference in Production: Architecture, Ring Memory Partitioning, and Network Latency

    Decentralized and Peer-to-Peer LLM Inference in Production: Architecture, Ring Memory Partitioning, and Network Latency Frontier open-weight models such as Llama 3.1 405B, DeepSeek-V3, and Command R+ have expanded model capabilities, but their parameter scales exceed the physical memory limits of individual consumer and edge workstations. Running a 405-billion parameter model in 16-bit precision requires over 810 GB of memory, and even 4-bit quantized variants require roughly 230 GB of contiguo

    1 min

  • Waymo Details Custom 5nm ASIC and Heterogeneous Compute Architecture for Robotaxis

    Alphabet subsidiary Waymo has disclosed the architectural specifications of the computing platform powering its sixth-generation autonomous driving system, revealing a custom 5-nanometer application-specific integrated circuit (ASIC) engineered specifically for front-end sensor ingestion and edge machine learning. The disclosure, authored by VP of Engineering Satish Jeyachandran and Compute Lead Daniel Rosenband, details how the company processes raw data from onboard sensor arrays in real time

    1 min

  • Liquid AI Releases Quantization-Aware Distilled Q4_0 Checkpoints for LFM2.5 Models

    Liquid AI has released Quantization-Aware Distillation (QAD) Q4_0 GGUF checkpoints for its LFM2.5 model series, allowing edge runtimes to execute 4-bit quantized non-transformer architectures without the accuracy degradation typically associated with standard post-training quantization (PTQ). The release covers four models in the LFM2.5 family: LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B. The checkpoints are packaged in the standard GGUF format and run across llama.cpp and c

    1 min

  • On-Device LLM Inference in Production: Architecture, Runtimes, and Hardware Constraints

    Deploying generative language models directly onto edge devices such as smartphones, laptops, embedded systems, and browser sandboxes marks a fundamental shift in AI systems engineering. Moving inference from centralized GPU clusters to client silicon eliminates cloud API costs, cuts network latency to zero, guarantees data privacy by keeping user inputs local, and enables offline functionality. However, executing modern autoregressive models on resource-constrained client hardware presents str

    1 min

  • Liquid AI ships LFM2-VL-3B, a 3B vision model built for the edge

    Liquid AI has released LFM2-VL-3B, a 3 billion parameter vision-language model designed to run on edge hardware rather than in a data center. It is the largest model in the company’s LFM2-VL family, which previously topped out at 1.6B parameters, and is available now under the LFM Open License on Hugging Face and through Liquid AI’s LEAP platform. The model pairs Liquid AI’s densest text backbone, LFM2-2.6B, with a SigLIP2 400M NaFlex vision encoder. That combination lets it take in images at t

    1 min