Subscribe
Category / page 14

Chips & Infrastructure

314 articles

Articles, page 14

261-280 of 314

  1. Beijing Clears Initial Shipments of 10,000 Nvidia H200 Chips to ByteDance and Tencent

    Chinese regulators have authorized the delivery of initial batches of Nvidia H200 artificial intelligence processors to mainland tech giants, marking a pivotal development in Beijing's management of high-performance compute access. According to reporting from the Financial Times, ByteDance and Tencent have each received roughly 10,000 H200 accelerators at their mainland data center facilities in recent weeks. Several additional domestic technology companies are currently awaiting clearance for

    AI
    1 min read

  2. FlashDecoding: How Sequence Partitioning Solved the Memory Bandwidth Bottleneck in LLM Generation

    FlashDecoding: How Sequence Partitioning Solved the Memory Bandwidth Bottleneck in LLM Generation In large language model serving, execution divides into two distinct operational regimes: prompt prefill and autoregressive token generation (decoding). While FlashAttention transformed prefill throughput by eliminating High Bandwidth Memory (HBM) round-trips for intermediate attention matrices, standard FlashAttention algorithms encounter a severe hardware utilization bottleneck during decoding.

    AI
    7 min read

  3. Z.ai Opens GLM-5.3 API Access at $1.40/$4.40 per Million Tokens with Prompt Caching

    Chinese foundation model developer Z.ai (Zhipu AI) has opened public API access to GLM-5.3, offering developers direct endpoint integration following the model's initial release. The company kept base token rates aligned with the prior generation while introducing discounted prompt caching. GLM-5.3 is priced at $1.40 per million input tokens and $4.40 per million output tokens on the Z.ai platform. For workloads utilizing prompt caching, cached input tokens are billed at $0.26 per million, an 8

    LLM
    2 min read

  4. FP8 Mixed-Precision Training: Formats, Scaling Recipes, and Fine-Grained GEMMs

    FP8 Mixed-Precision Training: Formats, Scaling Recipes, and Fine-Grained GEMMs Training frontier large language models requires navigating harsh memory bandwidth and compute boundaries. For years, 16-bit formats such as IEEE FP16 and Brain Floating Point (BF16) served as the standard arithmetic precision for deep learning workloads. While BF16 provided sufficient dynamic range to prevent gradient underflow without manual loss scaling, training clusters still spent substantial time bounded by me

    AI
    8 min read

  5. Z Lab Releases DFlash 2 for Qwen 3.8 27B: Block Diffusion Speculative Decoding with Target KV Injection

    Z Lab has released DFlash 2 checkpoints for Alibaba's Qwen 3.8 27B model family, advancing block-diffusion speculative decoding for open-weights LLM serving. By replacing conventional autoregressive draft models with a non-causal diffusion mechanism paired with direct target key-value (KV) cache injection, the framework achieves up to 3x to 4.3x throughput speedups in production inference engines like SGLang and vLLM without altering output token distributions. For where the 27B fits in the line

    Inference
    2 min read

  6. Pennsylvania Restricts Speculative AI Data Centers in Executive Order 2026-05

    Pennsylvania Governor Josh Shapiro has signed Executive Order 2026-05, introducing strict regulatory standards on high-capacity data center construction and ending the state's expedited permitting program for computing facilities. The directive requires prospective developers of large-scale facilities to enter legally binding commitments with the Commonwealth to safeguard local power grids, protect municipal water supplies, and secure approval from local governments before receiving state enviro

    Policy
    2 min read

  7. Activation Checkpointing in Large Language Models: How Selective Recomputation Eliminates Memory Bottlenecks

    Large language model pre-training and fine-tuning are fundamentally constrained by GPU memory (VRAM). While distributed techniques such as Fully Sharded Data Parallel (FSDP), ZeRO, and Tensor Parallelism successfully shard model parameters, optimizer states, and gradients across hundreds or thousands of GPUs, activation memory presents a distinct scaling bottleneck. During the forward pass of a transformer model, intermediate tensor outputs must be preserved in GPU memory so that backpropagatio

    AI
    8 min read

  8. Cerebras Unveils CS-4 Rack-Scale System Powered by Three WSE-3 Turbo Chips and Nexus Architecture

    Cerebras Unveils CS-4 Rack-Scale System Powered by Three WSE-3 Turbo Chips and Nexus Architecture Cerebras Systems has announced the CS-4, a rack-scale AI accelerator system designed around three of its next-generation Wafer Scale Engine 3 Turbo (WSE-3 Turbo) chips and a modular hardware architecture dubbed Nexus. Cerebras confirmed that initial customer shipments for the CS-4 are scheduled to begin in the current quarter. The new system marks a structural shift from Cerebras's single-wafer CS

    Hardware
    2 min read

  9. Self-Hosted Embedding and Reranking Serving in Production: TEI vs. Infinity vs. vLLM Architecture, Dynamic Batching, and Serving Economics

    While generative large language models dominate inference infrastructure discussions, vector embeddings and cross-encoder rerankers handle order-of-magnitude higher request volumes in production retrieval-augmented generation (RAG) and search pipelines. Serving embedding and reranking models presents fundamentally different computational characteristics than auto-regressive text generation. Without auto-regressive token generation loops or key-value (KV) cache state management, the primary engin

    AI
    5 min read

  10. Modular Open-Sources Mojo Language Compiler and Toolchain Under Apache 2.0

    Modular Open-Sources Mojo Language Compiler and Toolchain Under Apache 2.0 Modular has released the complete source code for the Mojo programming language compiler, standard tooling, and runtime infrastructure under the Apache 2.0 license with LLVM exceptions. The announcement, delivered on August 18, 2026 during the company's ModCon developer conference, fulfills a multi-year roadmap commitment to transition the systems programming language to a fully open development model. The compiler sour

    AI
    2 min read

  11. Alibaba Demonstrates Native Qwen 3.8 27B Inference on XuanTie C950 RISC-V CPU at 30 Tokens per Second

    Alibaba's semiconductor division, T-Head, announced day-zero native inference support for its latest open-weight model, Qwen 3.8 27B, running directly on the XuanTie C950 RISC-V server processor. Operating without discrete graphics processing units, the 64-core RISC-V chip delivered sustained decode throughput of 30 tokens per second alongside a time-to-first-token latency of 1.9 seconds. The benchmark demonstrates how architectural extensions on general-purpose open instruction sets can handle

    Hardware
    2 min read

  12. Local LLM Inference on Apple Silicon: Architecture, Unified Memory, and Serving Benchmarks for MLX, llama.cpp, and Ollama

    Local large language model (LLM) serving on consumer hardware has historically faced a hard trade-off between memory capacity and execution bandwidth. Discrete consumer GPUs offer high memory bandwidth (up to 1,008 GB/s on an Nvidia RTX 4090) but are capped at 24 GB of VRAM, requiring model sharding or quantization to fit models beyond 14 billion parameters. Apple Silicon platforms bypass this capacity ceiling through a Unified Memory Architecture (UMA), where the CPU, GPU, and Apple Neural Eng

    LLM
    5 min read

  13. Mistral Expands Platform to Host Third-Party Open Weights Starting with GLM-5.2

    Mistral AI has broadened its API platform to host external open-weight foundation models, beginning with Zhipu AI's GLM-5.2. The move marks a strategic shift for the Paris-based AI company from serving only in-house architectures (such as Mistral Small, Mistral Medium, Mistral Large, and Voxtral) toward operating as a sovereign managed inference hub for third-party open weights. The integration introduces GLM-5.2 under the model identifier zai-glm-5-2 in public preview. The model is hosted with

    Mistral
    2 min read

  14. Multi-Token Prediction: How Future Token Supervision Densifies Representations and Speeds Up LLM Serving

    Standard autoregressive language models are trained under a strict next-token prediction objective. At every sequence position, the model consumes a prefix of tokens and predicts the single immediate successor token using a cross-entropy loss. While this paradigm has scaled language modeling across orders of magnitude, it suffers from an architectural limitation: myopic optimization. By evaluating loss exclusively on the immediate next step, standard training fails to reward representations that

    AI
    6 min read

  15. Velaura AI Raises $110M Series A at $1B Valuation for Low-Power AI Silicon

    Velaura AI Raises $110M Series A at $1B Valuation for Low-Power AI Silicon Velaura AI has closed a $110 million Series A funding round at a valuation exceeding $1 billion. The financing was led by Seligman Ventures, with participation from Capricorn Investment Group alongside existing backers including Samsung Catalyst Fund, StepStone Group, Maverick Silicon, Celesta Capital, and Mayfield. The capital will fund the commercialization and deployment of Velaura's silicon IP and physical design te

    AI
    3 min read

  16. Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    AI
    5 min read

  17. LLM Serving Frameworks in Production: SGLang vs. vLLM vs. TensorRT-LLM Architecture, Memory Management, and Throughput Trade-Offs

    Serving large language models in production has moved beyond simple iteration-level scheduling. While basic continuous batching solved the fundamental problem of GPU starvation during autoregressive decoding, modern production demands have shifted the engineering bottleneck toward KV cache memory reuse, compilation overhead, and structured generation efficiency. Three dominant serving runtimes currently lead high-throughput deployments: vLLM, SGLang, and NVIDIA TensorRT-LLM. Each framework appr

    Inference
    4 min read

  18. Test-Time Compute Scaling in Large Language Models: How Search, Verification, and Reasoning Chains Trade Latency for Accuracy

    Large language model performance has historically been dictated by pre-training compute scaling laws. As formalised by Kaplan et al. and Chinchilla scaling, increasing model parameters, dataset size, and pre-training FLOPs yielded predictable reductions in cross-entropy loss. During inference, however, computation remained strictly linear and deterministic: one forward pass per generated token. The emergence of inference-time reasoning architectures has introduced a third scaling axis: test-tim

    AI
    4 min read

  19. Etched in Talks to Raise $700M Led by Jane Street at $21B Valuation

    AI inference chip startup Etched is in negotiations to raise $700 million in a new financing round led by existing investor Jane Street, according to reporting from The Wall Street Journal. The proposed financing would value the San Jose-based semiconductor company at approximately $21 billion. The planned round follows a $300 million Series C led by Sequoia Capital that valued the startup at $10.3 billion, effectively doubling its valuation within weeks amid intensifying enterprise demand for

    Hardware
    2 min read

  20. LG Partners with Nvidia on 10,000-Square-Meter Robot Data Factory Targeting 100,000 Training Hours

    LG Electronics hosted senior Nvidia leadership at its Yangjae R&D Campus in Seoul on August 18, 2026, advancing a joint initiative to build physical AI training pipelines and target 100,000 hours of embodied robotics data by the end of the year. The site review took place five days after LG Group and Nvidia signed a strategic memorandum of understanding at Nvidia headquarters in Santa Clara on August 13. The accelerated timeline reflects LG's effort to convert its industrial manufacturing infra

    Robotics
    2 min read