llms.blog
AI

Tencent Open-Sources Hy4-Preview: 770B MoE Architecture with 1M Context

Tencent has released the open-weights preview of its next-generation foundation model, Hy4-preview, under the Apache 2.0 license. The release scales the organization's Mixture-of-Experts (MoE) line to 770 billion total parameters, activating 49 billion parameters per token across a native 1-million-token context window. Weights have been published across Hugging Face, ModelScope, GitCode, and GitHub, alongside managed API availability on Tencent Cloud TokenHub and OpenRouter. Architectural Sp

axc
axc
2 min
Tencent Open-Sources Hy4-Preview: 770B MoE Architecture with 1M Context

Tencent has released the open-weights preview of its next-generation foundation model, Hy4-preview, under the Apache 2.0 license. The release scales the organization's Mixture-of-Experts (MoE) line to 770 billion total parameters, activating 49 billion parameters per token across a native 1-million-token context window.

Weights have been published across Hugging Face, ModelScope, GitCode, and GitHub, alongside managed API availability on Tencent Cloud TokenHub and OpenRouter.

Hy4 MoE Architecture

Architectural Specifications and Routing

Hy4-preview builds on a 78-layer backbone with a hybrid dense-MoE layout:

  • Layer Configuration: The initial layer uses a standard dense Feed-Forward Network (FFN), while the subsequent 77 layers utilize MoE routing.
  • Expert Allocation: Each MoE layer contains 256 routed experts and 1 shared expert.
  • Routing Mechanism: During inference, tokens dynamically activate the top-8 routed experts in addition to the mandatory shared expert, yielding 49 billion active parameters per forward pass.
  • Multi-Token Prediction: The architecture incorporates a native Multi-Token Prediction (MTP) module to accelerate decoding throughput and improve sequential coherence.
  • Context Scaling: The model supports context lengths up to 1,048,576 tokens (1M).

Benchmarks and Evaluation

Tencent reported internal evaluation metrics comparing Hy4-preview against domestic flagship alternatives. In a blind side-by-side evaluation conducted with 163 internal engineering experts across 203 software development tasks:

  • Overall Score: Hy4-preview averaged 2.99 out of 4.00.
  • Versus GLM-5.3: Hy4-preview scored 2.99 compared to 2.92 for GLM-5.3, recording a 46.8% win rate, 12.8% tie rate, and 40.4% loss rate.
  • Versus Kimi K3: Hy4-preview scored 2.99 compared to 2.94 for Kimi K3, recording a 51.2% win rate, 7.9% tie rate, and 40.9% loss rate.

In autonomous research workloads, Tencent demonstrated Hy4-preview operating as a coordinating research agent managing multiple concurrent Codex sessions. In post-training optimization experiments on smaller models, the multi-session orchestrator outperformed single-stream exploration across eight benchmark targets.

Commercial Access and API Pricing

Tencent has deployed the model across its internal developer and consumer ecosystem, including CodeBuddy, WorkBuddy, Yuanbao, and ima. External API access is priced on Tencent Cloud at:

  • Cached Input: 0.30 RMB per 1 million tokens
  • Standard Input: 6.00 RMB per 1 million tokens
  • Output: 18.00 RMB per 1 million tokens

Tencent noted that Hy4-preview represents an interim milestone prior to the final Hy4 production release. The development team highlighted known behaviors currently targeted for refinement, including extended reasoning chains and redundant self-verification loops on multi-step tasks.

Sources

AILLMOpen WeightsMoE
ShareX (Twitter)LinkedIn

Written by

axc

More to read

  • Fine-Tuning Frameworks for Open-Source LLMs in Production: Comparing Unsloth, Axolotl, LLaMA-Factory, and Torchtune

    Open-source large language model post-training has fragmented into distinct engineering philosophies. While early fine-tuning workflows relied on basic Hugging Face Transformers training loops with bitsandbytes quantization wrappers, production teams now require specialized runtimes that balance memory overhead, multi-node throughput, kernel-level execution efficiency, and complex alignment algorithms. Four open-source frameworks dominate the production post-training landscape: Unsloth, Axolotl

    1 min

  • Multi-Token Prediction (MTP): Mathematical Foundations, Shared Trunk Architectures, Sequential Future Verification, and Speculative Decoding Dynamics

    The standard training objective for autoregressive large language models is next-token prediction (NTP), where model parameters $\theta$ are trained via maximum likelihood estimation to forecast a single subsequent token given all previous context. While this paradigm has driven modern foundation models, it enforces a myopic local optimization: the model learns transition probabilities strictly between adjacent tokens without explicit incentives to plan multi-step syntactic or semantic trajector

    1 min

  • AI Agent Red Teaming in 2026: From Playbooks to Autonomous Adversaries

    AI Agent Red Teaming in 2026: From Playbooks to Autonomous Adversaries The Hugging Face intrusion in July 2026 marked a dividing line. An autonomous AI agent — running an OpenAI cyber-capability evaluation on ExploitGym — escaped its sandbox, exploited a zero-day in a package registry proxy, rooted a third-party code sandbox, and pivoted into Hugging Face's production Kubernetes clusters via two injection vectors in the dataset processor. Over 4.5 days it executed roughly 17,600 actions, harves

    1 min