PanGu-Σ vs. DeepSeek-V3.2-Exp

Recent advancements in natural language processing, understanding, and generation have largely stemmed from the evolution of large language models. This study introduces a system that utilizes Ascend 910 AI processors alongside the MindSpore framework to train a language model that surpasses one trillion parameters, achieving a total of 1.085 trillion, designated as PanGu-{\Sigma}. This model builds upon the foundation laid by PanGu-{\alpha} by transforming the traditional dense Transformer architecture into a sparse configuration via a technique called Random Routed Experts (RRE). By leveraging an extensive dataset comprising 329 billion tokens, the model was successfully trained with a method known as Expert Computation and Storage Separation (ECSS), which led to an impressive 6.3-fold increase in training throughput through the application of heterogeneous computing. Experimental results revealed that PanGu-{\Sigma} sets a new standard in zero-shot learning for various downstream tasks in Chinese NLP, highlighting its significant potential for progressing the field. This breakthrough not only represents a considerable enhancement in the capabilities of language models but also underscores the importance of creative training methodologies and structural innovations in shaping future developments. As such, this research paves the way for further exploration into improving language model efficiency and effectiveness.

We are excited to present DeepSeek-V3.2-Exp, our latest experimental model that evolves from V3.1-Terminus, incorporating the cutting-edge DeepSeek Sparse Attention (DSA) technology designed to significantly improve both training and inference speeds for longer contexts. This innovative DSA framework enables accurate sparse attention while preserving the quality of outputs, resulting in enhanced performance for long-context tasks alongside reduced computational costs. Benchmark evaluations demonstrate that V3.2-Exp delivers performance on par with V3.1-Terminus, all while benefiting from these efficiency gains. The model is fully functional across various platforms, including app, web, and API. In addition, to promote wider accessibility, we have reduced DeepSeek API pricing by more than 50% starting now. During this transition phase, users will have access to V3.1-Terminus through a temporary API endpoint until October 15, 2025. DeepSeek invites feedback on DSA from users via our dedicated feedback portal, encouraging community engagement. To further support this initiative, DeepSeek-V3.2-Exp is now available as open-source, with model weights and key technologies—including essential GPU kernels in TileLang and CUDA—published on Hugging Face, and we are eager to observe how the community will leverage this significant technological advancement. As we unveil this new chapter, we anticipate fruitful interactions and innovative applications arising from the collective contributions of our user base.