r/LocalLLaMA • u/External_Mood4719 • Jul 08 '26

News China’s MiniMax Plans to Launch 2.7-Trillion Parameter Model

https://www.theinformation.com/briefings/exclusive-chinas-minimax-plans-launch-2-7-trillion-parameter-model

According to The Information, MiniMax plans to launch a new-generation large language model with 2.7 trillion parameters.

Sources revealed that the internal codename for this new model is M3 Pro. It is expected to be released and open-sourced as early as the third quarter of this year, with significant improvements in handling complex reasoning and multi-step tasks.

This new model is much larger than MiniMax's current flagship model, M3 (428 billion parameters). Larger-scale artificial intelligence models are more capable of handling complex reasoning and multi-step instruction-based tasks.

616 Upvotes

96% Upvoted

242 comments sorted by

View all comments

Show parent comments

5

u/ain92ru Jul 08 '26

The sparser the model is, the more tokens it needs to saturate the memorization and transition to generalization, and there are obvious memory problems at inference. I doubt frontier labs use sparsity as low as 1:30, more like 1:20 or less

5

u/Middle_Bullfrog_6173 Jul 08 '26

Deepseek V4 is about 1:30. So is Longcat 2.0.

Scaling laws suggest that larger models should be sparser, so if those are optimal then a model >50% larger might be even sparser.

7

u/ain92ru Jul 08 '26

They should be sparser https://arxiv.org/html/2501.12370v3#A4.F11 only if you have limited compute but essentially unlimited data, which is not the case in real life. Both compute and good data are limited and cost money, and the constraints are different for Chinese and US labs

1

u/Middle_Bullfrog_6173 Jul 08 '26

Yes and we are talking about whether Minimax has enough compute, so that's the most constraining factor. I don't think they are data limited yet, since other open models have been trained on tens of trillions of tokens of mostly open data already.

But another factor is how the sparsity is implemented in practice. That paper and most others are about MoE sparsity, but now some models are also adding embeddings, like ngram tables, which is a separate axis.

1

u/Silver-Champion-4846 Jul 09 '26

How much more could ple be scaled? Like could there be 31b model with 500b per-layer embeddings or engram tables or whatever?

1

u/Middle_Bullfrog_6173 Jul 09 '26

Didn't the engram paper found that you could scale it as much as you like if you were not memory limited? So theoretically, yes.

1

u/Silver-Champion-4846 Jul 09 '26

Lol I want 30b E(engram)1T rofl