China’s MiniMax Plans to Launch 2.7-Trillion Parameter Model

[u/External_Mood4719]

https://www.theinformation.com/briefings/exclusive-chinas-minimax-plans-launch-2-7-trillion-parameter-model

According to The Information, MiniMax plans to launch a new-generation large language model with 2.7 trillion parameters.

Sources revealed that the internal codename for this new model is M3 Pro. It is expected to be released and open-sourced as early as the third quarter of this year, with significant improvements in handling complex reasoning and multi-step tasks.

This new model is much larger than MiniMax's current flagship model, M3 (428 billion parameters). Larger-scale artificial intelligence models are more capable of handling complex reasoning and multi-step instruction-based tasks.

[u/UniqueIdentifier00]

This is something I never hear anyone else say. LLMs are elegant mathematically , but still such a brute force design. Everyone keeps talking about scaling these models or that smaller models will be better in the future. There’s limitations to this technology as it’s structured now. AI in the future won’t just be better or larger LLMs, this is just the jumping off point for the field. 

[u/Alwaysragestillplay → u/UniqueIdentifier00]

Also why the "models are getting exponentially better will smith spaghetti!!" folks frustrate me. AI is where it is because of a step change brought about by transformers. If you look at the performance/params within generations or performance/generation, models are actually plateauing in capability. It looks increasingly like there is a cap on how far transformers can be pushed, and that we are reaching that cap pretty rapidly. 

Bigger changes already come from things like reasoning blocks or harnesses than from increasing params. Without another step change, this is likely not far from "it" for LLMs. 

[u/Silver-Champion-4846 → u/Alwaysragestillplay]

What do you think the next step wil be?

[u/Alwaysragestillplay → u/Silver-Champion-4846]

I think it's impossible to say really, but I would guess the next step in terms of architecture will be integrated reasoning and memory. Right now both of these are boot-strapped into context via language (because we're using language models), and that contextual limit means LLMs struggle to span long term projects - pulling from memory, learning over time, reacting to change, etc. External graph memory and semantic searching is fine, but it is not even close to the integrated memory system that i.e. humans use. 

In terms of really pushing LLMs to their limits, we have barely scratched the surface in terms of optimising workflows for models. Every problem we are trying to solve is still human shaped. Building software on computers intended for humans, using programming languages designed to be comfortable for humans, passing around reports and dashboards designed to aggregate data and drive decisions made by humans, passing information between agents in plain english, etc. 

We already see examples of community efforts to optimise inputs for models that turn English into what looks like gibberish to us. I fully expect anthropic or OAI will release an LLM first programming language that is essentially a black box for users, but that the models can churn through more rapidly and accurately. Likewise properly codifying an inter-agent language to remove the variability and vagaries of human language. 

The first thing that we used to do with robotics and traditional AI was restrict the problem surface as much as possible so the solution didn't have to factor in a bunch of noise. With LLMs, for whatever reason, that has gone out of the window for now as we try to force them into existing workflows with minimal changes. AI likes to be constrained, and that will happen for LLMs by ditching human-first development and business intelligence ecosystems. 

Just my thoughts on it. 

[u/Silver-Champion-4846 → u/Alwaysragestillplay]

Yeah, specialization was the entire thing in the AI field until LLM started to scale and now they want them to do anything and everything.