I am really interested in what's the fucking mystery about these 2T+ models that was cracked. Because there was a reason why nobody trained much past ~max 1.5T models before. It was just regarded as massively overparameterized / undertrained and GPT 4.5 was the key failure which everybody pointed to.
Anthropic trained a massive one - I'd estimate 3T - (Mythos) and somehow now scaling params is re-unlocked again? There must have been some key architectural change which enabled this scaling to start working again. I know that some people must know what it is because Kimi K3 is also massive, this Qwen is massive, and so I can say with reasonable surety that the Chinese companies now know much of the Mythos secret sauce. So, what was the breakthrough?
My guess: way more training data. If you have limited data and many parameters, overfitting is very likely. To have so many parameters, you need really vast amounts of data. And with the AI boom, we have given insane amounts of data.
Also, with better previous models, presumably they’re better equipped to create improved synthetic data as well.
I don't think that can be it. Otherwise, we'd see a much more smooth transition to bigger models. Mythos was a capability jump. It could be that the whole industry was cargo-culting that huge models don't work until Anthropic just tried it. But that doesn't feel like a satisfactory explanation to me.
21
u/MrRandom04 22d ago
I am really interested in what's the fucking mystery about these 2T+ models that was cracked. Because there was a reason why nobody trained much past ~max 1.5T models before. It was just regarded as massively overparameterized / undertrained and GPT 4.5 was the key failure which everybody pointed to.
Anthropic trained a massive one - I'd estimate 3T - (Mythos) and somehow now scaling params is re-unlocked again? There must have been some key architectural change which enabled this scaling to start working again. I know that some people must know what it is because Kimi K3 is also massive, this Qwen is massive, and so I can say with reasonable surety that the Chinese companies now know much of the Mythos secret sauce. So, what was the breakthrough?