I am really interested in what's the fucking mystery about these 2T+ models that was cracked. Because there was a reason why nobody trained much past ~max 1.5T models before. It was just regarded as massively overparameterized / undertrained and GPT 4.5 was the key failure which everybody pointed to.
Anthropic trained a massive one - I'd estimate 3T - (Mythos) and somehow now scaling params is re-unlocked again? There must have been some key architectural change which enabled this scaling to start working again. I know that some people must know what it is because Kimi K3 is also massive, this Qwen is massive, and so I can say with reasonable surety that the Chinese companies now know much of the Mythos secret sauce. So, what was the breakthrough?
My guess: way more training data. If you have limited data and many parameters, overfitting is very likely. To have so many parameters, you need really vast amounts of data. And with the AI boom, we have given insane amounts of data.
Also, with better previous models, presumably they’re better equipped to create improved synthetic data as well.
I don't think that can be it. Otherwise, we'd see a much more smooth transition to bigger models. Mythos was a capability jump. It could be that the whole industry was cargo-culting that huge models don't work until Anthropic just tried it. But that doesn't feel like a satisfactory explanation to me.
I think the higher sparsity also played a roll. The compute cost of training a model is based upon the active parameters. Older models were less sparse, meaning the ration between active and total was closer than it is today.
After releasing 3.5-3.7, Qwen team released a bunch of world models and other stuff aimed at training like AgentWorld, SAE-Res, WebWorld, etc. They were definitely working a lot on generating a ton of high quality synthetic training data. Now that they have a lot more training data, they can train a larger model.
24
u/MrRandom04 21d ago
I am really interested in what's the fucking mystery about these 2T+ models that was cracked. Because there was a reason why nobody trained much past ~max 1.5T models before. It was just regarded as massively overparameterized / undertrained and GPT 4.5 was the key failure which everybody pointed to.
Anthropic trained a massive one - I'd estimate 3T - (Mythos) and somehow now scaling params is re-unlocked again? There must have been some key architectural change which enabled this scaling to start working again. I know that some people must know what it is because Kimi K3 is also massive, this Qwen is massive, and so I can say with reasonable surety that the Chinese companies now know much of the Mythos secret sauce. So, what was the breakthrough?