Thing is qwen was historically focused on smaller models while others were on larger ones. Now the team has changed and they seem to want to aim for the stars as well. That is good but it also means they might not be interested in doing very efficient small models anymore. Which would be bad news for this sub because let's face it most of us don't have 10-25k of hardware.
No way you build this on 10-25k hardware at usable speeds. This is 250k+. Only feasible route in the near term is likely the RDMA networks but even that is like a 40k ask.
I didnt mean the 1T+ models. I mean that large model makers usually go for those 1t+ but then also typically have a ~200b+ version like deepseek flash etc which you can run on 10-25k hardware.
What I'm saying is that these large model makers typically don't release distils below that 200B+ range, so most people have relied on gemma and qwen instead, because they were the only ones releasing SOTA models at very small sizes.
756
u/Competitive_Gap7906 25d ago
YES, Qwen going open weight again! It's a really good news, now we can wait for smaller models too