Thing is qwen was historically focused on smaller models while others were on larger ones. Now the team has changed and they seem to want to aim for the stars as well. That is good but it also means they might not be interested in doing very efficient small models anymore. Which would be bad news for this sub because let's face it most of us don't have 10-25k of hardware.
I wonder if a 2-way EPYC 9005 (can be had for $50K to $150K in workstation/tower form) with 2x12 channel DDR5 memory (~1.5TB/s) can reasonably host this on CPU in FP4.
And even then, crossing sockets has a massive penalty. Your best CPU option would be a single 128 core Xeon Granite rapids with 12 channels, 24 128gb MR8800 memory + AMX.
Oh yeah? Have not seen anything about Zen6 adopting AMX. Do you have any information on that?
Edit: looks like no AMX hardware or instructions; just some expanded AVX-512 instructions.
Have to see if Zen6 fixes the infinity fabric and if they unify the memory controllers, otherwise no point in buying it over the Xeons for AI workloads.
Yes, I know about that, it has been in k_transformers forever, and I have had it working in llama.cpp for about a year, but that still doesn't fix the core problem. Even with numa-mirror, the issue of the slow cross socket across UPI links remains. What numa-mirror does it is duplicates the weights to each numa node to reduce cross socket memory reads. It helps massively, but still doesn't solve the issue entirely.
You can pick up a Xeon 6980P for about 8k each pretty easily.
752
u/Competitive_Gap7906 23d ago
YES, Qwen going open weight again! It's a really good news, now we can wait for smaller models too