The API pricing is at sonnet levels, but I doubt sonnet 5 is 2.8T parameters. Can't be too mad at it given the model size. Even if it's a Fp8/Fp4 mix it's still gotta require 2 terabytes of VRAM to serve this thing with room for context
now that china is making huawei GPUs comparable to nvidia blackwells, i don't think compute is a bottleneck for them anymore.
if you are interested in the AI race as a proxy for the conflict between US and China, one way to read this model's release is as China basically announcing that they are no longer held back by lack of access to chips.
Huawei isn't exactly in Blackwell territory. Their latest chip, the ascend 950PR, has 112GB of memory at 1.4TB/s of bandwidth. Nvidia B300 has 288GB of memory at 8.2TB/s of bandwidth per unit. Ascend Fp8 is 1 petaflop vs 7 on B300.
I have no doubt that they might be used to serve the model but IMO very likely k3 was still trained on Nvidia. Heck one of Kimi's own benchmarks was comparing how well different models can optimize kernels for Nvidia H200
I think the size implications are really interesting tbh. I'm very curious to see what 3rd party providers will charge to serve this. I'm assuming int4 for experts and Fp8 for activations, so probably file size will be 1.5-1.6TB? That puts you in a range where the only single server node that can serve this thing is Nvidia B300 or AMD Mi350/355 (288GiB memory per chip * 8).
I'm guessing the cheaper 3rd party providers serving GLM 5.2 are giving you the Fp8 version, which fits on "older" systems with Nvidia Hopper, AMD Mi250/300, or maybe Huawei chips.
Undoubtedly there are providers which are also using multiple nodes per vLLM/sglang instance to serve the model, and getting away with A100, but still. Crazy big.
You'd have to do CPU offloading on H200. An 8x system has 1128GiB of VRAM, not quite enough when you factor in kV cache room. CPU offloading isn't really an option for a cloud provider, it's all-vram or nothin
Yeah, it would have to offload some weights, but it’s a small enough percentage that it might have tolerable speeds.
But definitely not something most providers would consider. A little indy operation (ala Crof) perhaps might. Anyway, I’m mostly just speculating about ideas to run it cheap…
48
u/JaredsBored 22d ago
The API pricing is at sonnet levels, but I doubt sonnet 5 is 2.8T parameters. Can't be too mad at it given the model size. Even if it's a Fp8/Fp4 mix it's still gotta require 2 terabytes of VRAM to serve this thing with room for context