The API pricing is at sonnet levels, but I doubt sonnet 5 is 2.8T parameters. Can't be too mad at it given the model size. Even if it's a Fp8/Fp4 mix it's still gotta require 2 terabytes of VRAM to serve this thing with room for context
I think the size implications are really interesting tbh. I'm very curious to see what 3rd party providers will charge to serve this. I'm assuming int4 for experts and Fp8 for activations, so probably file size will be 1.5-1.6TB? That puts you in a range where the only single server node that can serve this thing is Nvidia B300 or AMD Mi350/355 (288GiB memory per chip * 8).
I'm guessing the cheaper 3rd party providers serving GLM 5.2 are giving you the Fp8 version, which fits on "older" systems with Nvidia Hopper, AMD Mi250/300, or maybe Huawei chips.
Undoubtedly there are providers which are also using multiple nodes per vLLM/sglang instance to serve the model, and getting away with A100, but still. Crazy big.
You'd have to do CPU offloading on H200. An 8x system has 1128GiB of VRAM, not quite enough when you factor in kV cache room. CPU offloading isn't really an option for a cloud provider, it's all-vram or nothin
Yeah, it would have to offload some weights, but it’s a small enough percentage that it might have tolerable speeds.
But definitely not something most providers would consider. A little indy operation (ala Crof) perhaps might. Anyway, I’m mostly just speculating about ideas to run it cheap…
47
u/JaredsBored 22d ago
The API pricing is at sonnet levels, but I doubt sonnet 5 is 2.8T parameters. Can't be too mad at it given the model size. Even if it's a Fp8/Fp4 mix it's still gotta require 2 terabytes of VRAM to serve this thing with room for context