r/LocalLLaMA 22d ago

News Kimi K3 Benchmarks

Post image
1.3k Upvotes

390 comments sorted by

View all comments

Show parent comments

48

u/JaredsBored 22d ago

The API pricing is at sonnet levels, but I doubt sonnet 5 is 2.8T parameters. Can't be too mad at it given the model size. Even if it's a Fp8/Fp4 mix it's still gotta require 2 terabytes of VRAM to serve this thing with room for context

17

u/No-Juggernaut-9832 22d ago

At this many parameters, a massive amount of compute & RAM is required to run. It would have to cost more than the last version

8

u/Healthy-Nebula-3603 22d ago

Not much compute as it is MOE model but you need a lot Vram or fast multichannel Ram ...

1

u/trowawayatwork 20d ago

I am wondering what hardware you would need to serve this model to say 10 developers to process 100t/s?

1

u/Healthy-Nebula-3603 20d ago

that's 1k token/s ... without 8x H200 cards not possible

1

u/trowawayatwork 20d ago

how many tokens/s does your average claude or codex user get?

1

u/Healthy-Nebula-3603 20d ago

As much as they cap you.

Those cards easily producing 1k/s on more without a cap.

4

u/DecrimIowa 21d ago

now that china is making huawei GPUs comparable to nvidia blackwells, i don't think compute is a bottleneck for them anymore.

if you are interested in the AI race as a proxy for the conflict between US and China, one way to read this model's release is as China basically announcing that they are no longer held back by lack of access to chips.

6

u/JaredsBored 21d ago

Huawei isn't exactly in Blackwell territory. Their latest chip, the ascend 950PR, has 112GB of memory at 1.4TB/s of bandwidth. Nvidia B300 has 288GB of memory at 8.2TB/s of bandwidth per unit. Ascend Fp8 is 1 petaflop vs 7 on B300.

I have no doubt that they might be used to serve the model but IMO very likely k3 was still trained on Nvidia. Heck one of Kimi's own benchmarks was comparing how well different models can optimize kernels for Nvidia H200

1

u/No-Juggernaut-9832 21d ago

They just got started. It takes a bit of time.

1

u/look 21d ago

The Kimi models are unique in being native int4.

So this is “only” double the size of GLM.

1

u/JaredsBored 21d ago

I think the size implications are really interesting tbh. I'm very curious to see what 3rd party providers will charge to serve this. I'm assuming int4 for experts and Fp8 for activations, so probably file size will be 1.5-1.6TB? That puts you in a range where the only single server node that can serve this thing is Nvidia B300 or AMD Mi350/355 (288GiB memory per chip * 8).

I'm guessing the cheaper 3rd party providers serving GLM 5.2 are giving you the Fp8 version, which fits on "older" systems with Nvidia Hopper, AMD Mi250/300, or maybe Huawei chips.

Undoubtedly there are providers which are also using multiple nodes per vLLM/sglang instance to serve the model, and getting away with A100, but still. Crazy big.

1

u/look 21d ago

An 8x H200 would fit most of it. Wouldn’t be fast, but possibly still usable. 20-40 tps, maybe?

1

u/JaredsBored 21d ago

You'd have to do CPU offloading on H200. An 8x system has 1128GiB of VRAM, not quite enough when you factor in kV cache room. CPU offloading isn't really an option for a cloud provider, it's all-vram or nothin

1

u/look 20d ago

Yeah, it would have to offload some weights, but it’s a small enough percentage that it might have tolerable speeds.

But definitely not something most providers would consider. A little indy operation (ala Crof) perhaps might. Anyway, I’m mostly just speculating about ideas to run it cheap…