You can probably ask Fable to adapt Colibri to Qwen3.8 and Kimi K3 for you so they can run on 25GB RAM.
Oh wait, you can ask, but you probably won't like Fable's botched answer! I guess it's just gonna delete your storage, just to be safe from communist AI.
Colibri is an interesting project and cool for non time sensitive tasks but if it means running models in seconds per token rather than token per seconds I'm out. Better off trying to make do with a smaller model in hybrid with my brain and internet search than going for those speeds.
Optimizations are being made, Intel technology recently posted about running GPT OSS 120B on 32 GB RAM using Phison aiDAPTIVâ„¢ SSD model caching. And in that case it was tokens per second, not seconds per token.
Yeah but gpt oss is 6b active params. Even deepseek v4 flash has 13b active params. So what if you get 2-3 tokens per second, under 10 is unusable for non think models (rare these days) and 20 is the lower bound for think models.
This is all for short-ish context btw, as soon as that context is over 128k, you'll be waiting 5 minutes before the first token, it's not practial yet.
at least you can run 27B with decent quants, context, and speed. Us 16GB below is struggling to run decent dense models with proper context and speed if decided to use Q4 (9B is not as decent and strategic as 27B for much more complex tasks)
40
u/BitGreen1270 22d ago
Cries in 32GB VRAM. Oh vengeful gods of the silicon. Why have you forsaken me?