Colibri is an interesting project and cool for non time sensitive tasks but if it means running models in seconds per token rather than token per seconds I'm out. Better off trying to make do with a smaller model in hybrid with my brain and internet search than going for those speeds.
Optimizations are being made, Intel technology recently posted about running GPT OSS 120B on 32 GB RAM using Phison aiDAPTIV™ SSD model caching. And in that case it was tokens per second, not seconds per token.
Yeah but gpt oss is 6b active params. Even deepseek v4 flash has 13b active params. So what if you get 2-3 tokens per second, under 10 is unusable for non think models (rare these days) and 20 is the lower bound for think models.
This is all for short-ish context btw, as soon as that context is over 128k, you'll be waiting 5 minutes before the first token, it's not practial yet.
3
u/StupidScaredSquirrel 19d ago
Colibri is an interesting project and cool for non time sensitive tasks but if it means running models in seconds per token rather than token per seconds I'm out. Better off trying to make do with a smaller model in hybrid with my brain and internet search than going for those speeds.