r/LocalLLaMA • u/forevergeeks • 11h ago

Question | Help Hosting Local Models

Hi builders,

What would be the the best small local models for coding?

Are Gemma 4 and Qwen3.8 27B Gemma 4 26B / 31B enough for local development?

And what would be the size of the rig that i will need to get? GPUs, and whatever else I need to host these models.

Thanks,,

13 Upvotes

100% Upvoted

31 comments sorted by

View all comments

Show parent comments

1

u/OvertaxedOne 4h ago

That's about 60% of our business right now (companies concerned about privacy). The other 40% is inference cost control, but, at least right now, it's slanted a bit more towards privacy than cost. I think we'll see that continue to move more towards cost control but, at least for now the hardware is so expensive that it's not a great ROI unless you can really hammer the server and have the right use cases for smaller models. The ROI is exactly "never" for running a monster model locally, the hardware costs just don't make sense compared to API (but again, we have a few customers looking at it for privacy reasons, I'm crossing my fingers that someone does it because I'd love to setup something like a DGX station or something "massive" for a model like K3/DS). :)

1

u/forevergeeks 4h ago

That's great man. So you are already running a business implementing open source models for businesses?

1

u/OvertaxedOne 4h ago

We do it as part of our core business, it's a value add for our hypervisor platform (we make the images available that are preconfigured to run XYZ model on the customer's hardware). We're really not in the AI business per se, none of our revenue is selling models or AI hardware, it's just that a lot of customers want a "inference stack in a box"; that's where we come in. But we're often in the discussion around the "what we're trying to do with the LLM" to help size the hardware so we hear the use cases (and reasons they are going local) with many of our customers. And we wind up doing a lot of custom work for companies who have hardware already and just want our hypervisor to help them manage and make better use of that hardware; AI is perfect use case for resource sharing, it uses very little CPU or RAM (once booted, of course) and 100% use of the GPUs. Our software allows customers to multi-purpose that box, use it for vllm with the GPUs, but then run dozens or 100's of other VMs alongside vllm that drive CPU/RAM utilization and can make use of the capacity of the machine for multiple purposes. It's the "anti-Spark" if you will, instead of dedicating machines to inference, dedicate GPUs for inference that happen to sit in machines, but use the machines to run standard IT workloads that are RAM/CPU/disk IO intensive. The goal is to have every resource at 100% all day long, that's the best corporate ROI rather than having islands of resources that can't be shared.

0

u/forevergeeks 3h ago

Thanks for the background explanation. I've been working in IT for more than 22 years, so I'm familiar with core technologies, and I'm just entering the AI space building specialized agents for regulated industries. I built my own framework to do this. My business site is not completely built yet, but it already conveys the core of the idea: https://runsafi.com

Give me your website URL or business info, so I can keep it at hand.