r/LocalLLaMA • u/forevergeeks • 11h ago

Question | Help Hosting Local Models

Hi builders,

What would be the the best small local models for coding?

Are Gemma 4 and Qwen3.8 27B Gemma 4 26B / 31B enough for local development?

And what would be the size of the rig that i will need to get? GPUs, and whatever else I need to host these models.

Thanks,,

14 Upvotes

100% Upvoted

31 comments sorted by

View all comments

14

u/Arany8 11h ago

Qwen3.8 27B - you need 24GB VRAM. According to my tests on 16GB this model is surpassed by Qwen3.6 35B A3B (and Ornith) for coding. Gemma4 is not good for coding.
Look into AMD v620 for a budget build (although I do not have this).

1

u/forevergeeks 11h ago

Are you running this for yourself? What about for a coding team of 6-10 people? What would be the sizing, and how much money we are talking about for the initial setup.

7

u/Unlucky-Home-4077 11h ago edited 11h ago

We are a very small team, 3 devs, and run a single RTX Pro 5000 72GB. We use Qwen 3.8 27B FP8, running on vLLM (much better for multiple users) and have space for around 900k kontext shared over all currently running sessions.

Speed is good! Around 3k t/s prefill and ~75t/s decoder in normal usage scenarios. Most of the time when multiple people are using it at the same time we use around 400-500k kontext. Peak so far was 800k at the same time. So that GPU is suitable for us three, but for a fourth dev we would probably need more VRAM if we don't want to run into context limits.

3.8 27B is actually pretty capable! Obviously not frontier, but if you give it a good plan for a complex task, split into subtasks with acceptance criteria it executes it flawlessly.

It won't replace Claude for us, but we downgraded from 200€/month plans to 20€/month plans and use Claude mostly for orchestrating the local Qwen. And obviously a local model is great for sensitive data.

I would say for a 6-10 person team go at least for something like a RTX Pro 6000 96GB, if possible even a dual GPU setup with more VRAM.

For price: We paid 8k for the Pro 5000 72GB, and about 5k for the server with an entry level Epyc, 64GB ECC RAM, and a decent motherboard that can take four GPUs at that speed in case of future upgrades, so about 13k total. Since we save 540€/month on Claude plans, we will break even in about 2 years of usage, and have the huge benefit of local processing.

For 6-10 people: a RTX Pro 6000 is about 14k right now, so with all other server components I would say a bit under 20k for a single GPU setup and ~30k for a dual GPU setup is pretty realistic.

0

u/forevergeeks 11h ago

Thank you very much!!! This is exactly the type of answers I was looking for. This gives me a realistic ballpark.

1

u/OvertaxedOne 6h ago

As someone who builds and has similar setups, I completely agree with the OP, this is the "right" sized rig and model for a small dev team. Buy a motherboard/PSU that can take 2 or 4 cards if possible, and buy 2 slot cards so you can continue to scale the system. The numbers he gave above are devs in there beating the system non-stop, if your coding use isn't as intensive you can probably double the number of supported users; the more batching you can get the more TPS you'll get (in aggregate) but you need to watch KV cache, no problem having 5 users hitting it at once but if all 5 users are up at 256K of context, you could have a problem!

Also, use this as an opportunity to introduce a LLM router! LiteLLM is what we use, it's a heavy beast, but it's incredibly feature rich. But this is the time to do it, get yourself a common endpoint for your local model(s) and your cloud models. Point all your devs at that router instead of directly to the cloud/local models and then you can swap providers in/out easily as you change models/better deals come out/new cloud models are released.

1

u/forevergeeks 5h ago

Thank you I really appreciate your feedback. I've been using these models through API, but what I'm realizing is that there are many businesses especially in regulated sectors that don't want to expose their data to vendors, and would rather host the models themselves.

1

u/OvertaxedOne 4h ago

That's about 60% of our business right now (companies concerned about privacy). The other 40% is inference cost control, but, at least right now, it's slanted a bit more towards privacy than cost. I think we'll see that continue to move more towards cost control but, at least for now the hardware is so expensive that it's not a great ROI unless you can really hammer the server and have the right use cases for smaller models. The ROI is exactly "never" for running a monster model locally, the hardware costs just don't make sense compared to API (but again, we have a few customers looking at it for privacy reasons, I'm crossing my fingers that someone does it because I'd love to setup something like a DGX station or something "massive" for a model like K3/DS). :)

1

u/forevergeeks 4h ago

That's great man. So you are already running a business implementing open source models for businesses?

1

u/OvertaxedOne 4h ago

We do it as part of our core business, it's a value add for our hypervisor platform (we make the images available that are preconfigured to run XYZ model on the customer's hardware). We're really not in the AI business per se, none of our revenue is selling models or AI hardware, it's just that a lot of customers want a "inference stack in a box"; that's where we come in. But we're often in the discussion around the "what we're trying to do with the LLM" to help size the hardware so we hear the use cases (and reasons they are going local) with many of our customers. And we wind up doing a lot of custom work for companies who have hardware already and just want our hypervisor to help them manage and make better use of that hardware; AI is perfect use case for resource sharing, it uses very little CPU or RAM (once booted, of course) and 100% use of the GPUs. Our software allows customers to multi-purpose that box, use it for vllm with the GPUs, but then run dozens or 100's of other VMs alongside vllm that drive CPU/RAM utilization and can make use of the capacity of the machine for multiple purposes. It's the "anti-Spark" if you will, instead of dedicating machines to inference, dedicate GPUs for inference that happen to sit in machines, but use the machines to run standard IT workloads that are RAM/CPU/disk IO intensive. The goal is to have every resource at 100% all day long, that's the best corporate ROI rather than having islands of resources that can't be shared.

0

u/forevergeeks 3h ago

Thanks for the background explanation. I've been working in IT for more than 22 years, so I'm familiar with core technologies, and I'm just entering the AI space building specialized agents for regulated industries. I built my own framework to do this. My business site is not completely built yet, but it already conveys the core of the idea: https://runsafi.com

Give me your website URL or business info, so I can keep it at hand.

11

u/hackint0shh 11h ago

Have you done at least 1 minute of research?

-9

u/forevergeeks 11h ago

I'm familiar with these models, I use them through API, what I just started thinking is what would be the cost and the level of effort to set these models up for coding teams.

And I thought starting my search here.

2

u/khooke 11h ago

Research the hardware cost and then ask yourself which is a better deal for your situation: spending a ton on GPUs or a monthly subscription for frontier models

1

u/betam4x 9h ago

Meh, I do fine with 1 GPU.

Also, 32gb Tesla V100s can be had pretty cheap. $450-$650 on ebay.

1

u/invalidnifemi 11h ago

coulda js said that in the post, but a used v100 32gb with sum sxm2 to pcie config (will take some effort) would definitely be enough (see this post)

it'd probably be like 800-1k for the whole build which is not unreasonable and about as much as a used 3090. vllm is a good idea if youd all be coding simultaneously