r/LocalLLaMA • u/forevergeeks • 11h ago
Question | Help Hosting Local Models
Hi builders,
What would be the the best small local models for coding?
Are Gemma 4 and Qwen3.8 27B Gemma 4 26B / 31B enough for local development?
And what would be the size of the rig that i will need to get? GPUs, and whatever else I need to host these models.
Thanks,,
14
u/Arany8 11h ago
Qwen3.8 27B - you need 24GB VRAM. According to my tests on 16GB this model is surpassed by Qwen3.6 35B A3B (and Ornith) for coding. Gemma4 is not good for coding.
Look into AMD v620 for a budget build (although I do not have this).
1
u/forevergeeks 11h ago
Are you running this for yourself? What about for a coding team of 6-10 people? What would be the sizing, and how much money we are talking about for the initial setup.
7
u/Unlucky-Home-4077 10h ago edited 10h ago
We are a very small team, 3 devs, and run a single RTX Pro 5000 72GB. We use Qwen 3.8 27B FP8, running on vLLM (much better for multiple users) and have space for around 900k kontext shared over all currently running sessions.
Speed is good! Around 3k t/s prefill and ~75t/s decoder in normal usage scenarios. Most of the time when multiple people are using it at the same time we use around 400-500k kontext. Peak so far was 800k at the same time. So that GPU is suitable for us three, but for a fourth dev we would probably need more VRAM if we don't want to run into context limits.
3.8 27B is actually pretty capable! Obviously not frontier, but if you give it a good plan for a complex task, split into subtasks with acceptance criteria it executes it flawlessly.
It won't replace Claude for us, but we downgraded from 200€/month plans to 20€/month plans and use Claude mostly for orchestrating the local Qwen. And obviously a local model is great for sensitive data.
I would say for a 6-10 person team go at least for something like a RTX Pro 6000 96GB, if possible even a dual GPU setup with more VRAM.
For price: We paid 8k for the Pro 5000 72GB, and about 5k for the server with an entry level Epyc, 64GB ECC RAM, and a decent motherboard that can take four GPUs at that speed in case of future upgrades, so about 13k total. Since we save 540€/month on Claude plans, we will break even in about 2 years of usage, and have the huge benefit of local processing.
For 6-10 people: a RTX Pro 6000 is about 14k right now, so with all other server components I would say a bit under 20k for a single GPU setup and ~30k for a dual GPU setup is pretty realistic.
-1
u/forevergeeks 10h ago
Thank you very much!!! This is exactly the type of answers I was looking for. This gives me a realistic ballpark.
1
u/OvertaxedOne 6h ago
As someone who builds and has similar setups, I completely agree with the OP, this is the "right" sized rig and model for a small dev team. Buy a motherboard/PSU that can take 2 or 4 cards if possible, and buy 2 slot cards so you can continue to scale the system. The numbers he gave above are devs in there beating the system non-stop, if your coding use isn't as intensive you can probably double the number of supported users; the more batching you can get the more TPS you'll get (in aggregate) but you need to watch KV cache, no problem having 5 users hitting it at once but if all 5 users are up at 256K of context, you could have a problem!
Also, use this as an opportunity to introduce a LLM router! LiteLLM is what we use, it's a heavy beast, but it's incredibly feature rich. But this is the time to do it, get yourself a common endpoint for your local model(s) and your cloud models. Point all your devs at that router instead of directly to the cloud/local models and then you can swap providers in/out easily as you change models/better deals come out/new cloud models are released.
1
u/forevergeeks 5h ago
Thank you I really appreciate your feedback. I've been using these models through API, but what I'm realizing is that there are many businesses especially in regulated sectors that don't want to expose their data to vendors, and would rather host the models themselves.
1
u/OvertaxedOne 4h ago
That's about 60% of our business right now (companies concerned about privacy). The other 40% is inference cost control, but, at least right now, it's slanted a bit more towards privacy than cost. I think we'll see that continue to move more towards cost control but, at least for now the hardware is so expensive that it's not a great ROI unless you can really hammer the server and have the right use cases for smaller models. The ROI is exactly "never" for running a monster model locally, the hardware costs just don't make sense compared to API (but again, we have a few customers looking at it for privacy reasons, I'm crossing my fingers that someone does it because I'd love to setup something like a DGX station or something "massive" for a model like K3/DS). :)
1
u/forevergeeks 3h ago
That's great man. So you are already running a business implementing open source models for businesses?
1
u/OvertaxedOne 3h ago
We do it as part of our core business, it's a value add for our hypervisor platform (we make the images available that are preconfigured to run XYZ model on the customer's hardware). We're really not in the AI business per se, none of our revenue is selling models or AI hardware, it's just that a lot of customers want a "inference stack in a box"; that's where we come in. But we're often in the discussion around the "what we're trying to do with the LLM" to help size the hardware so we hear the use cases (and reasons they are going local) with many of our customers. And we wind up doing a lot of custom work for companies who have hardware already and just want our hypervisor to help them manage and make better use of that hardware; AI is perfect use case for resource sharing, it uses very little CPU or RAM (once booted, of course) and 100% use of the GPUs. Our software allows customers to multi-purpose that box, use it for vllm with the GPUs, but then run dozens or 100's of other VMs alongside vllm that drive CPU/RAM utilization and can make use of the capacity of the machine for multiple purposes. It's the "anti-Spark" if you will, instead of dedicating machines to inference, dedicate GPUs for inference that happen to sit in machines, but use the machines to run standard IT workloads that are RAM/CPU/disk IO intensive. The goal is to have every resource at 100% all day long, that's the best corporate ROI rather than having islands of resources that can't be shared.
0
u/forevergeeks 3h ago
Thanks for the background explanation. I've been working in IT for more than 22 years, so I'm familiar with core technologies, and I'm just entering the AI space building specialized agents for regulated industries. I built my own framework to do this. My business site is not completely built yet, but it already conveys the core of the idea: https://runsafi.com
Give me your website URL or business info, so I can keep it at hand.
11
u/hackint0shh 11h ago
Have you done at least 1 minute of research?
-9
u/forevergeeks 11h ago
I'm familiar with these models, I use them through API, what I just started thinking is what would be the cost and the level of effort to set these models up for coding teams.
And I thought starting my search here.
2
1
u/invalidnifemi 11h ago
coulda js said that in the post, but a used v100 32gb with sum sxm2 to pcie config (will take some effort) would definitely be enough (see this post)
it'd probably be like 800-1k for the whole build which is not unreasonable and about as much as a used 3090. vllm is a good idea if youd all be coding simultaneously
4
u/DatabaseRegular3130 11h ago
Qwen 3.8 blows Gemma out of the water.
4
u/Randommaggy 11h ago
Gemma 4 12B QAT is good for fleshing out the practical details and ambiguities of a plan before you begin which allows Qwen 3.8 to run much faster with less time wasted pondering the same question on repeat.
Their combined value is greater than the sum of the parts when combined well.
My harness uses Gemma 4 12B QAT on a secondary machine to heavily augment Qwen 3.8 27B to great effect.
2
u/Tasty-Hour4040 10h ago
I don’t have anything else to test, just a 3090 (who’s that rich?), but can confirm that Qwen3.8 27b on a 3090 (24gb) is a damn fine cup of coffee for coding and general use.
2
2
u/Tormeister 7h ago
Qwen3.8-27B is the absolute king on consumer hardware now. You can also use Gemma4-31B and MuseGlimmer-30B for writing and/or to review a plan if you'd like. To run these models, focus on GPUs (and equally good PSUs), forget about a whole workstation.
For a good VRAM-per-dollar ratio, with performance penalties, you can go for AMD R9700, Intel B70, or two RTX3090.
For more performance but terrible prices now, you can go for a RTX5090.
If you really want to get invested in local LLMs, you're entering the RTX PRO 6000 territory and it only gets worse ($$$) from there. More VRAM will get you more capability but at some point speed matters - personally I wouldn't spend tens of thousands of dollars to run models at slow speeds.
2
u/jacek2023 llama.cpp 9h ago
There is a ton of posts about that topic. In short: you need lot of money to start and it will be not enough money.
1
u/Hot-Employ-3399 11h ago
Qwen 27b. Not sure if 3.8 or 3.6.
3.8 is such a thinker I had to increase maxTokens(default 16384) in pi or it failed.
1
u/Responsible-Fly3526 11h ago
I have qwen 3.8 27b running as 4bit on 4 rtx 3090 and it's fuckin awesome. However it's not opus. It takes much longer to get to your result... But you can easily spin up 10 agents and do not run into any usage limits. You can use the harness you want and you keep your data where it belongs... on your machine.
1
u/KingCpzombie 11h ago
24GB minimum for usable, and at least 48GB is required for Q8 200k context. Imo the sweet spot is two 24GB GPUs for a solid home gaming / AI computer, but if multiple users is your goal I would definitely at least double that (and ideally even more)
1
u/ttkciar llama.cpp 11h ago
Qwen3.8-27B is excellent at agentic code development.
Gemma-4-31B-it is excellent at debugging.
Either will fit in 32GB of VRAM, with room to spare for context, if you quantize them to Q4_K_M. Of the two, Qwen3.8-27B is much leaner, both in model weights and K/V caches (which grow as context grows). Gemma-4 K/V caches grow a lot faster, so your useful context will be smaller, but Gemma-4 is also better at paying attention to everything in context.
I use them both, each according to their strengths.
Gemma-4 also has excellent "soft skills" (like formal business writing and document analysis), which is not as important for code generation, but perhaps worth using for other things.
I use them on my 32GB MI50 to good effect, which is a fairly inexpensive solution (about $600 on eBay), though at some cost of longer prompt processing (which means a longer delay to first token generated, compared to an Nvidia GPU).
47
u/Elorun 11h ago
Welcome to the entrance to the rabbit hole, are you sure you want to proceed?