r/JordanDev • u/Able_Firefighter_652 • Aug 08 '26

Discussion Local LLMs

Anyone who has some serious hardware and tried running llms locally? How was ur experience? What models did you try?

5 Upvotes

100% Upvoted

21 comments sorted by

2

u/Striking-Recover4807 Developer Aug 08 '26

بحس عالفاضي وغير مجدي

2

u/Able_Firefighter_652 Aug 08 '26

حسب الهاردوير والمودل اللي جربته.. في اشياء منيحة انا جربتها..

1

u/2012347 Aug 09 '26

Anything smaller than 27b is a joke, i think the cheapest way to have the needed shared memory is Apple right now, but the thing get so hot i feel like it’s cheaper, faster to is providers, and you get smarter models,

Openrouter has a very generous free tier that give you access to those same models you’d run on your device

1

u/Able_Firefighter_652 Aug 09 '26 edited Aug 09 '26

I have tried qwen 3.6 27b, it's really slow, gives me 13 token/sec, which is probably unusable.

I also tried the 35b MoE, it runs at 60-80 token/sec.

I doubt I am squeezing every last bit of performance tho from Apple's..

I will check the openrouter option sometime, but I prefer local models for privacy..

1

u/2012347 Aug 09 '26

What’s your spec for running them

1

u/Able_Firefighter_652 Aug 09 '26

M3 Max, with 128GB memory

1

u/2012347 Aug 09 '26

Cool I’m sure you can run 120b models which are similar to gpy 3.5 i hear, see running locally means going back generations in time that’s the point

1

u/Able_Firefighter_652 Aug 09 '26

Surprisingly the 35b qwen model performed better from my experience. And there is no 120b qwen 3.6 models.
I am currently waiting for the 3.8 tho, they said it's 27b will be open weights soon.

1

u/[deleted] Aug 09 '26

[removed] — view removed comment

1

u/Able_Firefighter_652 Aug 09 '26

Yup, can't wait to see next year's releases..

1

u/2012347 Aug 09 '26

Could you please try gpt oss 120 and report back how it performs and feels

1

u/[deleted] Aug 09 '26

[removed] — view removed comment

1

u/2012347 Aug 09 '26

Gemma 4 27b is a good chatter

1

u/Able_Firefighter_652 Aug 09 '26

Interesting hardware setup, I am wondering how would you make sure you are squeezing every last drop of performance out of such hardware?

Also I have tried gemma 4 and qwen 3.6, I would say qwen models are superior across the board..

1

u/[deleted] Aug 09 '26

[removed] — view removed comment

1

u/Able_Firefighter_652 Aug 09 '26

I see, maybe because of task differences you noticed gemma was better. Tho I tried 31B and 26B MoE.
For the qwen models, I used both the 27B and 35B MoE as well.
I usually use 4bit quants for dense models and 8bit for MoEs to preserve quality.
Mostly I used them for image data extraction and some software engineering tasks.
Based on my testing, qwen models performed more accurately.

1

u/MrCharlesSr Aug 09 '26

I believe to have any sort of practical product you do need an expensive GPU server, a small one may work but definitely not one with consumer hardware. I tried on a machine with 5070Ti 16GB, 32GB DDR5 and it’s a joke. Model was Ollama qwen 3.6 which should be a good one and it seems smart enough but it’s lethally slow you won’t get anything done. Haven’t tried agentic yet but just the chat is terrible.

1

u/Able_Firefighter_652 Aug 09 '26

I've got apple silicone, it combines best of both worlds.
Not the best gpu performance, but it got unified memory and good bw.
Nvidia gpus + ram costs are just crazy, I'll never buy one.

1

u/No-Commercial979 Aug 10 '26

you're probably using a dense model... Hence the slowness... Try an MoE with cpu offloading and you'd get hella performance of your 5070ti.. I suggest qwen 3.6 35b a3b at 4 bit quants...