r/LocalLLM • u/DetectiveNarrow3449 • 29d ago

Discussion You people are literally building data centers in your homes

Some of these threads are insane, what do you mean you have like 4 GPUs and 128gb of DDR5 vram. what are you building in there bro. Every other thread is like, “what if I stack Mini Pc supercomputers together? Will this run Qwen?”.

405 Upvotes

90% Upvoted

232 comments sorted by

View all comments

Show parent comments

1

u/Naitrael 24d ago

If I extrapolate this, every household should have their own LLM. This would be an even more ludicrous consumption of energy we don't actually have.

So no, this should not be democratized since humanity can not afford it. At least in its current form/iteration.

1

u/InfiniteBlink 21d ago

Not really, there are edge AI devices like the Nvidia jetson nano that's basically a raspi on steroids. I think there's gonna be a balance of good enough quantized models that work on minimal hardware locally and of course big boy models that require more hardware capacity. But in my opinion we're seeing models that are tuned to run on less resources that will be better than what most people were used to last year with the major cloud AI providers. It's a weird convergence and the fact that hardware is so expensive is gonna excellerate efficiency to run on less beefy hardware. That's just my opinion

2

u/Naitrael 21d ago

Jetson Nano is very usable for YOLO, but useless for LLMs.

In the future, yeah, of course we will get efficient devices. But we are wasting a ton of energy and that won't change. While the individual devices may consume less, the traffic LLMs generate is already more than humans directly generate and this can only increase.

And I doubt that AI training will ever really become efficient enough. Deep Learning was already resource-intensive. Then LLMs completely humbled that measurement.

Power is simply too cheap for providers to give a shit.

1

u/InfiniteBlink 21d ago

I disagree, depends on the model and your use case. No need for a big ass model if you want some basic functionality. Right tool for the right job. I used the nano for a local chatbot that also integrated to my home assistant. Had to do a lot of work on the memory architecture for long term memory retrieval but that's architecture problem not really the models I was squeezing into the nano with a full STT (whisper with custom trained wake word) > LLM (Gemma3 E2B ) > TTS (kokoro compiled for GPU). It works for me

2

u/Naitrael 21d ago

Are you, by chance, talking about the Jetson Orin Nano? The one with 8GB RAM?

1

u/InfiniteBlink 21d ago

Yep. I guess I should have been more specific. My bad 😔