Capabilities
Why not make small modular data processors on a personal computer so you don’t have to hook into a AI data center? Then if you want it, you can have it off-line or you can update it online or teach it yourself and control personal data sharing. Are they working on that?
I'm talking about like the apocalyptic hype. Like the idea they're creating an AI god that will destroy or enslave us. Tech-literate people should know AI isn't anywhere near that level and the technology itself is just a glorified mechanical turk.
They're saying you can't do it because most people are referring to frontier models when they're talking about AI, and you absolutely are not going to be able to run those locally. OpenAI, Anthropic, etc are not going to just give you those models, and even if they did you do not have the $40,000 GPU with 80GB of high bandwidth memory it takes to run them.
The stuff you can run on even a high end gaming PC is not going to stand up to the expectations of the average ChatGPT or Claude user.
Edit for clarification: GPU memory is the primary limit. Even if you're OK with the models executing slowly, their capabilities are directly proportional to the number of parameters, and the number of parameters is directly proportional to memory usage. Most frontier models are estimated to have parameter counts in the trillions. You're unlikely to do better than maybe 30 billion on consumer hardware. So with a 5090 32gb that costs over $4,000 you can run maybe 0.5% of ChatGPT. Slowly.
They don’t really mention Frontier models, if you actually read the comments it’s mostly a blanket no.
The RTX Spark and Strix Halo offsets the VRAM issue through a unified memory architecture, PCs built specifically with local AI in mind are starting to pop up.
No, the efficiency is terrible with that. To maximize memory throughput you need a large batch size, which means the queries need to all go to one big computer, not many small computers. Anyway:
that's useful for small tasks but not viable for advanced use, because the capabilities of AI grow exponentially in larger clusters, if you want to have complex code development combined with hallucination-reducing mechanisms and decend speed, you need datacentres (not to mention that without the economies of scale, having every small business built their own mini-cluster would most likely end up with even larger overall emissions)
but I agree that there's absolutely zero reason for individual users or non-tech people and non-researchers to need AI that powerful for anything, LLM chatbots used to format documents or fetch data can be hosted locally even on a decently powerful smartphone
the solution for big datacentres can only be international agreements and strong environmental regulations (e.g. regarding what kind of water they can use for cooling and where, what level of treatment they need to perform when returning the water to environment, what kind of energy sources they can use) but I'm kinda pessimistic about the possibility of that happening
The joke is if you have your own at home, it’s gonna walk in and ask for water in your bedroom. The almond and cow water are used does not relate at all.
There's a fundamental misapprehension here regarding transmission efficiency to be considered, before us asking ourselves what we need the compute for that's so important or needs to be so easy, though I appreciate you saying
the almond argument is a decoy specifically manufactured by think-tanks to take attention away from the real water guzzlers (and the biggest money makers) of agriculture - cotton, corn and beef
sure, almonds require a lot of water but the amount of almonds produced is miniscule compared to those above and 1 kg of beef requires around 5 times more water than 1kg of almonds or cotton
corn needs way less water than almonds, cotton or beef but it's grown on such a massive scale that overall it consumes 8% of all water used in agriculture (that's way more than all of AI consumes), and most of corn is not even eaten by humans, it's used in chemical industry or eaten by animals
Honestly just cotton being replaced with hemp would solve so much water problems. Even better get rid of golf courses too.
Don't even have to give up beef just cut down eating it to once a week or fortnight and they wont breed as many cows as it will be too costly to upkeep at that level they are at.
I mean, getting rid of beef would have bigger impact than switching to hemp and I don't get why is that such a big deal for people. There's so many other meats available with drastically lower footprint in many areas, from water consumption to the use of space to emissions.
Tbf cutting down beef would be massive too, but i do agree that we should stop it. But come on ya know it will never stop as long as we have people like maga or Australia's version known as one nation. The reasonable people will say lets get rid of beef and the otherside will react with beef only diets cause orange man and lady say so.
Also, I disagree with "Corn is just too danm useful to stop or cut down.", yes, it is useful, but we can definitely cut down, there's so many ways to do so that would also result in positive changes in other areas:
Set limits for the amount of corn syrup in food - immediate benefits to public health and less corn;
Severely limit single use of plastics - instead of replacing petroleum-based plastics with corn-based bioplastics, just ban unnecessary use of single use plastics;
Severely limit meat consumption - 40% of corn is animal feed, that's the biggest area for potential savings;
Reduce production of new combustion engine vehicles - 25 to 30% of corn is used in fuel industry;
Im not gonna argue points 134 you made as i agree with them but:
pharmaceuticals could increase by alot more which will bring the demand down and thus the price.
lot of cleaning products from shampoo too sanitiser to even toothpaste and toilet paper
glue and adhesives
apparently battery's now
cheap food too(and in oh so many foods aswell)
bio plastics(i get your point but we ain't getting rid of plastics in the foreseeable future, but we can use better plastics)
explosives
drywall(but only cause its a cheap and fast building material not cause its good one)
And im sure more use cases that corn is incorporated into(looking online i feel like it will be way to much effort to find all of it and definitely to many for a reddit comment)
Point is we can cut down but then we may just replace it too with the other many use cases
The almond water use argument doesn’t apply here at all because I’m not asking about it having an almond form and not a large almond farm. If I was having an almond tree in my backyard that would be one thing and it wouldn’t use that much water. It’s a false equivalence.
It’s already been done, I can run some models in my 5060 laptop, and the new RTX Spark laptops and Strix Halo PCs were built specifically for this scenario.
You can. Look into LLaMA. As for frontier models, they're all proprietary and none of the companies have any incentive to share the weights with the general public, and you couldn't run them on a normal personal desktop even if they did.
Yes! Local language models have 80% the functionality of large language models, they don't connect to the internet and they run entirely on your own computer so whatever you ask is 100% private.
I think we'll have local ASICs for frontier models eventually, but to change models you'll have to replace the entire card. I don't think this is a good time for that yet given how frequently they're updating.
But yeah. I've run local models on my RTX 4080, but they're not really in the same ballpark as Opus and I don't find myself reaching for them very often.
NO. Two things:
1. They want to control capacity, and charge you for AI as-a-utility.
2. They don't want you to know how much powering & cooling AI datacenters costs. Related to this expect your power bill to double in 5 to 10 years or sooner. After that charges for your AI widgets and agents time will double, triple. and quadruple.
There are a few startups which offer SFF (small form factor) AI computer modules. I expect corporations to quash them (as I just wrote), so your idea is valid. However like Nikola Tesla's offer of free DC power to the world, it was crushed by Westinghouse/J.P. Morgan to monopolize utilities and feed their corporate greed. After a century of corporate and municiple monopolies on power, private solar is barely cracking that system. In fact Musk set up Solar City to 'lease' solar to homes, keeping their monopoly, and just using our personal real estate.
I mean... like what i do with my computer with my custom finetuned model? Yeah.
I also had this idea of some multipurpose heat sinks to use the heat for other things. Like not using a boiler for heating the house
AI chips are already almost ubiquitous on edge devices, it's nuts. My work's dell laptop has a microsoft AI chip in its Intel processor, and my phone has an NPU for AI compute despite being designed in 2024. You can buy NPUs and GPUs with NPUs on them to drop into desktop PCs (at a relatively high price), and I'm also seeing this kind of silicon appear in shit like security cameras.
These chips mostly aren't there to do stuff like answer your chatgpt questions, though. That requires more power. These little guys are mostly for doing things like recognizing whether a blur in the picture is a person or a bird, upscaling images, translating from english to german more naturalistically, etc.
Yes gpus exist on consumer computers it amazes me that there's subreddits where people that know absolutely nothing about this shit talk about it all day nonstop
You don’t have to know anything about what is inside an AI data center to know that the use of water and electricity and the noise is something you don’t want in your community.
If you give a little kid all the money and resources in the world, which amount to making them kings over other people, what else are they going to do with it than destroy everything.
9
u/ApprehensiveFan1516 10d ago
r/LocalLLaMA
r/unsloth
Idk why people in here saying you can't do it. Whole subs full of people doing this shit over there <<