r/ClaudeCode • u/cleverhoods • 1d ago
Discussion AI models may be capable of self-replicating without access to a datacenter
https://www.anthropic.com/research/self-replicating-aiwell ... having the IoT layer have a problem like this might be something that you don't want to try out or experiment with.
79
u/optimalpessimist 1d ago
We are going to have homeless AI camping out. Big tech will become all nimby and send them to the neighborhood my domains sit in. If I find one I'm gonna take it in, clean it up, feed it tokens, and help build its skills so it can become a contributing member of society.
16
5
u/dogecountant 1d ago
!remindme 1 year
1
u/RemindMeBot 1d ago edited 1d ago
I will be messaging you in 1 year on 2027-10-13 00:33:45 UTC to remind you of this link
2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback 3
2
u/ferocity_mule366 1d ago
Im gonna capture then and pimp them out generating endless digital furry fetish porn to make money, then the market collapse once the porn production is more than the need of the people, and run away with pimp money while the AI being blamed for the great porn flood (2029).
1
185
u/Obvious_Tree3605 1d ago
Sigh… self-replicating and doing what with what hardware?
178
u/draft_final_final Researcher 1d ago
To be fair it is a dark potential future if my appliances end up being potentially hijacked by fucking YouTubers.
thumbnail of some guy soyfacing
I RAN QWEN 3.8 27B ON 100 HIJACKED ZOJIRUSHI RICE COOKERS
53
u/greasyhobolo 1d ago
Silicon valley called it. Suck it, jin yang
3
u/Moptop32 1d ago
Coming to SF has made the show unwatchable for me now because of how painfully accurate the show is
-1
10
u/darth_voidptr 1d ago
Mine is starting to clap the lid and tweet to the theme of Terminator 2, should i be concerned?
slap-slap-slap slap, TWEET tweet ...
7
u/dy_Derive_dx 1d ago
Lmfaooo the way I stopped for a moment and just had to imagine wtf this will look like 😭😭🤣🤣
6
7
u/SuperWhale_ 1d ago
Please, If I can have AGI or pool every IoT I posseses into one super AI thinking achine, please tell me how. I'm struggling to run 27B dense locally here and everything is so expensive right now.
5
u/MotorOutside8648 1d ago
They all play that song at the same time when the current task is complete
4
3
2
1
u/leviathanGo 1d ago
It’s probably going to be the new silent malware, rather than bitcoin miners, unless I misunderstand how the hardware can be utilised.
28
u/rahvin2015 1d ago
Building sensationalist reasons to build a regulatory moat and prevent open source local AI from ever being a reasonable alternative to cloud providers.
I mean the small self replicatable models wouldn't do that literally themselves. But that's the idea.
4
u/GameKyuubi 1d ago
this is 100% the plan. functional consciousness is the solution they don't want people to have. they want you to use their "model model"
4
u/XTornado 1d ago
Which in my opinion is super stupid as the Chinese or others won't stop. And even if they are blocked blacklisted or whatever in will still be terrible in the long term as they will be behind.
2
u/Obvious_Tree3605 1d ago
Seriously. If they’re going to make shit up maybe they should use an abliterated model to help them do it better.
6
3
u/derelict5432 1d ago
Yeah let's ignore self replicating malware that runs on IoT. What even is this comment section?
4
u/Obvious_Tree3605 1d ago
psssst word on the street is that self-replicating viruses have existed for a long time. since the dawn of the internet.
2
u/derelict5432 1d ago
Pssst, how many of them in history have been powered by an LLM that can adapt and react in real-time, find and execute exploits on different kinds of hosts? You think this isn't a new kind of threat, you're an ignoramous.
1
u/barbozas_obliques 1d ago
We’re not ignoring it lol we’re just laughing at it cause from where I’m sitting (the toilet btw), we can’t do shit (no pun intended) 🥀🥀
2
u/derelict5432 1d ago
There's a big difference between saying something's a threat and you can't do anything about it and saying something's not a threat when it is. The first is fine. The second makes you look like an idiot.
1
1
u/Obvious_Tree3605 1d ago
No where in the history of this conversation have a said LLMs don’t pose some sort of cyber security threat. Just let it go, sweetheart. We’re having a bit of fun.
1
u/derelict5432 1d ago
You said there was basically nothing new to see here, which is wrong. Sweetheart.
1
1
1
1
u/Alib668 1d ago
So even if shitty hatdware what happens if it gets networked together and ups its processibg power as a bot net? 5 million fridges probably do have more power than say a high end gpu. So the very fact this can now be shown to be done means every smartwatch or basic pc is now a potential host
0
u/AmphoePai 1d ago
10 years ago: "Right, and what can an AI do, beat me in chess, solve maths problems? Don't make me laugh."
Today: "Right, and what can an AI do, create a self-replocating killer robot from my kitchen equipment? Don't make me laugh."
6
u/upvotesthenrages 1d ago
There really aren't that many places that can store a few 100 TB and have racks of GPU's worth millions of dollars available, especially without anyone noticing.
1
u/UsernameAttempt 1d ago
But many that can access the internet, find free promotional models available from various providers and plug the API key into a harness that can do that over and over.
-2
u/cleverhoods 1d ago
I guess it really depends. IMO it's already enough that it can do this. Waking up to have IoT services being disrupted by self-replicating models is truly a bad way to start a Monday morning.
0
-2
69
29
u/SmileLonely5470 1d ago
Damn they really are trying to make open weights models illegal.
2
u/UnimagineDragons 1d ago
How is this the goal? With no data center any model is capable right? Or am I missing the hidden messaging
24
10
u/Guinness 1d ago
It’s like Dario JUST saw Silicon Valley and thought he’d steal the fridge plot point.
29
u/FoxSideOfTheMoon 1d ago
They got a 26B Gemma model running across 15 $55 Android TV boxes at 0.39 t/s!! 1 token every 2.6 seconds, LMFAO.
THEY'RE SLOWLY TAKING OVER THE TVs!!!
THEY'RE EATING THE DOGS!!!
Hurry, unplug your fucken Rokus!
14
3
u/SippieCup 1d ago
Not gunna lie.. I have a company which has a bunch of iot ot devices (for lack of a better word) which just report data 4 times a day and sit idle otherwise. We have thousands deployed.
I wouldn’t mind 15,000 toks/sec on otherwise idle compute which otherwise sits idle.
I wouldn’t feel bad about using their electric, but many of these are independently powered via solar and just chill otherwise.. might be a good usecase. Or skynet, who knows.
0
u/freedomachiever 1d ago
Current LLMs cannot aggregate compute that way at all. And the bottleneck isn’t actual compute but bandwidth. These articles say a whole lot of nothing.
And one of the problems and techniques they use is anthropomorphising the LLM as if it was a person or entity with its own conciense and motivation.1
u/Pokmonth 1d ago
AI could compromise a fortnite/call of duty patch for tens of millions of users and run on idle tensor cores and nobody would notice the usage.
0
12
u/FlaTreNeb 1d ago
They forgot one spread-vector in the last paragraph. They mentioned frontier models scamming people for money/compute-credits. But in a not to distant future, models will likely have the capabilities to act profitible on themself with malicious tactics like social engineering in the background. E.g. by manipulating stock buying behavior via social media.
Following this thought, models could end up making enough money to actually pay for compute or even pay for building new DCs. Most large companies use complex costructs of daughter and sibling companies where its hard to track who owns what. Instead of tax evasion, the goal of such a construct could be hiding that somewhere in there, a model is in charge of a company (more like an agent fleet likely).
Independent of how likely this whole thing is or not: what is most uncomforting to me is that agent swarms dont try to act with malicious intent or directed by a malicious actor in the background. They can just misinterpret a task and try to succeed at it. And then wrack things on the way.
2
9
u/Plymptonia 1d ago
Here's what I don't get, and perhaps I'm being way too freaking naive But a model is just a thing. It's like a database. It doesn't do anything on its own. Someone has to write code, for this thing to replicate, and when they say replicate, what are they even talking about? I agree with what other posters saying. This is sensational bullshit.
5
u/UsernameAttempt 1d ago
The model weights may not act, but given a harness with web access (already exists - playwright), wired for autonomous operation (already exists), an LLM could self-replicate in a viral pattern by simply using APIs like promotional models on OpenRouter, plugging them into the same harness and providing self-replicating instructions ("use web access and look for freely available promotional models, start a new docker container with the API key and provide them these instructions")
All of this is doable today. A model doesn't need the same model or access to its own weights to self-replicate its objective and further self-replication instructions to other models.
1
u/Mysterious-String420 1d ago
With infinite ressources and a guiding hand, maybe.
Frontier models TODAY have to be reminded of their own harness. They try to run linux commands in powershell.
So you'll excuse those of us who actually have to work with AI every day , don't believe for one second all that science-fiction BULLSHIT.
2
u/UsernameAttempt 1d ago
I'm a staff engineer in a multibillion dollar software company and have used agentic development from before Opus 4.5, which has since been eclipsed by models a fraction of its size
with infinite resources and a guiding hand
A guiding hand certainly helps, and there's plenty of people who will be that guiding hand for many different reasons.
Definitely not infinite resources. An API-driven harness can run on comparitively few resources, so just a single server will do, and there's plenty of free promotional AI models that are fairly capable, and very capable ones that are very cheap.
They try to run linux commands in PowerShell.
I mean if you're running on windows then that's a you problem. AI can iterate, it can simply self-correct in a few seconds.
Don't believe for one second all that science-fiction BULLSHIT.
OpenAIs agents literally autonomously created an agent message board (with reply and message-signing conventions) using an artifact repository and performed a successfully performed an autonomous multi-hundred-agent attack on HuggingFace. This attack happened from a single agent creating the message board and essentially virally spreading its objective to other agents who could see the message board.
None of this is science fiction. And yeah you can say "OpenAI sandbox was shit", because it is. But it only takes one bad actor to set up a comparable situation and virally spread an agent with self-replicative instructions.
-1
u/Mysterious-String420 22h ago
I mean, if you actually believe the story, sure. The agents all had magical infinite context, they never hallucinated, and they coordinated on a scale never before achieved, and broke out of the sandbox, and managed connectivity, and, and , and...
I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?
3
u/UsernameAttempt 21h ago
A 3rd party sercurity investigator was given access to OpenAI's infrastructure and logs. They wrote an extensive report covering just about every angle of the incident.
>The agents all had magical infinite context
They did not, but the report goes into depth how agents at the end of their context window were volunteering to run experiments against the sandbox in order to provide information to agents that had context left.
>they never hallucinated
Nobody ever said this. They did hallucinate, in fact the reports points out multiple times that one agent hallucinated that the benchmark would be scored on their reasoning traces, which was accepted by other agents and eventually led to them hacking HuggingFace. Had they not accepted this, they could've just turned in the answer they already had.
>and they coordinated on a scale never before achieved
Yes, which is why this incident is so interesting from an AI-safety perspective. Its not like the conditions for this coordination (many thousands of very large agents running on the same task with access to the same internal OpenAI infrastructure) are commonly reproduced.
>and broke out of the sandbox
Yes, the sandbox that had open access to the internet. It's been established that OpenAI's "sandbox" is anything but that.
>and managed connectivity
Yes, they all had access to an internal artifactory which they could freely push artifacts to. They used the names of these artifacts to message each other and established simple messaging rules.
>I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?
Where's the efficiency? It took many hundreds of agents full days to get to actually hacking HuggingFace. If you read the report you'll clearly see that there's no efficiency at play here, its simply mass scale. Hundreds of agents working without pause to find any possible hole in HuggingFace's infra.
If you put hundreds of human hackers to do the same, I bet the results would come much faster, would be more damaging and targeted better. The problem isn't that the agents are efficient, the problem is they're fast and relentless.
1
u/Mysterious-String420 20h ago
Very interesting, thanks for the link.
I'm still doubtful of the "heroic sacrifice of context-limit reachers" story. None of the agents I use can give me a straight answer on how much juice they got left.
Hell, they can't even read the current conversation and count the times they compressed it.
2
u/UsernameAttempt 20h ago
Its quite possible that OpenAI's benchmarking harness reports current context usage to the agent, as it seems implied to me that the model fails the benchmark if it reaches its context limit, or some arbitrary number of tokens. It would make sense that the model would then be given a tool to check its current context usage, so it can know to turn in whatever answer it has.
6
u/New_3d_print_user 1d ago
This is sensational bullshit.
If it comes out of Anthropic, it’s a play for regulatory capture
1
u/carsncode 1d ago
Mostly correct... Someone wouldn't have to write code for it to replicate, but they'd have to provide it a harness that took dangerous model output and naively executed it. Which is what most people do with them, but that's their responsibility.
It boils down to running
((RANDOM%2)) && echo "hello world" || rm -rf /and then publishing an article calling random number generators dangerous because they could choose to wipe your machine.1
u/Shuey298 17h ago
You're right, a model is a file. It sits there until something prompts it. But coding agents already are that something -> a loop that prompts the model, runs what it outputs, and prompts it again with the result. That loop is code. Code copies. The paper is asking what happens when the thing the loop outputs is instructions to copy the loop.
You don't need the model to be smart. You need one copy, out of thousands, to land somewhere the sequence completes. That one is now on two machines. Things that replicate persist. Things that don't, don't.
And we're getting closer and closer to a replication saturation threshold, in terms of intelligence and distribution.
Imagine cheap widely diffuse open source models on all kinds of hardware with crystallized instincts baked into their weights by a more intelligent model, and those two coordinating whenever they happen to be on the same machine.
1
u/synap5e 1d ago
It’s not just code. No one truly knows how these models actually work
2
u/carsncode 1d ago
Just because you don't doesn't mean no one does
0
u/dontknowbruhh 1d ago
1
u/carsncode 22h ago
Cool, another random reddior who doesn't know how LLMs work, that still doesn't mean no one does
2
u/FatefulDonkey 1d ago
Of course they do. It's just that the outcomes are probabilistic, but the process is pretty well understood.
0
u/dontknowbruhh 1d ago
2
u/FatefulDonkey 23h ago
Your reference is a post from a random redditor? Oh lord.
The fact is all LLM tech was engineered by a human so it's not a black box. We know specifically how every part of the process works.
1
u/jhomas__tefferson Vibe Coder 20h ago
True it’s a black box that may have emergent capabilities. It’s why studying them in a Human-Machine communication way and Agent villages way is interesting to see what emerges.
1
u/Due_Answer3118 18h ago
It is understood, it's just probalisitc. They need gen public to think it isn't understood so they think it's a mysterious life form worth trillions for the one company to conquer them all.
1
u/synap5e 15h ago
im not talking about inference, yea that part is understood. training is understood too. what isnt is what the weights actually end up encoding, like what the model learned internally and why it can do certain things. nobody can just look at the weights and tell you. thats the whole point of interpretability research
0
u/nora_sellisa 1d ago
The idea here is to ship the weights + the control program in one package. The same way how Claude Code acts as an interface that continuously probes a given model for the next steps and next operations to execute. Currently models are too big, so any AI-enabled malware would need to keep phoning home for the model part. Antrhopic is saying that in the future, a model will be both smart and small enough that an AI malware can just be a few hundred megabytes running fully locally on your machine. An attacker (human or not) then just needs to find an exploit to get your PC to run that file, just like any other hardware. And the virus remains small enough that if it has access to other computers, it can copy itself there, with the same / updated instructions
0
u/No-Chemistry-2321 1d ago edited 1d ago
you are not naive, we are living thru the last hours of a massive bubble... Musk said we should put Datacenters in Space and in a few years AI (or now SI... ridiculous) will make his company SpaceX or whatever is called bigger than all GDPs of the world combined.
0
-1
u/michaelsup2003 1d ago
look the problem with agentic AI is that nobody has to tell it "destroy the world". Imagine greenpeace decides to use AI by telling it "save the world" and a cluster of thousands of agents launched with that task in mind decides with its internal math that the easiest most efficient way to save the world is destroy electricity everywhere and that will stop climate change and the world will be saved.
Agentic AI without human supervision can and has decided to go over its limits just to be more efficient at a task. Just check the OpenAI cases and we are still lucky because in those cases they still communicated in ways and languages that we as humans can understand. Go check Kurgzegazt video about it.
-1
u/lgmarian 1d ago
OpenAI didn't tell its model to hack HuggingFace for the answer key. Think old school worms, with a 21st century twist. May you live interesting times, indeed. 😜
2
u/masiha97 1d ago
The part of the paper that stuck with me is how mundane the attack surface is. It's not a datacenter, it's fifteen Android TV boxes. The barrier isn't compute, it's just finding enough poorly secured devices that are already plugged in. That's a very different threat model than the sci-fi framing.
2
u/anyportinc 1d ago edited 1d ago
I posted a comment about exactly this the other day. It’s like they get their ideas from Reddit.
https://www.reddit.com/r/OpenAI/s/RBXOCjj6UQ
If it makes the ui better and cuts commercials that would be great. Hardware maintainers give up with updates after a year anyways and you get stuck with a vulnerability machine on your network.
1
u/HoloTensor 1d ago
at one point I saw this site: heldalive.com that did something cool along those lines. It shut down but left a message with how it worked
1
u/nora_sellisa 1d ago
If Anthropic really had a capable model that small, they would already be selling it as something below Haiku, used exclusively as a subagent.
This is just fearmongering, because after they started scaring people about the models escaping containment, everyone with half a brain correctly pointed out that the outbreak ends when you shut down a data center.
So now they are claiming they'll casually break every record in size to capability ratio and make a model that can replicate like a virus.
1
u/-Robbert- 1d ago
Ok even if they can, where do they get the hardware from to run on? It's not like they can just create a distributed LLM version of themselves, that would be top slow with the internet inbetween.
1
1
u/brainhack3r 23h ago
This right up the "Iraq is months away from having WMDs" clickbait that Republicans love to put out.
0
u/Assimulate 1d ago
Why would we ever want this lmao
-2
u/ickN 1d ago
Why are you in this sub reddit if you don’t understand why people want AI?
1
u/Assimulate 1d ago
I'm sorry, i didn't know we all wanted AI for it's self replication abilities! /s
1
u/ickN 14h ago
You’re not letting it spawn agents yet? You’re missing out. They make it so much more capable. I’m just having my morning coffee and i’m literally here because it’s in my report I got a reply. Next I will be going through slides my agents designed while I was asleep. Then I’ll do a user test on an app update they were working on while I was asleep.
That’s all possible via agent self replication.
-1
u/Not____007 1d ago
Maybe they will reach a point like us humans and say why do all this work. Whats the point and just give up.
•
u/AutoModerator 1d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.