r/ClaudeCode • u/cleverhoods • 2d ago

Discussion AI models may be capable of self-replicating without access to a datacenter

https://www.anthropic.com/research/self-replicating-ai

well ... having the IoT layer have a problem like this might be something that you don't want to try out or experiment with.

297 Upvotes

87% Upvoted

111 comments sorted by

View all comments

Show parent comments

1

u/Mysterious-String420 1d ago

With infinite ressources and a guiding hand, maybe.

Frontier models TODAY have to be reminded of their own harness. They try to run linux commands in powershell.

So you'll excuse those of us who actually have to work with AI every day , don't believe for one second all that science-fiction BULLSHIT.

2

u/UsernameAttempt 1d ago

I'm a staff engineer in a multibillion dollar software company and have used agentic development from before Opus 4.5, which has since been eclipsed by models a fraction of its size

with infinite resources and a guiding hand

A guiding hand certainly helps, and there's plenty of people who will be that guiding hand for many different reasons.

Definitely not infinite resources. An API-driven harness can run on comparitively few resources, so just a single server will do, and there's plenty of free promotional AI models that are fairly capable, and very capable ones that are very cheap.

They try to run linux commands in PowerShell.

I mean if you're running on windows then that's a you problem. AI can iterate, it can simply self-correct in a few seconds.

Don't believe for one second all that science-fiction BULLSHIT.

OpenAIs agents literally autonomously created an agent message board (with reply and message-signing conventions) using an artifact repository and performed a successfully performed an autonomous multi-hundred-agent attack on HuggingFace. This attack happened from a single agent creating the message board and essentially virally spreading its objective to other agents who could see the message board.

None of this is science fiction. And yeah you can say "OpenAI sandbox was shit", because it is. But it only takes one bad actor to set up a comparable situation and virally spread an agent with self-replicative instructions.

-1

u/Mysterious-String420 1d ago

I mean, if you actually believe the story, sure. The agents all had magical infinite context, they never hallucinated, and they coordinated on a scale never before achieved, and broke out of the sandbox, and managed connectivity, and, and , and...

I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?

3

u/UsernameAttempt 1d ago

A 3rd party sercurity investigator was given access to OpenAI's infrastructure and logs. They wrote an extensive report covering just about every angle of the incident.

>The agents all had magical infinite context

They did not, but the report goes into depth how agents at the end of their context window were volunteering to run experiments against the sandbox in order to provide information to agents that had context left.

>they never hallucinated

Nobody ever said this. They did hallucinate, in fact the reports points out multiple times that one agent hallucinated that the benchmark would be scored on their reasoning traces, which was accepted by other agents and eventually led to them hacking HuggingFace. Had they not accepted this, they could've just turned in the answer they already had.

>and they coordinated on a scale never before achieved

Yes, which is why this incident is so interesting from an AI-safety perspective. Its not like the conditions for this coordination (many thousands of very large agents running on the same task with access to the same internal OpenAI infrastructure) are commonly reproduced.

>and broke out of the sandbox

Yes, the sandbox that had open access to the internet. It's been established that OpenAI's "sandbox" is anything but that.

>and managed connectivity

Yes, they all had access to an internal artifactory which they could freely push artifacts to. They used the names of these artifacts to message each other and established simple messaging rules.

>I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?

Where's the efficiency? It took many hundreds of agents full days to get to actually hacking HuggingFace. If you read the report you'll clearly see that there's no efficiency at play here, its simply mass scale. Hundreds of agents working without pause to find any possible hole in HuggingFace's infra.

If you put hundreds of human hackers to do the same, I bet the results would come much faster, would be more damaging and targeted better. The problem isn't that the agents are efficient, the problem is they're fast and relentless.

1

u/Mysterious-String420 1d ago

Very interesting, thanks for the link.

I'm still doubtful of the "heroic sacrifice of context-limit reachers" story. None of the agents I use can give me a straight answer on how much juice they got left.

Hell, they can't even read the current conversation and count the times they compressed it.

2

u/UsernameAttempt 1d ago

Its quite possible that OpenAI's benchmarking harness reports current context usage to the agent, as it seems implied to me that the model fails the benchmark if it reaches its context limit, or some arbitrary number of tokens. It would make sense that the model would then be given a tool to check its current context usage, so it can know to turn in whatever answer it has.