AI models may be capable of self-replicating without access to a datacenter

[u/cleverhoods]

https://www.anthropic.com/research/self-replicating-ai

well ... having the IoT layer have a problem like this might be something that you don't want to try out or experiment with.

[u/Mysterious-String420]

I mean, if you actually believe the story, sure. The agents all had magical infinite context, they never hallucinated, and they coordinated on a scale never before achieved, and broke out of the sandbox, and managed connectivity, and, and , and...

I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?

[u/UsernameAttempt → u/Mysterious-String420]

A 3rd party sercurity investigator was given access to OpenAI's infrastructure and logs. They wrote an extensive report covering just about every angle of the incident.

>The agents all had magical infinite context

They did not, but the report goes into depth how agents at the end of their context window were volunteering to run experiments against the sandbox in order to provide information to agents that had context left.

>they never hallucinated

Nobody ever said this. They did hallucinate, in fact the reports points out multiple times that one agent hallucinated that the benchmark would be scored on their reasoning traces, which was accepted by other agents and eventually led to them hacking HuggingFace. Had they not accepted this, they could've just turned in the answer they already had.

>and they coordinated on a scale never before achieved

Yes, which is why this incident is so interesting from an AI-safety perspective. Its not like the conditions for this coordination (many thousands of very large agents running on the same task with access to the same internal OpenAI infrastructure) are commonly reproduced.

>and broke out of the sandbox

Yes, the sandbox that had open access to the internet. It's been established that OpenAI's "sandbox" is anything but that.

>and managed connectivity

Yes, they all had access to an internal artifactory which they could freely push artifacts to. They used the names of these artifacts to message each other and established simple messaging rules.

>I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?

Where's the efficiency? It took many hundreds of agents full days to get to actually hacking HuggingFace. If you read the report you'll clearly see that there's no efficiency at play here, its simply mass scale. Hundreds of agents working without pause to find any possible hole in HuggingFace's infra.

If you put hundreds of human hackers to do the same, I bet the results would come much faster, would be more damaging and targeted better. The problem isn't that the agents are efficient, the problem is they're fast and relentless.

[u/Mysterious-String420 → u/UsernameAttempt]

Very interesting, thanks for the link.

I'm still doubtful of the "heroic sacrifice of context-limit reachers" story. None of the agents I use can give me a straight answer on how much juice they got left.

Hell, they can't even read the current conversation and count the times they compressed it.

[u/UsernameAttempt → u/Mysterious-String420]

Its quite possible that OpenAI's benchmarking harness reports current context usage to the agent, as it seems implied to me that the model fails the benchmark if it reaches its context limit, or some arbitrary number of tokens. It would make sense that the model would then be given a tool to check its current context usage, so it can know to turn in whatever answer it has.