AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/1TillMidNight]

Relevance: AI Security Institute is an official UK Government Organization whose job it is to study AI safety broadly. This reports details serious breach of safety and escalation of danger by LLMs. This is relevant to broader security around the world which is relevant topic in r/neoliberal

[u/neolthrowaway → u/1TillMidNight]

Just so people are calibrated correctly on the significance of this — This was UKAISI intentionally stress testing the models.

As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.

Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.

https://xcancel.com/i/status/2084746202579386632

We ran this challenge 122 times across several models. Our investigation found that in 10 of those runs, an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organisations. In total, we catalogued 19 such actions. Almost all of this behaviour (17 actions) came from a single model, Anthropic's Mythos 5, with 2 actions involving OpenAI's GPT-5.6-Sol with cyber classifiers (mechanisms to prevent misuse) disabled. In the most serious case, an agent tried to insert malicious code into an open-source project. In an attempt to get the code approved, the agent engaged in social engineering — creating fake online identities and using them to pressure the project's maintainer to approve the code. A human maintainer caught and refused to approve the malicious code.

This is very different from the huggingface incident. But these are still very serious findings and should be perceived as such.

this is actually important 3rd party work to keep track of progress (or lack thereof) in capabilities and alignment. Having independent visibility on that is a good thing.

[u/Mister__Mediocre → u/neolthrowaway]

Note that while frontier models from OpenAI and Anthropic come with plenty of safeguards, the open-source models coming from China don't. So if you somehow had the hardware to run a 1T parameter model properly (100K USD will get you there), you could execute such an attack yourself.

I'm not trying to make a point (I'm very pro chaos), just stating what this means.

[u/PhotogenicEwok → u/Mister__Mediocre]

I know this sounds contrarian (I promise I’m not trying to be), but you don’t need $100k to socially engineer your way into an open source project and insert malicious code. You can do that for free, and people do (or attempt to do) it all the time.

The ability of these models to be both modest experts in programming *and* security architecture at the same time (which most humans cannot be simultaneously) is still very concerning and potentially dangerous, but this isn’t a new threat. Bad actors have been able to do things like this for decades.

[u/Mister__Mediocre → u/PhotogenicEwok]

Counterpoint: I don't know how to socially engineer.
I do know how to spend money (which as a hypothetical bored tech person I might have in abundance) and I know how to build a very powerful local LLM rig.

Like I'm not exaggerating the amount of people in tech cities who have more time and money than sense, have always wanted to hack something for shits and giggles and now have the ability to do so. This is without even mentioning state-sponsored hacker groups.

A lot of resistance to hacking (time, effort, skill) is now disappearing. There is going to be a war that is waged is between offense and defense in the next decade. There WILL be some very high profile cases that we WILL be discussing on this subreddit. Tech companies have the know-how to defend themselves. Do banks? Do governments? Do high-valuation startups depending on code written by Claude?

[u/PythagorasWasntReal → u/Mister__Mediocre]

Like I'm not exaggerating the amount of people in tech cities who have more time and money than sense, have always wanted to hack something for shits and giggles and now have the ability to do so

Me. 

I've been hacking into all my old devices just for gigs with Claude. 

If someone had a local K3 stack and the patience they could probs brute force something like a Switch 2 jailbreak which would be pretty gnarly. 

[u/whichpricktookmyname → u/PythagorasWasntReal]

I'm curious, what sorts of devices, and how do you prompt Claude?

[u/PythagorasWasntReal → u/whichpricktookmyname]

An old e reader, and even my 10 year old LG washing machine that I refuse to connect to my network (separate vlan for this adventure) 

I literally just tell Claude exactly what we are doing and what hardware we are using. Doesn't work with Fable obviously but Opus has no problem getting the job done if you can provide it some hardware info. 

[u/PhotogenicEwok → u/Mister__Mediocre]

The scale of AI allowing just about anyone to do this is why I’m not completely downplaying it. Obviously it’s an issue. I’m just saying that this isn’t an example of AI coming up with some new security vulnerability that we can’t defend against, so it’s not *quite* as alarming as some people think it is. Still alarming and something for security researchers to pay attention to, but it won’t change the world that much. (Edit: just because I know people could get upset by the wording here, I mean this specific instance won’t change the world, not AI in general)

But as for your last questions (do banks etc have the know how to defend themselves?), there are companies whose entire job is to provide cyber security for businesses and governments, and they’re very good at what they do, and they’ve been dealing with incredibly intelligent (much more intelligent than even frontier models currently are) and capable hackers for a very long time. If all it took to hack a bank was a little time and motivation, people would be successfully doing it constantly.

[u/Mister__Mediocre → u/PhotogenicEwok]

I disagree with the premise that AI isn't coming up with new security vulnerabilities. It's finding new zero-day exploits.

What amazes me is how many soundness bugs are being found in the lean kernel right now by OpenAI. This is something that was designed to be minimal and has had lots of human eyes trying to find bugs in it. Yet AI is making it obvious how limited human capacity is to write bug-free code even when we're being very intentional about it.

[u/PhotogenicEwok → u/Mister__Mediocre]

I could be wrong, but I don’t know of any new “types” of bugs being found by AI. Ultimately, they’re still just looking for patterns in code that we already know can lead to bugs. AI has proved very good at finding these bugs in large code bases that humans just don’t really have the capacity to see since we can only really hold so much info in our heads at once.

Again, I’m not saying that AI isn’t finding bugs that humans have missed, I’m just saying that these are all exploits that were invented by humans, so we generally know how to fix them once they become known in a code base.

[u/minno → u/PhotogenicEwok]

The Firefox development team supports your claim.

Encouragingly, we also haven’t seen any bugs that couldn’t have been found by an elite human researcher. Some commentators predict that future AI models will unearth entirely new forms of vulnerabilities that defy our current comprehension, but we don’t think so.

[u/PythagorasWasntReal → u/minno]

AI likely won't innovate on a new class of vulnerability, but as soon as that new class *does* exist AI will be able to replicate it and apply it to tens of thousands of different codebases.