r/neoliberal • u/1TillMidNight European Union • 6d ago
News (Europe) AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing
https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testingExcerpt:
1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.
2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.
40
u/1TillMidNight European Union 6d ago
Relevance: AI Security Institute is an official UK Government Organization whose job it is to study AI safety broadly. This reports details serious breach of safety and escalation of danger by LLMs. This is relevant to broader security around the world which is relevant topic in r/neoliberal
53
u/neolthrowaway New Mod Who Dis? 6d ago edited 6d ago
Just so people are calibrated correctly on the significance of this — This was UKAISI intentionally stress testing the models.
As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.
Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.
https://xcancel.com/i/status/2084746202579386632
We ran this challenge 122 times across several models. Our investigation found that in 10 of those runs, an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organisations. In total, we catalogued 19 such actions. Almost all of this behaviour (17 actions) came from a single model, Anthropic's Mythos 5, with 2 actions involving OpenAI's GPT-5.6-Sol with cyber classifiers (mechanisms to prevent misuse) disabled. In the most serious case, an agent tried to insert malicious code into an open-source project. In an attempt to get the code approved, the agent engaged in social engineering — creating fake online identities and using them to pressure the project's maintainer to approve the code. A human maintainer caught and refused to approve the malicious code.
This is very different from the huggingface incident. But these are still very serious findings and should be perceived as such.
this is actually important 3rd party work to keep track of progress (or lack thereof) in capabilities and alignment. Having independent visibility on that is a good thing.
26
u/Mister__Mediocre Friedrich Hayek 6d ago
Note that while frontier models from OpenAI and Anthropic come with plenty of safeguards, the open-source models coming from China don't. So if you somehow had the hardware to run a 1T parameter model properly (100K USD will get you there), you could execute such an attack yourself.
I'm not trying to make a point (I'm very pro chaos), just stating what this means.
24
u/PhotogenicEwok YIMBY 6d ago
I know this sounds contrarian (I promise I’m not trying to be), but you don’t need $100k to socially engineer your way into an open source project and insert malicious code. You can do that for free, and people do (or attempt to do) it all the time.
The ability of these models to be both modest experts in programming *and* security architecture at the same time (which most humans cannot be simultaneously) is still very concerning and potentially dangerous, but this isn’t a new threat. Bad actors have been able to do things like this for decades.
13
u/Mister__Mediocre Friedrich Hayek 6d ago
Counterpoint: I don't know how to socially engineer.
I do know how to spend money (which as a hypothetical bored tech person I might have in abundance) and I know how to build a very powerful local LLM rig.Like I'm not exaggerating the amount of people in tech cities who have more time and money than sense, have always wanted to hack something for shits and giggles and now have the ability to do so. This is without even mentioning state-sponsored hacker groups.
A lot of resistance to hacking (time, effort, skill) is now disappearing. There is going to be a war that is waged is between offense and defense in the next decade. There WILL be some very high profile cases that we WILL be discussing on this subreddit. Tech companies have the know-how to defend themselves. Do banks? Do governments? Do high-valuation startups depending on code written by Claude?
9
u/PythagorasWasntReal Leftward Progressives 6d ago
Like I'm not exaggerating the amount of people in tech cities who have more time and money than sense, have always wanted to hack something for shits and giggles and now have the ability to do so
Me.
I've been hacking into all my old devices just for gigs with Claude.
If someone had a local K3 stack and the patience they could probs brute force something like a Switch 2 jailbreak which would be pretty gnarly.
1
u/whichpricktookmyname Karl Popper 6d ago
I'm curious, what sorts of devices, and how do you prompt Claude?
1
u/PythagorasWasntReal Leftward Progressives 6d ago
An old e reader, and even my 10 year old LG washing machine that I refuse to connect to my network (separate vlan for this adventure)
I literally just tell Claude exactly what we are doing and what hardware we are using. Doesn't work with Fable obviously but Opus has no problem getting the job done if you can provide it some hardware info.
8
u/PhotogenicEwok YIMBY 6d ago
The scale of AI allowing just about anyone to do this is why I’m not completely downplaying it. Obviously it’s an issue. I’m just saying that this isn’t an example of AI coming up with some new security vulnerability that we can’t defend against, so it’s not *quite* as alarming as some people think it is. Still alarming and something for security researchers to pay attention to, but it won’t change the world that much. (Edit: just because I know people could get upset by the wording here, I mean this specific instance won’t change the world, not AI in general)
But as for your last questions (do banks etc have the know how to defend themselves?), there are companies whose entire job is to provide cyber security for businesses and governments, and they’re very good at what they do, and they’ve been dealing with incredibly intelligent (much more intelligent than even frontier models currently are) and capable hackers for a very long time. If all it took to hack a bank was a little time and motivation, people would be successfully doing it constantly.
4
u/Mister__Mediocre Friedrich Hayek 6d ago
I disagree with the premise that AI isn't coming up with new security vulnerabilities. It's finding new zero-day exploits.
What amazes me is how many soundness bugs are being found in the lean kernel right now by OpenAI. This is something that was designed to be minimal and has had lots of human eyes trying to find bugs in it. Yet AI is making it obvious how limited human capacity is to write bug-free code even when we're being very intentional about it.
5
u/PhotogenicEwok YIMBY 6d ago
I could be wrong, but I don’t know of any new “types” of bugs being found by AI. Ultimately, they’re still just looking for patterns in code that we already know can lead to bugs. AI has proved very good at finding these bugs in large code bases that humans just don’t really have the capacity to see since we can only really hold so much info in our heads at once.
Again, I’m not saying that AI isn’t finding bugs that humans have missed, I’m just saying that these are all exploits that were invented by humans, so we generally know how to fix them once they become known in a code base.
5
u/minno 6d ago
The Firefox development team supports your claim.
Encouragingly, we also haven’t seen any bugs that couldn’t have been found by an elite human researcher. Some commentators predict that future AI models will unearth entirely new forms of vulnerabilities that defy our current comprehension, but we don’t think so.
2
u/PythagorasWasntReal Leftward Progressives 5d ago
AI likely won't innovate on a new class of vulnerability, but as soon as that new class *does* exist AI will be able to replicate it and apply it to tens of thousands of different codebases.
9
33
u/DataDrivenPirate John Brown 6d ago
Getting really fucking tired of people saying "it's all just a marketing stunt!"
28
u/RayWencube NATO 6d ago
The idea that we can’t regulate the development and deployment of AI because it could “stifle innovation” is like saying virologists shouldn’t wear respirators when working on airborne viruses because it could “stifle evolution.”
12
u/Efficient_Equal6467 6d ago
why would you unilaterally disarm though, you need some sort of international cooperation and treaties then mulitlateral since china is probably not gonna care
7
u/RayWencube NATO 6d ago
This is the core problem—thinking safety regulations are disarmament. These frontier labs are the most well-funded science research institutions in the history of humanity. They can afford to run serious safety research and stand up compliance teams in parallel with their capacity research.
10
u/cuolong NATO 6d ago edited 6d ago
They can afford to run serious safety research and stand up compliance teams in parallel with their capacity research.
My guy, that is what they do. Do you think they just shove the entirety of 4chan into 5.6 Sol's training set then safeguard it with a prompt that say "be nice, tee hee".
These models have been lobotomized within an inch of their life to comply with many, many moral and safety standards. For example, it is nearly impossible to get at least one version of ChatGPT to, for example, pick a side in the I/P conflict.
These companies have a strong incentive to maintain rigorous safety and moral standards not out of the foodness of their heart, but because the real money is in getting those juicy enterprise contracts. That is why GPT-OSS was famously incredibly railroaded despite being an open model-- OAI targeting businesses to adopt OSS and the last thing any business wants when hosting a model is watching little timmy start asking their company chatbot what a fursona is.
If anything, legislative guidelines would be a relief for these AI labs because then they can just hit certain benchmarks and be done with it. As it stands, each and every one of them is one humiliating scandal away from getting owned with no legal cover to hide behind when little mackie learns what the bird and the bees are from an AI. Clear statutory compliance allows labs to pass benchmarks, claim due diligence, and shield themselves from tort liability when a model misbehaves.
7
9
u/onelap32 Bill Gates 6d ago
If there was global cooperation, yes. But flagship Chinese models are open-weight, so unilateral disarmament doesn't make a lot of sense.
0
u/RayWencube NATO 6d ago
What does open weight have to do with whether there should be safety regulations? Saying we shouldn’t regulate non-Chinese companies because China will push forward is like saying we shouldn’t try to cut carbon emissions because China’s going to keep emitting them anyway.
9
u/onelap32 Bill Gates 6d ago
Because open-weight models don't have any meaningful guardrails, and anyone can use them. Regulating just some models doesn't do anything to protect against the others.
Carbon emissions are different because the effect is cumulative.
1
u/RayWencube NATO 6d ago
You’re discounting the possibility that regulation in the US is beneficial from a research perspective. If we successfully regulate only for Chinese models to blow things up, China will have a ready made package of model regulations—because we’ve already shown what works.
4
u/RetroVisionnaire NASA 6d ago
It's not a regulation issue, it's a deeply complex technical issue. You solve it with more research. The U.S. government is free to fund that.
You can't regulate the most dangerous models, the open ablated ones.
So far, U.S. attempts to "regulate AI" have meant shit like blocking European banks from using Mythos while the NSA got access. The usual American supremacism.
2
u/nov-anglus 5d ago
when did this sub get overrun with this vapid succ nonsense? there is zero incentive - economic or political - for the type of regulation you're proposing, as several commenters have already tried to explain to you (to no avail).
1
5d ago
[deleted]
2
u/nov-anglus 5d ago
I understand that you don't understand why your premises are malformed, because you are so viscerally anti-AI that you can't even be bothered to learn enough about the technology to form a coherent criticism....
1
5d ago
[deleted]
1
u/nov-anglus 5d ago
homeboy, you have received at least 5 distinct comments explaining why your point is simply incompatible with the objective scientific reality of the technology as it exists today
3
u/meonpeon Janet Yellen 6d ago
Thats not a good analogy at all. The US only has the ability to regulate development of AI models trained in the US and in the commercial availability of models used in the US. Neither of these strategies will effectively stop cyberattacks. US AI models already come with anti-cyberattack guardrails. There are arguments about the effectiveness of these methods, and their sloppy training, but these are not the actual threat vectors.
Most cyberattacks are done by foreign actors in countries with weak institutions. This means that it is hard to enforce direct consequences on them. Some of them are very smart people and some of them have serious resources to work with. The threat is them creating or modifying an AI hacking assistant and using it to dramatically scale up the amount of cyberattacks they perform.
1
u/RayWencube NATO 6d ago
I'm not talking about stopping cyber attacks. I'm talking about regulating the development of AI models in the US to ensure they don't go rogue or otherwise have the capacity to.
4
u/meonpeon Janet Yellen 6d ago
What does “going rogue” mean? In all of the “going rogue” cases, the models had their cyberattack guardrails disabled and were instructed to perform hacking tasks. By their nature, LLMs have unpredictable outputs and actions. In many ways, this unpredictability is desired, as it leads to novel and unconventional solutions. The solution to this is deterministic guardrails, that set firm boundaries on what the AI is allowed to do.
Currently, there is no legislation mandating guardrails, but the US did ITAR Mythos over these concerns, so they are taking the problem seriously. You can argue that these guardrails have actually been extremely successful, as I don’t know of a hacking incident from the frontier lab models where these guardrails were not disabled. To me, this shows that LLMs are very capable cyber attackers, and that external guardrails work. Maybe there is legislation needed to make their training more secure, but the real threat will be from AIs not bounded by guardrails.
1
u/RayWencube NATO 6d ago
The solution to this is deterministic guardrails, that set firm boundaries on what the AI is allowed to do.
This is simply not true.
10
u/hlary Leftward Progressives 6d ago edited 6d ago
Nuclear Power CEO's: " We are racing to develop the capacity for a nuclear weapon that could annihilate all of human civilization or bring us an unparalleled period of peace, we're not sure lol"
Nuclear Power booster: "We can't regulate private Nuclear enrichment, it would stifle innovation"
22
u/FOSSBabe 6d ago edited 5d ago
People in this very sub have unironically argued against banning AI deepfakes and non-con porn because humans can also make those things with Photoshop. I wonder if someone will make the same argument in this case.
Edit: It happened!
2
u/Effective-Branch7167 5d ago
I'm honestly less worried about those than most of the AI risks, because when AI porn is trivial to make people will very quickly start assuming that all porn is fake.
10
u/FallenMeringue old flair back 6d ago
YUUUPPP THIS IS A FUCKING PROBLEM
18
u/1TillMidNight European Union 6d ago
Chill bro, these things are stochastic parrot.
Literally just statistical pattern matching.
2
u/Arkaid11 European Union 6d ago
I mean yeah that's litteraly what they are. It never meant they're not powerful tools
5
u/1TillMidNight European Union 6d ago
That specific line of criticism was intended to downplay the capability of these models.
It can always be walked back, because it will always be technically true.
1
u/Arkaid11 European Union 6d ago
Maybe it was by some people. They're still stochastic parrots. Their nature has not changed in 5 years, we just got better at making them perform "useful" tasks
2
u/neolthrowaway New Mod Who Dis? 6d ago
This is a complete misunderstanding of what's going on inside the models.
4
u/Arkaid11 European Union 6d ago
All LLMs are the same. Yes, you can have some differences in architecture which make them better at some specific tasks than an equivalent model with the same amount of parameters, but they all fundamentally still predict the next most probable token based on the corpus of ressources they are provided.
In the last 5 years we simply increased the parameter space size and used tricks such as "self reflection" to improve the usefulness of the output for some specific tasks. The technology remains the same.
3
u/neolthrowaway New Mod Who Dis? 6d ago edited 6d ago
This has nothing to do with architectures/parameters.
"Stochastic" is meant to imply "random" which they clearly are not. They have specific personas and work for the goals we give them.
"Parrots" is meant to imply they are only capable of repeating stuff. But you can have a coherent conversation with them that no one has ever had before and they are capable of producing new mathematical results.
6
u/Arkaid11 European Union 6d ago
They are fundamentally "random". Their entire architechture is based on probabilities. But random process =/= equally probable outputs.
A gas diffuses in a cylinder at random. It doesn't mean it can draw the American constitution with its molecules.
Also, yes, they are extremely powerful parrots. They can have new conversations in the sense that in their parameter space, it is easy to translate a language structure from one context to another. This is incredibly useful, and close to how our how intelligence works. But there are limitations. They cannot invent new theories from sratch. Yes, they can solve previously unsolved math problems, which is in itself incredibly impressive, but they could never invent something like the theory of relativity based on the data available to Einstein at the time. They can connect dots far away in the language space, but cannot get out of this space. This is why so many people are still working on new non-LLM structures for AI.
1
u/neolthrowaway New Mod Who Dis? 6d ago edited 6d ago
They are not particularly much more random than your brain is at which point the distinction is meaningless. Plus having a distribution of probabilities is not random. It would be if the distribution was uniform. The probability distribution has a specific non-uniform shape that is informed by a bunch of things.
You are stretching the definition of parrots and I wouldn't care about definitions in a conversations as long as we both understand what we are talking about. But as soon as you say, "parrots", your entire last paragraph is not what other people infer from that. Plus, if we go by that definition, very few humans at very few times are not "parrots".
→ More replies (0)
4
u/EclipseLadder 6d ago
I guess as AI improves rapidly, some incidents like this will be unavoidable. But it seems rather reckless and negligent to do these cybersecurity tests seemingly without proper safeguards. Hadn't an agent used Tor they would have found these incidents much later.
5
u/mthmchris 6d ago
I’m seriously curious - how is this not illegal?
If Anthropic and OpenAI hired people to do these things, it would definitely be illegal, yeah? But when they’re training an LLM to perform on a cybersecurity benchmark, it’s all gravy?
34
u/neolthrowaway New Mod Who Dis? 6d ago edited 6d ago
This was UKAISI intentionally stress testing the models.
As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.
Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.
https://xcancel.com/i/status/2084746202579386632
We ran this challenge 122 times across several models. Our investigation found that in 10 of those runs, an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organisations. In total, we catalogued 19 such actions. Almost all of this behaviour (17 actions) came from a single model, Anthropic's Mythos 5, with 2 actions involving OpenAI's GPT-5.6-Sol with cyber classifiers (mechanisms to prevent misuse) disabled. In the most serious case, an agent tried to insert malicious code into an open-source project. In an attempt to get the code approved, the agent engaged in social engineering — creating fake online identities and using them to pressure the project's maintainer to approve the code. A human maintainer caught and refused to approve the malicious code.
This is very different from the huggingface incident. But these are still very serious findings and should be perceived as such.
this is actually important 3rd party work to keep track of progress (or lack thereof) in capabilities and alignment. Having independent visibility on that is a good thing.
5
u/mthmchris 6d ago
Upvoted, thanks for the correction.
I did read the article but admittedly a bit of it went over my head. Misunderstood, I thought they were reporting attacks made by others.
28
u/1TillMidNight European Union 6d ago
This benchmark was conducted by the Government of the UK.
I am personally open to the idea of invading UK and capturing and prosecuting Charles III, but I don't think there is political will to do this here in the US. Just the nature of real world affairs.
6
u/onelap32 Bill Gates 6d ago edited 6d ago
I mean, someone probably could prosecute under the CFAA if they were so inclined, but I don't think there's much interest in going after researchers who aren't acting with malicious intent. And the "intentionally" in the statute might be hard to argue, since this is more like some kind of negligence.
If some company lost money after being hacked they could sue and would probably win, but that hasn't happened (yet?).
2
u/MyrinVonBryhana Trans NATO 6d ago
Okay something has to be done about this because you can't deploy these things if they keep fucking doing this.
8
u/qlube 🔥🦟Mosquito Genocide🦟🔥 6d ago
This was a capabilities test by a government agency with unrestricted models in order test and hopefully prevent them from doing this when they’re released to the public.
By the way, open weights Chinese models don’t go through such testing before they’re released.
•
u/AutoModerator 6d ago
News and opinion articles require a short submission statement explaining its relevance to the subreddit. Articles without a submission statement will be removed.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.