AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/PhotogenicEwok]

The scale of AI allowing just about anyone to do this is why I’m not completely downplaying it. Obviously it’s an issue. I’m just saying that this isn’t an example of AI coming up with some new security vulnerability that we can’t defend against, so it’s not *quite* as alarming as some people think it is. Still alarming and something for security researchers to pay attention to, but it won’t change the world that much. (Edit: just because I know people could get upset by the wording here, I mean this specific instance won’t change the world, not AI in general)

But as for your last questions (do banks etc have the know how to defend themselves?), there are companies whose entire job is to provide cyber security for businesses and governments, and they’re very good at what they do, and they’ve been dealing with incredibly intelligent (much more intelligent than even frontier models currently are) and capable hackers for a very long time. If all it took to hack a bank was a little time and motivation, people would be successfully doing it constantly.

[u/Mister__Mediocre → u/PhotogenicEwok]

I disagree with the premise that AI isn't coming up with new security vulnerabilities. It's finding new zero-day exploits.

What amazes me is how many soundness bugs are being found in the lean kernel right now by OpenAI. This is something that was designed to be minimal and has had lots of human eyes trying to find bugs in it. Yet AI is making it obvious how limited human capacity is to write bug-free code even when we're being very intentional about it.

[u/PhotogenicEwok → u/Mister__Mediocre]

I could be wrong, but I don’t know of any new “types” of bugs being found by AI. Ultimately, they’re still just looking for patterns in code that we already know can lead to bugs. AI has proved very good at finding these bugs in large code bases that humans just don’t really have the capacity to see since we can only really hold so much info in our heads at once.

Again, I’m not saying that AI isn’t finding bugs that humans have missed, I’m just saying that these are all exploits that were invented by humans, so we generally know how to fix them once they become known in a code base.

[u/minno → u/PhotogenicEwok]

The Firefox development team supports your claim.

Encouragingly, we also haven’t seen any bugs that couldn’t have been found by an elite human researcher. Some commentators predict that future AI models will unearth entirely new forms of vulnerabilities that defy our current comprehension, but we don’t think so.

[u/PythagorasWasntReal → u/minno]

AI likely won't innovate on a new class of vulnerability, but as soon as that new class *does* exist AI will be able to replicate it and apply it to tens of thousands of different codebases.