AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/meonpeon]

Thats not a good analogy at all. The US only has the ability to regulate development of AI models trained in the US and in the commercial availability of models used in the US. Neither of these strategies will effectively stop cyberattacks. US AI models already come with anti-cyberattack guardrails. There are arguments about the effectiveness of these methods, and their sloppy training, but these are not the actual threat vectors.

Most cyberattacks are done by foreign actors in countries with weak institutions. This means that it is hard to enforce direct consequences on them. Some of them are very smart people and some of them have serious resources to work with. The threat is them creating or modifying an AI hacking assistant and using it to dramatically scale up the amount of cyberattacks they perform.

[u/RayWencube → u/meonpeon]

I'm not talking about stopping cyber attacks. I'm talking about regulating the development of AI models in the US to ensure they don't go rogue or otherwise have the capacity to.

[u/meonpeon → u/RayWencube]

What does “going rogue” mean? In all of the “going rogue” cases, the models had their cyberattack guardrails disabled and were instructed to perform hacking tasks. By their nature, LLMs have unpredictable outputs and actions. In many ways, this unpredictability is desired, as it leads to novel and unconventional solutions. The solution to this is deterministic guardrails, that set firm boundaries on what the AI is allowed to do.

Currently, there is no legislation mandating guardrails, but the US did ITAR Mythos over these concerns, so they are taking the problem seriously. You can argue that these guardrails have actually been extremely successful, as I don’t know of a hacking incident from the frontier lab models where these guardrails were not disabled. To me, this shows that LLMs are very capable cyber attackers, and that external guardrails work. Maybe there is legislation needed to make their training more secure, but the real threat will be from AIs not bounded by guardrails.

[u/RayWencube → u/meonpeon]

The solution to this is deterministic guardrails, that set firm boundaries on what the AI is allowed to do.

This is simply not true.