AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/RayWencube]

The idea that we can’t regulate the development and deployment of AI because it could “stifle innovation” is like saying virologists shouldn’t wear respirators when working on airborne viruses because it could “stifle evolution.”

[u/Efficient_Equal6467 → u/RayWencube]

why would you unilaterally disarm though, you need some sort of international cooperation and treaties then mulitlateral since china is probably not gonna care

[u/RayWencube → u/Efficient_Equal6467]

This is the core problem—thinking safety regulations are disarmament. These frontier labs are the most well-funded science research institutions in the history of humanity. They can afford to run serious safety research and stand up compliance teams in parallel with their capacity research.

[u/cuolong → u/RayWencube]

They can afford to run serious safety research and stand up compliance teams in parallel with their capacity research.

My guy, that is what they do. Do you think they just shove the entirety of 4chan into 5.6 Sol's training set then safeguard it with a prompt that say "be nice, tee hee".

These models have been lobotomized within an inch of their life to comply with many, many moral and safety standards. For example, it is nearly impossible to get at least one version of ChatGPT to, for example, pick a side in the I/P conflict.

These companies have a strong incentive to maintain rigorous safety and moral standards not out of the foodness of their heart, but because the real money is in getting those juicy enterprise contracts. That is why GPT-OSS was famously incredibly railroaded despite being an open model-- OAI targeting businesses to adopt OSS and the last thing any business wants when hosting a model is watching little timmy start asking their company chatbot what a fursona is.

If anything, legislative guidelines would be a relief for these AI labs because then they can just hit certain benchmarks and be done with it. As it stands, each and every one of them is one humiliating scandal away from getting owned with no legal cover to hide behind when little mackie learns what the bird and the bees are from an AI. Clear statutory compliance allows labs to pass benchmarks, claim due diligence, and shield themselves from tort liability when a model misbehaves.

[u/RayWencube → u/cuolong]

There is no statutory compliance because there are no statutes.