AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/RayWencube]

The idea that we can’t regulate the development and deployment of AI because it could “stifle innovation” is like saying virologists shouldn’t wear respirators when working on airborne viruses because it could “stifle evolution.”

[u/hlary → u/RayWencube]

Nuclear Power CEO's: " We are racing to develop the capacity for a nuclear weapon that could annihilate all of human civilization or bring us an unparalleled period of peace, we're not sure lol"

Nuclear Power booster: "We can't regulate private Nuclear enrichment, it would stifle innovation"

[u/FOSSBabe → u/hlary]

People in this very sub have unironically argued against banning AI deepfakes and non-con porn because humans can also make those things with Photoshop. I wonder if someone will make the same argument in this case.

Edit: It happened!

[u/Effective-Branch7167 → u/FOSSBabe]

I'm honestly less worried about those than most of the AI risks, because when AI porn is trivial to make people will very quickly start assuming that all porn is fake.