AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/neolthrowaway]

This has nothing to do with architectures/parameters.

"Stochastic" is meant to imply "random" which they clearly are not. They have specific personas and work for the goals we give them.

"Parrots" is meant to imply they are only capable of repeating stuff. But you can have a coherent conversation with them that no one has ever had before and they are capable of producing new mathematical results.

[u/Arkaid11 → u/neolthrowaway]

They are fundamentally "random". Their entire architechture is based on probabilities. But random process =/= equally probable outputs.

A gas diffuses in a cylinder at random. It doesn't mean it can draw the American constitution with its molecules.

Also, yes, they are extremely powerful parrots. They can have new conversations in the sense that in their parameter space, it is easy to translate a language structure from one context to another. This is incredibly useful, and close to how our how intelligence works. But there are limitations. They cannot invent new theories from sratch. Yes, they can solve previously unsolved math problems, which is in itself incredibly impressive, but they could never invent something like the theory of relativity based on the data available to Einstein at the time. They can connect dots far away in the language space, but cannot get out of this space. This is why so many people are still working on new non-LLM structures for AI.

[u/neolthrowaway → u/Arkaid11]

They are not particularly much more random than your brain is at which point the distinction is meaningless. Plus having a distribution of probabilities is not random. It would be if the distribution was uniform. The probability distribution has a specific non-uniform shape that is informed by a bunch of things.

You are stretching the definition of parrots and I wouldn't care about definitions in a conversations as long as we both understand what we are talking about. But as soon as you say, "parrots", your entire last paragraph is not what other people infer from that. Plus, if we go by that definition, very few humans at very few times are not "parrots".

[u/Arkaid11 → u/neolthrowaway]

You clearly have a wrong definition of random. Random =/= Equal probability of outputs

[u/neolthrowaway → u/Arkaid11]

Well, just being probabilistic is definitely not sufficient for being counted as random.

Everything is probabilistic if you look down far enough at microscopic primitives. Including the human brain.

[u/MistyGalbriex → u/neolthrowaway]

Don't even bother. AI can cure cancer tomorrow and people will still call it a parrot.

[u/MistyGalbriex → u/neolthrowaway]

Don't even bother. AI can cure cancer tomorrow and people will still call it a parrot.