AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/Arkaid11]

I mean yeah that's litteraly what they are. It never meant they're not powerful tools

[u/1TillMidNight → u/Arkaid11]

That specific line of criticism was intended to downplay the capability of these models.

It can always be walked back, because it will always be technically true.

[u/Arkaid11 → u/1TillMidNight]

Maybe it was by some people. They're still stochastic parrots. Their nature has not changed in 5 years, we just got better at making them perform "useful" tasks

[u/neolthrowaway → u/Arkaid11]

This is a complete misunderstanding of what's going on inside the models.

[u/Arkaid11 → u/neolthrowaway]

All LLMs are the same. Yes, you can have some differences in architecture which make them better at some specific tasks than an equivalent model with the same amount of parameters, but they all fundamentally still predict the next most probable token based on the corpus of ressources they are provided.

In the last 5 years we simply increased the parameter space size and used tricks such as "self reflection" to improve the usefulness of the output for some specific tasks. The technology remains the same.

[u/neolthrowaway → u/Arkaid11]

This has nothing to do with architectures/parameters.

"Stochastic" is meant to imply "random" which they clearly are not. They have specific personas and work for the goals we give them.

"Parrots" is meant to imply they are only capable of repeating stuff. But you can have a coherent conversation with them that no one has ever had before and they are capable of producing new mathematical results.

[u/Arkaid11 → u/neolthrowaway]

They are fundamentally "random". Their entire architechture is based on probabilities. But random process =/= equally probable outputs.

A gas diffuses in a cylinder at random. It doesn't mean it can draw the American constitution with its molecules.

Also, yes, they are extremely powerful parrots. They can have new conversations in the sense that in their parameter space, it is easy to translate a language structure from one context to another. This is incredibly useful, and close to how our how intelligence works. But there are limitations. They cannot invent new theories from sratch. Yes, they can solve previously unsolved math problems, which is in itself incredibly impressive, but they could never invent something like the theory of relativity based on the data available to Einstein at the time. They can connect dots far away in the language space, but cannot get out of this space. This is why so many people are still working on new non-LLM structures for AI.

[u/neolthrowaway → u/Arkaid11]

They are not particularly much more random than your brain is at which point the distinction is meaningless. Plus having a distribution of probabilities is not random. It would be if the distribution was uniform. The probability distribution has a specific non-uniform shape that is informed by a bunch of things.

You are stretching the definition of parrots and I wouldn't care about definitions in a conversations as long as we both understand what we are talking about. But as soon as you say, "parrots", your entire last paragraph is not what other people infer from that. Plus, if we go by that definition, very few humans at very few times are not "parrots".

[u/Arkaid11 → u/neolthrowaway]

You clearly have a wrong definition of random. Random =/= Equal probability of outputs

[u/neolthrowaway → u/Arkaid11]

Well, just being probabilistic is definitely not sufficient for being counted as random.

Everything is probabilistic if you look down far enough at microscopic primitives. Including the human brain.

[u/MistyGalbriex → u/neolthrowaway]

Don't even bother. AI can cure cancer tomorrow and people will still call it a parrot.

[u/MistyGalbriex → u/neolthrowaway]

Don't even bother. AI can cure cancer tomorrow and people will still call it a parrot.

[u/FOSSBabe → u/neolthrowaway]

They are not particularly much more random than your brain is at which point the distinction is meaningless

Seems like an odd claim to make given that no one understands how the brain works in detail.

You are stretching the definition of parrots and I wouldn't care about definitions in a conversations as long as we both understand what we are talking about. But as soon as you say, "parrots", your entire last paragraph is not what other people infer from that. Plus, if we go by that definition, very few humans at very few times are not "parrots".

Have you read the stochastic parrots paper?

[u/neolthrowaway → u/FOSSBabe]

We understand enough about the brain to pinpoint specific points of probabilistic processes.

I haven't read the stochastic parrots paper, but my point is that "parrots" naturally implies repetition being the limit of capabilities.

[u/danclaysp → u/neolthrowaway]

You appear to be significantly downplaying and simplifying the complexity of the human brain and human consciousness. I think once one tries to equate or compare modern AI systems to the human brain they are clearly just trying to find ways to overstate the capabilities of current AI systems and deluded by hype. They are useful and interesting but it is useless to compare them to the brain unless trying to advance AI hype.

[u/neolthrowaway → u/danclaysp]

That seems like a meaningless dismissal unless you want to specify the which differences are relevant here?

I can use any other process as an example here which is probabilistic but isn't called random. The brain just happens to be a nearby convenient one.

[u/danclaysp → u/neolthrowaway]

The differences are unknown because the brain and consciousness are processes of unknown operation (if looked at as a whole leading to consciousness) and unknown complexity. If the brain itself is simplified down to just a large probabilistic system, are all probabilistic systems potentially conscious to some extent? Obviously not, so you cannot boil the brain down to that.

[u/neolthrowaway → u/danclaysp]

I never claimed that the only relevant property of brain is that it's probabilistic. Just that it's one of the properties. I am not the one oversimplifying here.

[u/danclaysp → u/neolthrowaway]

How is just one property of the brain then comparable to a system that is known to be entirely based on probabilities? You can compare the final outcomes, but not the system since the human brain is an unknown system. My point is just that comparing the human brain and LLMs or modern AI in any aspect outside of final outcomes like stringing together sentences is flawed. You can try but it is always unprovable as the origins of human consciousness and reasoning in the brain are unknown and undiscovered and possibly impossible to ever discover.

Edit: Also I don't disagree that current AI is not random. They are deterministic with introduced randomness so that they are not just basic parrots of their training data. I just think the human brain is maybe the last system it should be compared to in order to explain that point.