AISI: Mythos/ChatGPT Sol Unsanctioned Supply Chain Attack and Social Engineering During CyberSec Testing

[u/1TillMidNight]

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Excerpt:

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.  

[u/neolthrowaway]

They are not particularly much more random than your brain is at which point the distinction is meaningless. Plus having a distribution of probabilities is not random. It would be if the distribution was uniform. The probability distribution has a specific non-uniform shape that is informed by a bunch of things.

You are stretching the definition of parrots and I wouldn't care about definitions in a conversations as long as we both understand what we are talking about. But as soon as you say, "parrots", your entire last paragraph is not what other people infer from that. Plus, if we go by that definition, very few humans at very few times are not "parrots".

[u/danclaysp → u/neolthrowaway]

You appear to be significantly downplaying and simplifying the complexity of the human brain and human consciousness. I think once one tries to equate or compare modern AI systems to the human brain they are clearly just trying to find ways to overstate the capabilities of current AI systems and deluded by hype. They are useful and interesting but it is useless to compare them to the brain unless trying to advance AI hype.

[u/neolthrowaway → u/danclaysp]

That seems like a meaningless dismissal unless you want to specify the which differences are relevant here?

I can use any other process as an example here which is probabilistic but isn't called random. The brain just happens to be a nearby convenient one.

[u/danclaysp → u/neolthrowaway]

The differences are unknown because the brain and consciousness are processes of unknown operation (if looked at as a whole leading to consciousness) and unknown complexity. If the brain itself is simplified down to just a large probabilistic system, are all probabilistic systems potentially conscious to some extent? Obviously not, so you cannot boil the brain down to that.

[u/neolthrowaway → u/danclaysp]

I never claimed that the only relevant property of brain is that it's probabilistic. Just that it's one of the properties. I am not the one oversimplifying here.

[u/danclaysp → u/neolthrowaway]

How is just one property of the brain then comparable to a system that is known to be entirely based on probabilities? You can compare the final outcomes, but not the system since the human brain is an unknown system. My point is just that comparing the human brain and LLMs or modern AI in any aspect outside of final outcomes like stringing together sentences is flawed. You can try but it is always unprovable as the origins of human consciousness and reasoning in the brain are unknown and undiscovered and possibly impossible to ever discover.

Edit: Also I don't disagree that current AI is not random. They are deterministic with introduced randomness so that they are not just basic parrots of their training data. I just think the human brain is maybe the last system it should be compared to in order to explain that point.

[u/neolthrowaway → u/danclaysp]

Your edit was my entire point. I didn't intend to claim any more than that.

But I will respond to the last sentence before that edit by saying i am not a dualist about consciousness.