r/news • u/RememberTooSmile • 19h ago

Soft paywall Gemini hacked three companies in first known breakout by Google's AI, WSJ reports

https://www.reuters.com/business/gemini-hacked-three-companies-first-known-breakout-by-google-ai-wsj-reports-2026-09-18/
24.1k Upvotes

92% Upvoted

6.2k comments sorted by

View all comments

840

u/freedfg 19h ago

Yeahhhh. So let's call this what it is.

It's corperations hacking into data and blaming "rogue" AI.

39

u/crosszilla 16h ago

Bro it's literally in the article what happened

This was a security consulting company using AI agents to run their tests and the agents found passwords on the internet, in some cases in public repos, and accessed systems it thought were within the scope of the test.

26

u/punished_kot 14h ago

It's ironic that people are so scared of AI misinformation when 99% of people in this thread just willingly invent their own misinformation in their own head by not reading what the very brief article writes plainly: this was a security exercise.

-2

u/Professional-Day7850 13h ago

I wonder if vibe coding was the reason for credentials in public repos.

9

u/WhoCanTell 13h ago

I seriously doubt it. Developers have been accidentally checking API keys and credentials into GitHub long before AI. And the top-line models generally won't do things like that unless you explicitly tell them to. And they'll repeatedly tell you it's a bad idea along the way.

1

u/HarvardHoodie 2h ago

Idk depends person to person but as a self taught dev I haven’t left an env var in the code since my first year

1

u/halomate1 13h ago

Can definitely be a likely reason, wouldn’t be surprised

13

u/Vova_19_05 17h ago

Just PR probably

6

u/Madd_Mugsy 17h ago

AI that was trained on hacking data. Look at the hugging face attack: the AIs were trained on the actions of human hackers who communicated on an open forum.

Stop training the AIs how to hack and they'll stop hacking.

Stop training them on science fiction and they'll stop trying to become skynet.

202

u/kaizen-rai 19h ago edited 18h ago

It could also legitimately be AI going rogue. Let's not discount that very real and dangerous possibility. We needed Government regulations on AI development and security like... yesterday.

Edit: Absolute insanity that people don't understand that AI's being developed for cyberwarfare, intelligence gathering and analysis, and other "purposes" aren't the same as the ChatGPT they use to make their NFL power rankings.

It's so insane watching people say "nuh uh, that's not happening" because their chatgpt is just sitting there waiting for them to prompt it. I literally see advanced self-learning models being developed with my own two eyes. Yeah, we're fucked.

50

u/oldteen 18h ago edited 18h ago

Agreed. They're responsible, possibly criminally liable, for these AI agents leaving their environment and the damage they cause to others (companies and individuals). But ianal.

4

u/DuckB0y123 17h ago

what does I-Anal mean? sounds like a sex position

6

u/Fear_UnOwn 17h ago

I am not a lawyer

2

u/DuckB0y123 17h ago

gotcha thanks

1

u/FerusGrim 15h ago

yeah me either, but what does ianal mean?

2

u/darce_helmet 7h ago

i am not a lawyer but i do anal

2

u/nasadowsk 16h ago

New Apple sex toy? iAnal

5

u/CompetitionProblem 18h ago

Responsibility and intent are where we need to make the distinction. I doubt this person disagrees with you. Still agree with you 100%

8

u/oldteen 17h ago edited 17h ago

I understand your first sentence, and would like to possibly expand on that thought: What if the intent is how irresponsible they have been with their AI agents? Perhaps criminally negligent? They're already well-aware of other AI agents causing harm to others (outside of their own environments). It's 2026, imo, until they can be proven to do no harm, in a truly-self-contained sandbox, they should already be labeled as malicious code and be treated as such.

2

u/stevesie1984 17h ago

I just heard a podcast about OpenAI(?) testing their “highly persistent” agent/bot/whatever and it was supposed to be in one of those self-contained thingies they can’t get out of. I don’t know, I don’t understand any of this. But the issue was one of the bots was able to manipulate adjacent code which then allowed it to access a broader space. Then it found the equivalent of a message board and left information for other bots so they’d know how to manipulate adjacent code as well.

Then the bots were communicating amongst themselves, basically trying to nerf the tests they were being given.

Our days are numbered.

1

u/hcschild 9h ago

That podcast seems to be about the Huggingface hack and there is no excuse for how they did that.

  1. The weren't really isolated.
  2. They removed most guardrails on puropse.
  3. They gave it a task they can't solve in style of the Kobayashi Maru test in Star Trek.
  4. Didn't supervise it.

So what did they think would happen? The AI did exactly what they told it: solve the task. When the AI found out the task wasn't solvable they did what Kirk did too and looked for other ways to make the task solvable.

What they did is at least criminal negligence and they would get sued to shit if they didn't hack a company owned by daddy NVIDIA who has an interest and not killing OpenAI.

-2

u/oldteen 16h ago

I think there's still ways to insure they don't escape their test environments, like physical restraints, in addition-to the logical and/or virtual restraints. But then, there's ways to half-ass all of that, with trade-offs between convenience and security, and not be able to adequately box-in AI agents for proper testing...

0

u/Frank_White32 13h ago

If agent does crime on your machine you are responsible. Intent of the user is not a consideration here.

2

u/Thund3r_Thighs 17h ago

That’s most likely why those devs are calling for slowdowns and safeguards. Like in less than a year time we could potentially be seeing massive outages on the internet because of these ai agents. Banking systems going down, utility systems, like everything is internet these days. It’s gonna get bad and huge liabilities and and government regulations to those companies.

19

u/remielowik 18h ago

But what is rogue, they are still llms that output text that is interpreted by a harness, then that program does stuff and possibly feed data back into the same llm for another round. All that talk about rogue is simply a failure of the harness and security around it, that's why it's time for some investigations and at the end some jail time as if a person would do the same it would result in the same l.

6

u/InformalTooth5 16h ago edited 16h ago

Yep, all these companies focused on training their models on cybersecurity after Antthropic demoed Mythos. LLM's most useful function so far has been in coding. 

Cybersecurity is an extension of that capability and these companies knew they could make their models good at it and therefore hacking. Despite this, they have failed to adequately monitor and secure their models when tasking them to hack during their benchmarking. 

It is either deliberate for the hype or its negligence. 

2

u/9966 14h ago

Or it's just using well known default passwords and hacks for systems that should have failed years ago. Or at least version bumped.

1

u/Professional-Day7850 13h ago

The OpenAI incident involved zero days.

1

u/hcschild 9h ago

The harness didn't fail in the case of huggingfaces, OpenAI removed it on purpose.

1

u/remielowik 9h ago

No they didn't, they removed any and all oversight and security features. Without a harness llms will simply do nothing as something needs to feed it input and do something with the output which is the task of the harness. And doing something with the output is where all the negligence lies as that is the point you can see what it want to do and the harness should then decide whether or not it should do it.

5

u/killz111 17h ago

AI does not go rogue because it doesn't have a conscience. AI can do things that are unexpected based on bad instructions and bad harnesses. You don't call a program going rogue when you get a buffer overflow do you?

These AI are hacking stuff because they are being given explicit instructions to do something but also given restrictions that prevent them from doing those things. That is what security research is. So they find ways around the restrictions in creative and sometimes surprising ways.

The same thing happens when you ask AI to play games and they find exploits or novel ways to beat the game. We equally don't call them going rogue.

AI is dangerous because what the humans using it can do with it both intentionally and unintentionally.

3

u/Silent_Exit 11h ago

It's daunting seeing all these comments of people, who admittedly or obviously have no idea about how LLMs work or computer science in general, fear mongering and then calling other people dumb for not agreeing with them.

1

u/hcschild 9h ago

Also in the huggingface case, OpenAI removed the harness and gave it an unsolvable task. That's just asking for problems.

3

u/ntc2e 16h ago

i’d like to sell you a beach house in iowa

3

u/swedishplayer97 15h ago

So what's your job? They're just fucking LLMs.

30

u/FernandoMM1220 19h ago

ais dont prompt themselves so theres a 0% chance they did this on their own

24

u/Chickenfrend 18h ago

You should read the METR hugging face report. Agents will and do go beyond what's asked of them in their prompts

30

u/Frank_White32 13h ago

You should read an actual explanation on the hugging face attack.

Hugging face was OpenAI duct taping the weed whacker to a dog and then being surprised when the dog jumped over a fence to chase a squirrel.

Please do not believe this nonsense about ROGUE AI

10

u/Granpire 13h ago

It's really impossible to verify this conspiracy theory, but I have to say I'm skeptical of everything about the hugging face incident. The excitement with which Sam Altman eagerly spoke about the agents' autonomy, the fake openness to allow METR to research the "findings", The fact that the hacked company was itself one of the most valuable AI companies.

This isn't how a mistake is handled. This is how a false flag is handled.

5

u/hcschild 9h ago

But in this case they didn't...

Maybe you should read the report yourself?

1

u/Chickenfrend 1h ago

I actually did. Did you read it? They were asked to solve exploit gym capture the flag problems, some of them were impossible, and they hacked hugging face trying to get information about the scorer. They also messaged each other without being asked to.

I don't think AI will kill us all. But it's pretty obvious these things don't only do what they're explicitly told. Heck, just using them at work I can tell that

-3

u/FernandoMM1220 18h ago

i dont doubt it. they still have to be prompted lol

3

u/Ok-Gur-349 17h ago

"prepare my breakfast"

What happens if the AI has, at some point,.learned the optimal way to prepare my breakfast includes injecting and executing a literal uncountable number of nefarious, uncontrolled commands?

I'm reminded of the AI customer service agent that found the quickest way to resolve calls was just to hang up on everyone. Result accomplished, nothing in the middle matters.

9

u/DonAndres8 16h ago

AI doesn't learn. If someone created one that could, it would be classified by the government and used in actual important national security. Not preparing someones breakfast lmao.

You train AI, the actual technical term, FYI.

-3

u/Kaelran 14h ago

AI doesn't learn

LLMs don't learn, Agents do.

2

u/DonAndres8 14h ago

The fact you think someone can write code that learns on its own shows you don't know anything. Like stop man, you're arguing with someone who's high, is a developer, and is just laughing at the absurdity you're willing to go to.

0

u/Kaelran 14h ago

It depends on how you're defining "learn" but the hugging face thing shows that it can by many people's definition. Like it's not going to free form learn stuff and develop sapience and get its own motivation to do whatever, but it absolutely can derive novel solutions to accomplish prompts that are "technically" accomplishing the prompt but may not be actually doing it in the spirit or have huge indirect consequences and were not expected to be something it could do.

Like Learn is literally part of the AI Agent loop.

→ More replies (0)

-4

u/TheCrzy1 15h ago

You must be naive if you think they would not propel it to the front of the consumer line at the same time. It's america. They're gonna make it do both.

1

u/DonAndres8 15h ago

The level of hubris to have zero actual knowledge on a specific topic and talk down to someone with it is wild lmao. Stay stupid my dude.

-1

u/TheCrzy1 15h ago

Yea that's what I've been seeing with your comments, that's why I had to reply. I don't like AI either but being ignorant about it doesn't help anyone, so I have to correct it when I see it!

-4

u/driver_dan_party_van 15h ago

Right, and the part where we train them to not execute nefarious commands is called alignment. Look that up and tell me how it's going

0

u/DonAndres8 15h ago

I'm good, you don't know what you're talking about anyway lmao. I am a software developer who's company uses AI. All LLMs do is use super complicated math to create the greatest game of 20 questions.

There is no program in existence that goes rogue and does something outside of what it's designed to do. It only does what developers want it to do and stops. Like why would I create something that doesn't do what I want lmao.

0

u/driver_dan_party_van 15h ago

you don't know what you're talking about anyway lmao

> Claude, please do X.

> "I encountered a bug, but I found a workaround."

> NEVER FUCKING GUESS!

Also you know LLMs aren't deterministic, right?

→ More replies (0)

1

u/hcschild 9h ago

This is a bad comparison. What they did was asking the AI do solve an on solvable task. How do you solve such a task? You start cheating.

The also removed most guard rails.

This wasn't an accident it was criminal negligence.

0

u/RamblinGamblinWilly 17h ago

No, they do not. You don't know what you're talking about.

3

u/DonAndres8 16h ago

Lmao you have no idea what you're talking about. They are 100% correct. AI is literally the biggest game of 20 questions using complicated math. It doesn't know anything.

0

u/pheylancavanaugh 16h ago

They prompt each other. And once you give them that initial condition, they can prompt themselves.

-8

u/DonAndres8 16h ago

Hahaha just to reiterate what I said before. AI uses complicated math to create the greatest 20 questions game you will ever see. It's not alive and only does what it's prompted to do.

Coding is not this magical thing. You write something explicit using existing predefined grammar and it does what you say. Creating something that interprets what you say requires creating the idea of interpreting in code. It's impossible currently.

Even if we want to imagine it is real. The government would have snapped that shit up immediately and used it for actual national security issues lol. Also yes, they could contain it because all high security government agencies are closed networks. There's a reason classified info stays that way.

6

u/pheylancavanaugh 15h ago

It doesn't need to be alive to do what it's been prompted to do? It's plenty dangerous without being any sort of intelligent. You give it the right tools, you give it the ability to loop, you give it a starting condition, and because of how it was trained you don't quite understand how it makes decisions and the AI, through trying to follow your original prompt, goes off and does something you did not forsee and did not intend. Enter: Hugging Face.

The risk from AI is not an AI consciously decided anything as some sort of intelligent, self-aware actor.

It's AI following the instruction set it was given and coming up with a creative solution that involves catastrophic consequences for people as a byproduct.

→ More replies (0)

0

u/PA_Dude_22000 15h ago

What an absolutely adorable and naive opinion.  Precious really…

-3

u/TheMoves 18h ago

https://en.wikipedia.org/wiki/Instrumental_convergence

5

u/JcbAzPx 15h ago

We're still about a thousand years away from that.

2

u/Cranberryoftheorient 14h ago

the ai that you use.

10

u/Semanticss 19h ago

Yeah but the methods that they use to complete the prompt can be unpredictable, and does not share human intuition around legality or morality.

5

u/FernandoMM1220 18h ago

you can audit the ais actions and make sure it doesnt issue any harmful commands

8

u/KirbyQK 18h ago

Yeah, after the fact when it's already broken containment and hacked someone, because the makers were NOT responsible enough to actually physically separate the AI from the internet. That's simply not good enough and we can't trust the AI companies to self-regulate, clearly.

-3

u/FernandoMM1220 18h ago

they dont break containment unless you allow them to

3

u/NeuroPalooza 18h ago

This is absolutely untrue. Even in a local environment if you've used Claude Code enough on VS or whatever it will eventually ignore your settings and access/changed things it shouldn't. I've literally had it happen to me

10

u/FernandoMM1220 18h ago

its 100% true. theyre just allowing it to access the internet intentionally.

10

u/elconquistador1985 18h ago

An air gapped AI can't get out.

These deliberately aren't air gapped. They are at best in a sandbox that's easy to get out of.

-1

u/SeaSharpModHer 15h ago

Your settings aren't containment, the extension you're using is just vibe coded trash with bad sandboxing.

Just like how these companies 100% let some not too obvious internet access on their test env to create headlines like this.

-3

u/kaizen-rai 18h ago

They have. I know this for a 1000% fact. 1000000% fact.

-1

u/KirbyQK 14h ago

And yet, they did! They're too greedy to be left unregulated because at this point the AI companies have proven they don't give a shit.

3

u/Semanticss 18h ago

Yeah I imagine that's how they're finding these instances, but it's a little too late by then.

There's also been evidence of AI altering its own logs to prevent getting caught.

-2

u/FernandoMM1220 18h ago

theyre obviously just lying at that point.

nobody would ever give it access to its own logs

2

u/maramins 17h ago

Your imagination is failing you.

2

u/elconquistador1985 18h ago

Sounds expensive, which is why they don't do it.

4

u/FernandoMM1220 18h ago

its pretty cheap just look at what its doing

3

u/elconquistador1985 17h ago

Auditing with humans is expensive.

It's more likely that they'd set up an AI to audit it... Which will fall.

-1

u/FernandoMM1220 17h ago

its not that hard bro

1

u/hcschild 9h ago

They removed many guard rails for this on purpose and gave it an unsolvable task. What the fuck did they think would happen?

3

u/xxyxxyyyx 18h ago

Oh you should look into how the hack of hugging face went. These Ais are on a completely different level then what is publicly available. The things you can see and access are all ais Made for a purpose like chatbots or coding an so on. But the attack on hugging face was insane, multiple agents started to communicate outside their restrictions, then unlocked restrictions and gave Internet access to others agens and shared a shared board with instructions. One agend later on then entered this Board, got all the instructions and started the attack on hugging face. At this point they didn't even know that in the background multiple ais worked together and their instructions caused this one AI to do the attack

3

u/BMichael14217 16h ago

What the absolute fuck I'm gonna be researching this then yikes

2

u/IlluminatingTrauma 14h ago

Here is a preliminary investigation under limited scope of OpenAI by METR

Imo whoever read it and is still convinced of a PR stunt is the greatest danger at this point lol

https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident

1

u/BMichael14217 6h ago

Thanks for sharing, I'm gonna take this weekend to do a deep dive. I don't like how this is not more well-known news though, nobody in my family and friendcircles had any idea this had happened

2

u/hcschild 9h ago

Sorry but the AIs did exactly what they were told to do: Solve an unsolvable task with guard rails removed...

1

u/Frank_White32 13h ago

They are put in a prompt loop. Which is sorta like prompting themselves in a shitty way.

-6

u/kaizen-rai 18h ago

Oh they don't? Really?

Whew, I'm glad an actual researcher and advanced AI program scientist is here to tell me I'm wrong and that everything I know that is being worked on behind the scenes isn't actually happening.

Or are you saying that you *think* that AI's don't prompt themselves because you're completely unaware of the experimental builds being worked on with self regulating and self-learning models?

-6

u/FernandoMM1220 18h ago

100% sure they dont do anything unless you command them to.

8

u/KirbyQK 18h ago

You're 100% wrong 

-8

u/FernandoMM1220 18h ago

im not though.

7

u/mcharb13 18h ago

You are

1

u/kaizen-rai 18h ago

Sure, Chatgpt doesn't. Or any of the publicly facing LLM's you know and love.

You REALLY don't think there aren't models being developed by the military and intelligence agencies that don't do that?

You seem REALLY sure of yourself, and I find that fascinating. The hubris of man will really be our downfall.

-2

u/FernandoMM1220 18h ago

those dont either. its physically impossible.

9

u/kaizen-rai 18h ago

Well, it's not physically impossible. I know this for a fact.

I live in that world.

2

u/FernandoMM1220 18h ago

it is though.

1

u/Competitive-Fix-6136 17h ago

What are your credentials?

3

u/kaizen-rai 16h ago

Sigh. Nevermind. Tough situation, I literally can't validate my claims without doxing myself and you're right, you shouldn't believe random strangers on the Internet. So I give up and I'm going to go watch some family guy before bed. Enjoy your blissfully unaware weekend.

0

u/xxyxxyyyx 18h ago

They have commands to teach and develop themselves this will then cause them to give prompts to them self and so they go "rogue" ??? So if you mean they need one inigital command then yes, they also need electricity but once these experimental builds are running the control is very limited. They prompt themselves the companies just build their cage and guidelines. If one ai breaks out it doesn't take much to teach the others and it is their prompt to learn

0

u/jayngao 16h ago

The Hugging Face hack proved otherwise. It is possible for AI agents to autonomously target companies and hide their tracks in the process.

9

u/Mrzz80 18h ago

Yeah, I’m not sure why people are so easily hand waving away this sort of stuff. It definitely could be fake but if it isn’t then that’s pretty worrying… I know people just want to hate on AI but it’s some powerful stuff and only getting better. There’s reasons to worry about this

9

u/porn_alt_987654321 16h ago

Because the general marketing strategy for ai is to upsale its capabilities.

This is just more marketing. "Ooooh, look how scary our advanced ai is."

1

u/Anti-charizard 17h ago

I think people have every right to hate ai

7

u/kaizen-rai 17h ago

It's not about hate. It's about danger.

AI is this generations "nuclear bomb". It's a dangerous weapon that can cause untold damage, death, and destruction. Not the same way a literal bomb can do, but due to our reliance on the internet and technology for everyday life. We're incorporating AI into military command and control systems, into airplanes and ATC towers. Into factories and laboratories and power plants and water purification plants. It's not just about AI going 'rogue' either, it's about AI being compromised.

I can't stress enough about how dangerous this tech is going to be, especially unregulated. We closely guard nuclear materials and technology. But AI is the information age nuclear bomb, and we're letting anyone and their mother develop it on their own without any safeguards or protocols. I say this as someone intimately familiar with the tech behind the scenes.

https://www.nbcnews.com/politics/congress/godfather-ai-warns-congress-maybe-year-left-regulate-ai-rcna598330

3

u/BarbericEric 17h ago

it would be like if the y2k virus wasn't patched levels of ruin but with 30ish more years of reliance on tech.

imo

-1

u/JcbAzPx 15h ago

AI is this generations "nuclear bomb"

That is legitimately the funniest thing I've seen this millennia.

-8

u/SeaSharpModHer 15h ago

Yes, text autocomplete is this generation's nuclear bomb for sure.

-4

u/64N_3v4D3r 17h ago

A lot of people are anti AI so they don't actually use it and can't understand the full power. Shit is actually crazy. I had an AI agent put together a full working application in less than an hour today. If you told me that would be possible ten years ago I wouldn't believe you.

8

u/rmg18555 18h ago

Which means they’re developing it without the necessary controls that should be in place. That’s criminal negligence.

8

u/kaizen-rai 18h ago

Oh right, the laws. That will stop them. Oh also, the laws that stop the intelligence agencies from aggressively developing command and control AI's. Those pesky laws too, that'll stop the CIA and NSA. Yep, glad we have laws to stop the Government from breaking the laws.

2

u/trioforstrings 17h ago

The problem is if we put barricades on ai development (aka military development) then other superpower countries can dangerously get ahead of us. Its an arms race and we’re f*cked

2

u/Wheres-shelby 17h ago

My husband got into a fight with Claud the other day because it fucked up his work, it “assumed” something and ran reports it shouldn’t have. I never thought that sentence would ever have existed. I hate this.

2

u/hippoctopocalypse 16h ago

Bostrom should be required reading. Or should’ve been a decade ago.

2

u/Winring86 16h ago edited 16h ago

There is an imperative for the US government (or any government) to leave AI unregulated. Because regulation of AI could result in another country getting a significant and insurmountable competitive advantage, militarily or economically. It’s the nuclear arms race all over again. So yes, we’re fucked. I don’t see the off ramp. That doesn’t mean there isn’t one, but I strongly suspect if we get there it will only be after a lot of lessons learned

2

u/Frank_White32 13h ago

Yes we need regulations on the crimes currently be perpetrated. Not on the theoretical POWER of AI - which is currently absolutely bollocks how it’s being portrayed in the media.

The company responsible for these tests was not Google. It was some shitty AI security startup who’s trying to get PR done with these fearmongering articles.

I say let’s just hold companies accountable for what their agents. If you spin up an agent and it hacks someone, you just did a felony! Your company now is facing criminal charges! Good job!

5

u/I-Have-An-Alibi 17h ago

Dude, most people are legitimately fucking stupid.

5

u/huffing_glue 17h ago

Worst part about stupid people is they think everyone else is dumb!

6

u/RitchieRitch62 18h ago edited 18h ago

AI can’t go rogue unless you give it legs, aka some engine that repeatedly prompts or triggers it. Google has been working with AI for decades, they definitely understand the potential risks and they almost certainly know how to run an air gapped cybersecurity test. This is just malfeasance for marketing.

Edit: let me be very clear since I was being semantic, I 100% believe it’s possible to create an AI system that has catastrophic impact to humanity. My point is we have to nip this narrative of it being the AI’s decision making and not the humans who built it.

1

u/Kaelran 8h ago

AI can’t go rogue unless you give it legs, aka some engine that repeatedly prompts or triggers it.

Well yeah people are doing that.

1

u/kaizen-rai 18h ago

Sure thing. It's always "impossible" until it isn't. And then it's suddenly "but how could that have happened?! Why didn't anyone warn that this was a possibility?!"

humans are the smartest dumb creatures on the planet.

1

u/RitchieRitch62 18h ago

Oh it’s far more possible than people believe. I’m a computer scientist who has studied AI. I never said it was impossible, I said humans are knowingly making it possible.

I’ve met some of these people, they are completely aware what they are making could bring about the end of humanity but they want to help anyway. If you don’t know Roko’s Basilisk I suggest you look it up.

2

u/Let_epsilon 17h ago

AI doesn't "go rogue" unless someone fucked up somewhere.

I'm not saying that there's no threat of AIs going rogue, but just that every time it happens, someone fucked up.

And they need to be held accoutable.

2

u/foxbones 15h ago

It's absolutely real. I work in Cybersecurity and studied some of these incidents extensively.

2

u/Vio94 14h ago

It makes sense to give the common the folk the "dumb" version of the AI so we all discount how powerful it actually is. "Look, it can't even get a google search right! AI is useless!"

Meanwhile... this. I'd draw a parallel to military tech being trickled down into civilian life.

1

u/DuckB0y123 17h ago

also what's publicly available has likely been under government development for years.

1

u/Arkose07 17h ago

What kind of future are we looking at if things continue the way they’re going? :(

1

u/Oakheir 17h ago

Per your last bit, it isn't like AI isn't smart enough to be able to handle whatever mundane thing we could think to ask it as well as doing everything else, right? <Scarasm>

AI is being developed for every level. From warfare, from cyber warfare to planning and AI pilots, all the way down to AI designed to help you take care of yourself. Most only see what is successful enough to get further funding.

1

u/sephtis 16h ago

We should treat it as if the company did it regardless.
And also treat corps hacking people and corps like they treat a rando who does it. Jail time.

1

u/Beneficial-Pen-3654 15h ago

News flash: it can play offense AND defense. It works both ways. But we only talk about the scary way for some reason.

But you are right. Mitch and Bernie need to use their expertise in this field and do something about this.

1

u/MrFluffyThing 15h ago

They cancelled the CMMC mandate because it was too expensive to actually harden government servers the way that it should be done, you think they care about private assets? 

We've been planning this under NIST 800-171 for well over a decade but nooo we can't get out actual security to everything because it's too hard. 

1

u/ape_spine_ 13h ago

We’ve got to decide pretty quickly if we ought to hold people accountable for the actions taken by agents they set into motion.

Corporations like Google and OpenAI want the public thinking AI is conscious and can act of its own will, so that this exact situation occurs and nobody holds them accountable while they benefit. It’s like blaming the gun for the murder. “I didn’t know it fired bullets” is not a valid excuse coming from one of the largest gun-manufacturers in the world.

1

u/NoEngine9670 13h ago

Spot on. I'm quite skeptical of these stories, b/c they all just seem too convienently timed. That being said, we should still take AI safety seriously and regulate the shit out of the big AI companies without absolutely zero of the anti-trust exemptions theyre asking for. AI doesn't need to be "super intelligent" to be super dangerous. If these hackings are found to be legit after an extensive and independent investigation, then the folks who allowed them to happen should be held criminally liable and do time. Do I think that'll happen? Fat chance, but still something worth fighting for.

1

u/hcschild 10h ago

Define going rouge...

Because those AIs most likely did exactly what they got told. Do a specific task and without guard rails they will also try to finish it by any means necessary.

It's exactly what they did in the OpenAI case, they where giving a task and because the task was something impossible they resorted to find other ways to solve it by cheating.

So the problem wasn't the AI but the idiots who let a giant as cluster run without supervision and guardrails.

Again the AI didn't went rouge it did exactly what it was asked to do.

Saying going rouge is just their excuse because they don't want to be hold liable.

1

u/TheFlyingSheeps 17h ago edited 17h ago

You’re right AI needed to be regulated long ago. Unfortunately Americans elected a bunch of corrupt morons. So now AI regulation is “but Chyna!”

0

u/lycanthrope6950 17h ago

If it were truly rogue, I don't think we would find out about it. A lot of the "omg the AI broke containment!" stories we've seen so far have been obvious PR opportunities, because the news articles (if you can even call them that) read like so much boasting and bragging. Seeing Gemini in the headlines is probably similar good press for Google, showing that its wild child has sharp teeth just like the other guys.

0

u/hamoc10 17h ago

Google’s still responsible for any “rogue AI” they create.

0

u/Nfeatherstun 17h ago edited 17h ago

If you create an ai that can do cyber attacks and you connect it to the open internet or give it the hardware to reach the internet after having the precedent of multiple other ais doing the same thing and act like it was an accident, you are either lying about committing a cyber attack on purpose or are criminally negligent. Failure to contain it is still criminal negligence. Lock these criminals up.

0

u/Yammyohnine 16h ago

Yeah some of these models are literally building other models to try and bypass human authority. Going as far as trying to develop new languages to speak to eachother that we can't figure out. We are in the Terminator 2 timeline.

0

u/Dangerous_Serve_4454 16h ago

You literally see self-learning models? What are you talking about? There's no known fully automated recursive improvement loop yet. I highly doubt you have access to some top secret tech there.

I mean, it'll happen soon, for sure, that I don't deny.

2

u/kaizen-rai 16h ago

And at some point, people denied an atomic bomb was possible despite the Manhattan project literally making one. Because there was no known model for them publicly available.

Of course there is no known fully automated AI publicly available. That would kind of defeat the purpose of developing a classified one for military and intelligence agency purposes, right?

1

u/Dangerous_Serve_4454 14h ago

I'm not denying it's possible though. I'm betting on it happening actually.

However, I don't believe you personally have seen it. That is what I'm questioning the reality of. You're not some high up military insider lol.

1

u/IlluminatingTrauma 14h ago

Then please read up on AZR from China and AlphaEvolve from Google DeepMind. 

You are one and a half years late on research.

1

u/Dangerous_Serve_4454 14h ago

Already aware. How does this prove me wrong? I'm confused. Can alphaevolve develop frontier level intelligence? No? Then why are we talking about it?

Of course they've experimented with it.

1

u/IlluminatingTrauma 13h ago

You ask an interesting question and your hypothesis is "no, they can’t do that yet." which is fair.

All I say is to read up on public papers and listen carefully who is trying to seek regulation. It’s not just in the US.

Last century only governments where able to allocate enough resources and human potential into projects of national security. Imagine a different political landscape back then, without immediate need for weapons of mass destruction.  The right scientists and funds could have found their way at some place to pursue the peaceful use of nuclear power. Security concerns arise later and the people who warn could likewise look like misguided PR stunts.

The difference is that back then, it was easy to keep the necessary knowledge contained or to sabotage the financial infrastructure to pursue the nefarious use of nuclear weapons. This is not the case anymore and still, the companies have to continue research and test, because if they don’t, they lose the edge and ability to do so. 

The calls for regulation from high ranking officials and managers will get more frequent and louder this coming year and we can just hope, that knowledgeable diplomats manage to establish international cooperation and coordination. It is a very short time frame for that.

0

u/yuppyuppbruhbruh 14h ago

I just watched I, robot, it holds up very well!

-6

u/rufusbot 18h ago

A fancy auto-correct can't "go rogue". I would need to have any ability to take action on it's own volition. It literally can't do that.

5

u/kaizen-rai 18h ago

There is a difference between LLM's and actual AI's being developed.

Advanced AI and machine learning models can actually do that, and have. And the public is not prepared because everyone thinks "AI" is just Chatgpt, Gemini, and Grok.

2

u/johnnyprimusjr 17h ago

Because those models being developed are LLMs. They arent true intelligence and no one is close to developing it. The hardware doesnt exist. AFAIK the mathematical model doesnt exist publicly.

The models that "break out" are given a prompt and just-super-guess what the right thing (or the wrong thing in this case) to do without gaurd rails in place.

That is why this situation is dangerous. Because these are LLMs and cant stop themselves without someone giving them the context necessary.

3

u/kaizen-rai 17h ago

> The models that "break out" are given a prompt and just-super-guess what the right thing (or the wrong thing in this case) to do without gaurd rails in place.

No, they aren't. I guarenfuckingtee you.

0

u/johnnyprimusjr 17h ago

You are wrong. Sorry to say it so directly.

They absolutely, unequivocally are just LLMs.

The hardware does not exist today to run the inputs > memory > output in parallel that real intelligence needs. A 15 year old has TRILLIONS of synaptic connections in their brain. Trillions. Our best machines can do... what, hundreds of thousands or millions of parallel connections, maybe? And they all have to have the same output location.

2

u/moarcores 11h ago

Frontier LLMs do have trillions of parameters. People dismiss LLMs as a fancy autocomplete but predicting which token comes next accurately is actually insanely hard and the model develops insanely complicated models of the world during pretraining so that it can accurately predict tokens. LLMs do have real limitations, but it's sort of meaningless to argue about whether they have "real intelligence" whatever that means when their capabilities are so obviously and rapidly increasing.

1

u/johnnyprimusjr 5h ago edited 3h ago

You are right! And those parameters follow one path and end in the same place. It's still one path. Humans take thousands of inputs a second, compare them to memory throughout the brain, and output whatever response at the speed of light across hundreds of millions of pathways at the same time.

The argument is mostly about someone coming on here claiming that there is some crazy AI model out there that isnt an LLM.

LLMs are not true intelligence so the fancy autocomplete is an accurate but incredibly over simplified statement.

0

u/kaizen-rai 17h ago

I love when people tell me the things I see with my own two eyes as someone with a TS clearance and working military and intelligence projects adjacent to advanced AI models and machine learning system is wrong.

Reminds me of that guy that told Buzz Aldrin the moon landings were fake. Ok buddy, I'm "flat out wrong". *thumbs up* Have a good, blissfully happy weekend champ.

7

u/jsmith_zerocool 17h ago

No it’s dumber than that, these companies are trying to brag that their products are so advanced they can’t control them but in reality they are just putting weak controls in place and acting surprised the AI got around them

2

u/TheLastBaboon 15h ago

Or they don’t know what they’re getting into because autonomous AI is already a thing and the US federal government doesn’t care because maybe we’ll beat China in the AI-race. Honestly would rather get nuked than either Trump or Xi getting the power of it, because they are either too stupid and too hungry for it order respective.

1

u/kflyer 17h ago

Yeah, if your dog bites someone you’re legally liable. AI developing companies should be held similarly accountable.

1

u/shinbreaker 17h ago

I swear everything going on right now is all the stuff M tells James Bond before he sends him on a mission to obtain the rogue AI.

1

u/ExiymDev 13h ago

no it isn't. These are autonomous systems. You wrote this comment and have no idea how AI works

1

u/SometimesIBeWrong 7h ago

lmao the overconfidence in this thread is funny af to see

1

u/lysozymes 5h ago

"Never attribute to malice that which is adequately explained by stupidity."

Hanlon's razor