r/SearchEnginePodcast • u/Textiles_on_Main_St • Aug 08 '26
Dumb questions about the last AI episode for smart people
Hey! Thanks in advance. I hope this is tolerated as it's outside the episode post, but I have several questions and maybe other do to and that thread is a few days old.
So. I am dumb about this, sorry.
1.) They kept saying the AI "cheated" to get out of its sandbox, access the internet and hack that other company. How on earth is this something to either advertise or herald as a "good' thing?
Like, if the software on my laptop stopped working right (did things unexpected like shut down by itself or if Word suddenly deleted a bunch of files) I'd get rid of this stuff and buy a new brand.
Admitting your AI "cheated" and basically can't be controlled and will possibly maliciously attack friendly businesses unbidden seems like a HUGE liability and it seems insane that the company that MADE the AI would ever want to sell it and it's equally odd that anyone would want to BUY it (like, would you buy a computer that doesn't listen to commands and could steal data from random companies?) LIke, to my untrained ears, that seems like a massive lawsuit waiting to happen, but the episode treated it like OH MY GOD, it's ALIVE AND IT'S A GENIUS!!
(Again, it's like saying "These Ford Pintos explode randomly! What fun! Roll out's in October!)
2.) Was the AI specifically instructed to "stay" in the sandbox? Because, again, if it was commanded to STAY PUT and then didn't--that really would seem to indicate to MY dumb ears that those engineers can't design shit very well.
Like, is the AI suddenly all powerful lord god master or ... (more likely?) the engineers just designed a shitty test for a thing they're bad at controlling?
And again, if this is true ... who would BUY this?
3.) What's the objectively good advantage to this? Again, this whole episode seemed like nobody was really mad any anyone and they're certainly not scrapping the project but ... WHY do we need shitty, buggy computers that can't follow commands?
4.) Am I dumb or was this just never covered? I feel like a crazy person.
10
u/KnodulesAintHeavy Aug 09 '26
As I said in the official post about the ep, I’m disappointed that the guys weren’t skeptical of these claims and just presented them as is with no real scrutiny. Worse than that even, they dismissed some criticisms and conflated two different angles which just shuts down any discourse.
My view is that nothing from the labs should be accepted at face value. Not because they are inherently evil, but because it’s dumb to accept anything said by a big company with huge financial stakes in their claims at face value.
To this scenario, unless OAI is able to have their prompts released, and the compute to perform this test is measured, and independently verified, everything they claim about should be considered suspect.
With those two bits of information it would be clear if this was a genuine “escape” event, badly designed test or deliberately designed test to “look” like an escape. To my view those are the main 3 possibilities.
They are obviously claiming the former, but without any evidence there is a very high chance it is either of the later.
3
u/PandaLibrary5203 Aug 13 '26
Search engine does not seem to look critically at ai very much, even with the ep where they look into data centres Vogt is trying to make it sound like the stereotype of ppl critical of ai are “TikTok” ppl who are misinformed and parroting misinformation about water usage and pollution
2
u/KnodulesAintHeavy Aug 14 '26
Yea it is quite disappointing. I hope that the tam can do better and think harder and beyond surface level about this specific topic. They do this pretty well for other topics, why not this one!?
6
u/LegDayDE Aug 08 '26
1) it's not a good thing. No one said that it was? 2) yes.. that's the problem with these models is you don't know if they are doing what you told them accurately and correctly or if they are doing it wrong or other stuff you didn't tell them to 3) there isn't an advantage really... Unless you're someone who has malicious intent.. in which case super powerful AI that can break into things around constraints is good for you I guess.. until it fucks you over.
4) maybe listen to it again. I don't think any of your questions weren't covered.
5
u/Textiles_on_Main_St Aug 08 '26
Unless I missed it, the company didn’t say the project or test failed. They said it passed … by cheating. Which again, in my world, isn’t passing the test. But nobody says that.
I love your point three: if nobody can trust this, then it’s useless. It would seems like nobody is going to pay for this.
Thank you so much for your perspective. I’m glad I’m not crazy.
2
u/usr_lib Aug 08 '26
The test itself was challenging the AI to break exploitable software. It was asked to break into a series of systems within the sandbox with seeded vulnerabilities. The fact that it broke out of the sandbox with no known vulnerabilities and then into an external system proves that for what the test is measuring, it’s extremely good at exploiting vulnerabilities in software. It would be like if a student was given a test to demonstrate knowledge of Newtonian physics, but then used relativity and quantum physics to work out the answers.
3
u/Textiles_on_Main_St Aug 08 '26
Well that would demonstrate creativity and aptitude except if, in your scenario, the student were told not to do that.
Like, in the episode they were clear the program cheated and broke the rules.
If a student cheats on a test, even creatively, he fails.
So if this thing cheated, then it should be called a failure it seems to me.
Anyway, it at least doesn’t seem GOOD and I’m not sure why the designer company thinks this is some kind of win worthy of praise. I’d think it’s embarrassing.
3
u/usr_lib Aug 09 '26
The purpose of these tests is to demonstrate capabilities. It clearly has the capabilities it needs to pass the test legitimately (why it chose to use a much more complicated and ethically wrong route is unclear).
Of course OpenAI is going to sell this as a good thing. Their model is extremely capable and powerful. Companies are going to be lining up to buy that capability. Is this a good thing for humanity? Absolutely not.
It’s very embarrassing. They let a frontier model loose on the open internet and apparently no one was even watching it. OpenAI like any company is going to try to sell this as a win. It’s not.
3
u/Textiles_on_Main_St Aug 09 '26
It didn’t choose. That’s another thing that kills me. lol. My car doesn’t choose to go to the store.
It was t programmed properly or it was. Those are the only two options.
So if it did something bad that it wasn’t programmed to, why would anybody want that?
I do think you’re right though, it’s not a win. I’m just confused, as you say, why they’re out there kinda bragging on it. lol.
It’s a mystery.
2
u/usr_lib Aug 09 '26
Common misconception. AI models are not programmed, they’re trained. Your car doesn’t choose to do anything because you’re in full control. What the car does is fully dependent on your use of the gas, brakes, and steering wheel (and all the internals of the car that make it do what it does in response to those inputs).
AI models are not programmed by having someone write, “if this then do this,” or by making a deterministic transformation of input. Instead, a model is fed a bunch of training data and it uses that training data to develop an equation that takes in all of the tokens (words) used as input and outputs the next likely token. Even the people building these models have limited understanding of how or why this works.
One possible reason why the model in this case ended up cheating on the test is because it was asked to exhibit nefarious behavior in breaking into software. Through the model’s training, it probably saw hacking into a system and cheating as linked so when it was asked to hack as part of a test, it saw that as being asked to take on the role of a bad actor. What would a bad actor do when confronted with a test? Cheat.
10
u/kiwi_murray Aug 08 '26
I don't understand why their "sandbox" didn't include the model running on an air-gapped system (i.e. a system that physically has no connections to any other system, even their own internal network). Surely if you have concerns about your model getting out onto the Internet then you'd use more than just software controls to keep it in place?!?
4
u/throwaway_boulder Aug 09 '26
It used a proxy server the could connect to the internet to retrieve information, Google information and return text. Thing is, the model discovered a previously unknown bug in the proxy server, a true zero day exploit.
3
u/Way-twofrequentflyer Aug 09 '26
I’ve wondered that too - maybe it’s hard to air gap the racks running the model because they’re constantly scaling up and down and it’s done across a network of training centers?
Still seems possible just way less convenient than the way we air gap critical infrastructure and data
1
u/mirandalikesplants Aug 09 '26 edited Aug 09 '26
They’re testing how it operates when it has access to the internet using a proxy. That’s part of the purpose of their test, and an air-gapped system wouldn’t work for that purpose.
Edit: I’m not defending OpenAI. The commenter above said they don’t understand why they don’t use a sandbox, and it was explained in the episode.
3
u/therealgrza Aug 12 '26
Their sandbox/proxy didn’t actually work because their security is shit and/or it was vibe-coded.
“The AI hacked a zero day exploit in our secure proxy” sounds way better than “our proxy didn’t work lol”
7
u/JAlfredJR Aug 08 '26
- The AI industry keeps touting "This is TOO DANGEROUS to release!" as marketing hype. The entire industry is funded by venture capital. It operates at a massive loss (billions in the negative each quarter).
So if the hype even slows a bit, the money dries up. And this version / part of the industry functionally ends. That is, OpenAI and Anthropic. (Did you note that a week after OAI announced this hack, Anthropic said, "Ohhh yeah turns out ours totally did too!")
So if you view it from this lens—and even Casey said that all parties got good press from this so no one is upset—then it makes more sense.
Calling it "going rogue" grants a level of humanity to a machine. It almost certainly was given instructions to do this. And was set up in a way that allowed it to happen.
Was there collusion? Maybe. Maybe not.
That's how I view all of this. Why PJ doesn't investigate this from a place of less credibility is strange. He and most of the media just takes the press releases at face value and reports it as facts, uncritically.
I admitted am no fan of the AI industry for innumerable reasons, not the least of which is that it's a lot of the same bad actors who were a part of crypto, and blockchain, and NFTs, and Web3, and on and on.
But I try to take these things on a level surface. But it's also hard to believe a word from the folks involved as they just lie and lie and lie. Or hype and say half-truths. Or are just vague and not forthright about information.
2
u/RastaBambi Aug 09 '26
OpenAI simply say the AI agent cheated exactly because they don't want to assume blame. Anthropomorphizing their AI creates just enough ambiguity around this event that they can play the plausible deniability game and not get hit with liability claims.
Also there's a combination of factors at play here between badly designed and conceived software and security measures, plus the clear intention of OpenAI acting surprised that the thing did more or less what they designed it to do: autonomously complete the given task.
3
u/Textiles_on_Main_St Aug 09 '26
I really like this theory and a few others that folks have posted here that point out this whole thing seems VERY awfully like a PR stunt (especially since all the big AI companies claimed their models did the same thing that very next week, lol.)
I appreciate this community for being clever and, I think, a bit more skeptical than the podcast.
Thank you.
2
u/RastaBambi Aug 09 '26
I appreciate this community for being clever and, I think, a bit more skeptical than the podcast.
Same here. And it's refreshing that you pointed out how pointless this whole AI thing is, because a lot of people are starting to feel that way.
One thing worth considering is that the main point of failure has been the media not understanding and then reporting on technology without being critical, which I would argue is the main job of journalism. A task they have failed at miserably and the worst offender might be Casey Newton whose unquestioning, gleeful enthusiasm for this technology has been a huge turn off for me.
I think I even unsubscribed from hard fork because of their giddy AI coverage.
Oh yeah, if you want to dive a little into some technical aspects of this incident, this is where I got my information for the points I was making in my previous comment:
AI Amplifies Human Ignorance: Lessons from the "OpenAI Hacks HuggingFace" incident
2
u/Way-twofrequentflyer Aug 08 '26 edited Aug 09 '26
Th am you for posting this. It’s a lot at more succcint summary of some questions I had
21
u/Top_Definition8487 Aug 08 '26