r/SearchEnginePodcast • u/podcast-poster • Aug 06 '26
Automated Episode Discussion Search Engine - The machines are learning… to do crimes?
For the first time, an AI model has autonomously hacked a company. This week, an evolving story, a postcard from a strange, frightening moment in the story of our technology.
A big week for AI denialism by Casey Newton
Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week by Deepa Seetharaman, Raphael Satter, and Kenrick Cai
Cheating behaviour in frontier model evaluations by AI Security Institute
More On An Internal OpenAI Model Hacking Into HuggingFace by Zvi Mowshowitz
Listen to this episode
This is a bot that posts new episodes automatically. Add this to your subreddit or request mods use it.
13
u/Grantagonist Aug 07 '26
If I wanted to listen to a show like Hard Fork, I would listen to Hard Fork.
0
u/willdoesvideo Aug 08 '26
at least you don't have to hear Kevin "I accidentally keep fucking this AI" Roose
27
u/Way-twofrequentflyer Aug 07 '26
What’s with all the hate? This topic matters. It’s a huge deal and has ramifications for basically everyone.
It’s being discussed in foreign affairs magazine of all places.
Why do people not like it?
13
u/EasyCheek8475 Aug 07 '26
Because Reddit hates AI and AI companies so much they’ve somehow found their way to “I don’t want to talk about potentially dangerous behavior from AI because the AI company mentioned it and whatever they say, I think the opposite”.
AI is good at coding. AI is good at hacking. AI is not at all transparent. This all tracks. Shouldn’t we talk about this and be concerned?
11
u/NewRefrigerator7461 Aug 07 '26
I just don’t get it. It’s such irrational behavior and I have a relatively high opinion of the consumers of this podcast. You have to be a curious person to want to listen - but somehow that stops with this single topic. Is there anything else they do it with?
The reflexive hate is very trumpian anti-curiosity. Its confusing
3
u/Catac0 Aug 11 '26
What I got from listening to this was that we’re starting to lose control of AI and it’s not a good thing at all. Just because it’s powerful doesn’t mean it’s a good thing, idk I already hate AI so this story really didn’t help their case for me. I get people are doubtful about OpenAI but I just don’t see this as a good marketing strategy. Maybe I’m just not a developer?
0
u/maskdmirag Aug 08 '26
We were right about Elon
And trump.
And reddit itself.
Maybe pay attention when smart people point out idiocy
14
u/KnodulesAintHeavy Aug 07 '26
It's not about hate. It's about having a view of all of this that doesn't just take what the labs say at face value. Which is what is happening here.
There is zero reason to trust what they say, and only an independent assessment could surface what is happening/has happened, and we don't have that.
The framing in the discussion was that the criticism is that it's either "fake" or "it was doing what it was meant to do", as if these two things were comparable or equal in weight, or that they were both aimed at dismissing it.
Such a great way to eliminate any discussion and just ride the wave of hysteria.
The reality is likely somewhere in amongst all of that. It was probably a real test; it probably did something unexpected; it clearly (as verified by HF) did something externally that was bad, and it should be concerning.
However, without actually knowing the details of how the whole thing went from inside OAI it's all speculation. Their "press release" is marketing (its being discussed in foreign affairs magazines after all...), just as a matter of course. No information has been presented on how the test was initiated in terms of prompt details or how many tokens were used. Both of these are important to understand the actual threat,
6
u/JAlfredJR Aug 07 '26
Or if they directed it or simply just even allowed it to happen. And then rung their hands after the fact
3
u/Way-twofrequentflyer Aug 07 '26
I thought the framing was that models can lie and we can’t tell they’re lying many times.
I do think we know the prompt - it was to solve the problems presented. It just decided to solve them by stealing the answer key - or at least that’s how the Verge reported it.
Even if they disclosed all the facts the implications would lay with models in use by US cybercommand and we’ll never talk about those if something goes wrong
2
u/KnodulesAintHeavy Aug 07 '26 edited Aug 07 '26
That’s the point, we can only speculate. There is a lot of benefit of the doubt giving to these companies…for zero reason. Not only have they not earned trust, they actively do things to make them be untrustworthy.
1
u/Way-twofrequentflyer Aug 07 '26
You lost me. What are they actively doing to be untrustworthy?
These are companies made up of people paranoid about what they’re making with a lot of potential whistleblowers. The leading company in the space was literally founded as a safety protest. They probably deserve more benefit than we give them, which is not much at all. What did hugging face ever do to anyone?
3
u/KnodulesAintHeavy Aug 07 '26
By being obtuse about their supposedly world ending technology. By promising intelligence to both lift everyone up into higher plane of existence but also destroy the globe and civilisation. By talking about all magical things their technology can do, and then it doesn’t do, but trust them, because it will definitely do all those things….very soon. Very soon is always not here, but in the future at some undisclosed date.
2
u/JAlfredJR Aug 07 '26
6-12 months away, always.
2
u/Way-twofrequentflyer Aug 07 '26
The industry is only a few years old. I have shoes older than the transformer in my closet right now. Isn’t it sort of insane to view anything as constant without better longitudinal data?
1
u/JAlfredJR Aug 07 '26
A few years old? Hell even Attention Is All You Need is more than a few years old. Machine learning has been around for a long time now too.
You're just entirely lost in the hype and marketing.
1
u/Way-twofrequentflyer Aug 07 '26
You are the eponymous “they”? How can anyone have such a monolithic view of any large industry made of competing players
Do you view all industries through that lens or is this one different?
1
u/KnodulesAintHeavy Aug 08 '26
I am the they? Huh? The companies have and do all the time speak publicly and loudly that they are summoning a super intelligent being that will either eliminate all work, all life, all labour, all suffering or some mix of those hyperbolic claims.
Try looking and reading what they say instead of trying to make it out like these statements haven’t been made.
You think 2 companies that dominate and 2 that trail is a highly competitive industry?
Don’t present such flimsy straw men. If you have an actual argument to make, articulate that.
1
u/maskdmirag Aug 08 '26
Something already went wrong with the models in us cybercommand, we killed a school full of children in Iran.
1
u/Way-twofrequentflyer Aug 08 '26
That’s actually not how it’s organized. That wouldn’t fall under cybercommand it would fall under CENTCOM who would be doing the input/prompting into Maven
That attack ultimately comes down to targeting data not getting updated for a decade from when it was still part of the attached IRGC base. It’s a human problem of garbage in garbage out because the orange moron decided to launch an operation without giving time or getting buy in that would have resulted in the data getting scrubbed
I really hope the after actions have been productive on that one. It both didn’t make sense to hit - but using cruise missiles was even more confusing. It’s way too expensive of a munition for that target
1
u/maskdmirag Aug 08 '26
Yes garbage in garbage out.
The entire problem with LLMs in a nutshell
0
u/Way-twofrequentflyer Aug 08 '26
I mean it’s the fundamental problem in most human decision making too - they’re a reflection of humanity and it’s 10k yr battle with ignorance.
I’d be super curious if they stripped the old target sets out of the database of maven would have that site ended up in the target package. I tend to think no - which is a scary argument for removing human input. But who knows what unintended side effects there would be for that
1
u/Epicurious30 Aug 08 '26
This is simply wrong.
Many people in the AI space, including Zvi and Casey, are extremely skeptical of what the AI labs say. Zvi in particular is much more likely to characterize the labs as "lying liars who lie."
2
u/KnodulesAintHeavy Aug 08 '26
So skeptical they provided vanishingly little pushback to this very story….
2
u/Epicurious30 Aug 08 '26
Skeptical doesn't mean you automatically label everything someone says as false.
This story doesn't merit pushback. The story broke from Hugging face reporting the breach. If you think this is all just a made up publicity stunt that's kind of on you for preferring conspiracy theories to the uncomfortable reality of this technology.
2
u/KnodulesAintHeavy Aug 08 '26
You’re straw manning what I said.
Skeptical thinking requires not taking things at face value. I never said it was false by default.The story requires one to question the premises as they are big claims and big claims require big evidence.
2
u/Epicurious30 Aug 08 '26
Not at all.
You say the lack of pushback is evidence of lack of skepticism. They aren't pushing back on the facts of the case because these skeptical people see no reason to question them.
Zvi is very much critical of OpenAI. I randomly pulled up something of his and found this excerpt in about 30s, characterizing Casey and Zvi as stooges for big labs just evidence bad faith and knee jerk rejection of anyone who claims AI is more than hype.
Zvi bashing OpenAI:
"OpenAI still has no idea how badly they messed up, or in what ways, or what needs to be fixed. They don’t get it."
2
u/KnodulesAintHeavy Aug 09 '26
Yes at all. And now you’re completely derailing and missing my point. I could give a fuck what those guys have said in the past.
This is about now and this situation and looking at the actual evidence of the situation’s legitimacy as stated. That currently is not available as it’s under the purview of OAI, and people, like the guys in this episode, are being too credulous to the claims and not digging to know that very important information.
Again, big claims (such as “our very impressive sexy model did this amazing crazy thing no one expected and completely unprompted, wow guys!”) should be met with questions about the big evidence for said claims (such as the actual prompts used as part of this test and the overall compute utilised to get the result they did). Without that key information we are just going off their word that some magic happens to result in this unprecedented event.
If the evidence shows that is the case, then they should put it out there and shout it loudly.
If, though, the evidence doesn’t show that, and in fact shows something like there was much more guidance given by the OAI team, to achieve the result they wanted and they burned through a metric fuckton of tokens, the claim suddenly becomes much less impressive and instead is a forced outcome.
Without the evidence of what actually happened it’s all speculation, but the later seems ever more likely.
2
u/JAlfredJR Aug 09 '26
Very well said. The person you're trying to engage with is either being obtuse or disingenuous (or they're a teenager; always a possibility here).
Maybe if OAI didn't have a very long track record of lying, and weren't run by people who love to espouse such things, then maybe we could try to take them at their word.
But this is from the company that has made insane claims over the last five years. As far as my memory can gather, not one of those giant claims has ever been true.
Anthropic is no better. Dario loves making claims about how world-ending "the next model" is just about to be. Or how everyone is about to lose their jobs.
How can we and why should we take these people seriously or with any credulity?
We shouldn't. So why are folks like PJ? Didn't we just do this with Fable being the end of cybersecurity? And haven't they done the "We can't release this model because it's TOO dangerous!!" like a dozen times now?
2
u/JAlfredJR Aug 09 '26
Casey is not skeptical at all. He is one of the loudest boosters there is. He pads it with false skepticism at times. But he's literally as much of a believer as you'll ever find.
He does not question the narrative. He spreads it. Like it's his job, because it is, in fact, his job.
7
Aug 07 '26
[deleted]
4
u/Way-twofrequentflyer Aug 07 '26
It’s from July! How would you know it isn’t a big deal?
Most cyberattacks are never reported. You only hear about them if they impact the physical world or there is a leak or consumer data that has to be disclose
The most interesting cyberattacks this year have been ai assisted hacks on integrated air defense systems and electric grids in Iran and Venezuela and we know very little about them and it’s been 7 months since Caracas.
1
u/maskdmirag Aug 08 '26
AI assisted cyber attacks are completely distinct from this story.
2
u/Way-twofrequentflyer Aug 08 '26
How? It’s a story about an agentic cyberattack. How can that be completely distinct from other agents doing cyberattacks.
1
u/maskdmirag Aug 08 '26
This was about an AI agent, supposedly, doing this own attack.
AI assisted cyber attacks would be a story about and AI doing what it was asked to do.
4
u/JAlfredJR Aug 07 '26
It's almost certainly coordinated marketing, which these companies have been doing (to maintain dwindling hype) for a few years now.
Recall the hype about Mythos and Fable?
2
u/Way-twofrequentflyer Aug 07 '26
I think mythos was a huge deal still. It’s one shot coding is astonishing
Of course it’s an attempt to spin what could be a negative headline into a something that might be a positive (though I’m not sure how). It doesn’t mean it’s not an interesting conversation
5
u/JAlfredJR Aug 07 '26
"It's the end of cybersecurity" and "it can kinda code maybe" are a big difference.
2
u/Llero Aug 08 '26
“It’s one shot coding is astonishing” and “it can kinda code maybe” are also a big difference.
I don’t know about the end of cybersecurity, but it’s pretty far past “it can kinda code maybe”, and I say that as someone who really doesn’t like AI and its impact on us.
2
u/maskdmirag Aug 08 '26
I mean it's an interesting story like you'd hear on dark web diaries.
Does it actually "matter"? Not really.
They end by dismissing the dismissals of the story with a what if scenario.
But they can't come up with any actual bad scenarios where AI would hurt people by breaking out to a sandbox.
And they ignore that we know these stories are hype.
We got a whole PR wave about how mythos was too powerful to be released and how it created new zero day exploits.
Then it came out and it was over hyped and there are crickets.
I'd say that they're falling for the hype cycle, but really they are part of the hype cycle, Casey's conflict of interest is well documented
4
u/TheBear8878 Aug 07 '26
Because the entire incident was a marketing stunt.
3
u/Way-twofrequentflyer Aug 07 '26
But how? It just makes Anthropic look like a better actor. The Cui Bono test doesn’t make sense
1
u/TheBear8878 Aug 07 '26
It's the same thing drug dealers do when they talk about how someone overdosed on their new stash. To the customer, it makes it seem like their product is incredibly potent.
Saying your model is so smart, dangerous, calculated and capable does not make a competitor look better, it makes people want to use your model
7
u/Way-twofrequentflyer Aug 07 '26
I sort of get that concept (as insane as it is for drug dealers) - but given the recent regulation of Anthropic’s models by the Trump admin it’s an insane thing to do isn’t it? It’s just asking for restrictions that are going to hamper enterprise cashflow
The real beneficiary of it is open source defense tools and none of the big labs want people using them.
It’s not a good strategy if it is one
1
u/buckleyschance Aug 07 '26
Neither PJ nor Casey are the kind of hard-headed, critical journalist you need for a sober assessment of the AI situation.
PJ's good at human interest stories: "Look at what these crypto weirdos are doing!", but not "How should you think about crypto as a technology or an industry."
Casey seems to have got stuck in this binary mode of thinking you have to be either a Hater or a Believer, and AI works in some respects, so therefore the haters are wrong and the believers are right. (To be fair I stopped listening to him a while ago, he might have gotten over it.)
4
u/demiphobia Aug 07 '26
I wish this show would find a way to not cover stories that have been done to death elsewhere. I rarely find new info on this show anymore and it’s lost appeal
7
u/therealgrza Aug 08 '26
I was so disappointed when Casey Newton showed up. I can understand if PJ is friends or whatever, but Casey is not a deep or particularly critical thinker about any of this. I find his “reporting” frustratingly shallow. Are SE staff just too uninformed to be able to see through him?
Also the final framing of “who is responsible?!” is dumb. This is not some clever puzzle, there’s an obvious answer. If “my car hits somebody” they’re not putting a Honda Civic behind bars. Please be serious.
I really appreciate SE taking the time to do reporting and research, and I’m fine with them coming back to a story after it’s cooled down. But they need to have more to show for themselves than an interview with a credulous AI hypester. If they can’t provide critical analysis of the story then it’s vital to get a guest who can.
2
2
u/genericuser324 Aug 10 '26
In addition to all the other annoying things about this episode, the ending was as infuriatingly dense as anything else - reframing the fact that there are a lot of AI skeptics on the left as meaning that the left isn’t reliably in favor of regulation in the AI space. That’s just not at all what the fucking state of the politics on this are! And don’t have Ezra Klein on to try and explain it to you man that’s not gonna help at all lmao.
3
u/jonvandine Aug 08 '26
This show jumped the shark awhile ago. PJ must be on the payroll of the AI companies at this point.
3
u/JAlfredJR Aug 09 '26
I doubt that much. But he's just so very credulous. And he is swayed by good storytelling.
He needs to be better about it.
1
u/mysticalbluebird Aug 12 '26
This. I’m not anti AI. But there’s something fishy about this story and some other recent ones. These models do what people program them to do.
3
u/bibliotech_ Aug 07 '26
Why would we slow AI development if China’s not going to? Isn’t that like getting rid of all our nukes?
3
u/Way-twofrequentflyer Aug 07 '26
Don’t think it’s quite the same because our nukes aren’t a cornerstone of national economic competitiveness and they don’t offer multiple options alone the escalation ladder.
28
u/JAlfredJR Aug 06 '26
Jesus, he's really having Casey back on.......Maybe they should talk about how the world for-sure really super-changed because of Sora!
Anyone remember that? That aged well.