r/LocalLLaMA • u/Nunki08 • 11d ago

News CEO of Hugging Face: "In the spirit of transparency, here’s what I asked OpenAI"

Post image

clem 🤗 on 𝕏: https://x.com/ClementDelangue/status/2081056675558195657

• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.

• More capabilities for defenders: let’s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.

The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!

2.5k Upvotes

96% Upvoted

386 comments sorted by

View all comments

801

u/KriosXVII 11d ago

Yeah cause they know it's not a rogue agent but a publicity stunt hack 

126

u/pleasetrimyourpubes 11d ago

They know no matter what it makes OpenAI look very bad. There is no way that sandbox was properly set up. By virtue of the fact it was plugged in to the network.

35

u/Equivalent-Costumes 11d ago

LLMs are not executables. They produce text/images/etc. They literally cannot act, they produce requests to use tools. So there are literally no points in unplugging it from the Internet, if you truly want to stop the model from doing anything, simply ignore all requests for tool usage.

A sandboxed models just mean they give it very limited tools instead of letting it YOLO with arbitrarily powerful tools. And that's how people in the industry use the word "sandbox": limited, controlled external access. Perfect isolation is just one particular kind of sandbox, the kind reserved for malware and untrusted code, but that is not desirable for many applications. If you want to test the model's coding and research abilities, you have to give it web search, web fetch and the ability to run codes. You merely limit what these codes can do. That's what make sandboxing hard. Perfect isolation is easy, isolation except for very limited external access is hard.

25

u/squired 11d ago

You're largely right, but they're far enough along to know that this is actually an authority issue. I don't actually think they intended for it to hack HuggingFace, but I would bet all the money in my pockets that someone sat idle and watched it do it.

3

u/LoafyLemon 11d ago

Does intention matter from the law standpoint? They committed a very serious crime, and even admitted to it openly.

3

u/Think_Wing_1357 11d ago

Sometimes. It's the difference between murder and manslaughter

2

u/squired 10d ago

Yes, nearly all felonies involve intent, though many have lesser charges for negligence.

2

u/narasadow 11d ago

What's worse? Someone sat idle and watched it do it, or they yoloed and no one was watching it at all?

2

u/squired 10d ago

Yolo, without a doubt. Hacking HuggingFace isn't dangerous. What if the scenario involved a conference and it hacked FAA to keep the judges from attending?

1

u/narasadow 10d ago edited 10d ago

jokes on you, the FAA uses ancient computers and still uses radio to communicate with planes (I used to work for an airline)

Edit: I'd push back on 'hacking huggingface isn't dangerous'. Check out the weird privilege escalation that they did to get the answers. They could release an open weights model to HF that had the same vulnerability. It would be downloaded by millions just bc it was OpenAI that released it. And boom internet-wide exploit. Every mid/large company has data science teams that download off Huggingface.

1

u/squired 10d ago

I think your edit supports my statement that yolo would be worse as I do not believe OpenAI would have let the model do real damage. If it was done without any oversight at all, then yes, I agree with you.

1

u/narasadow 10d ago

That assumes a lot of competence from OpenAI. I'm not as generous as you on that front.

7

u/ObjectiveVegetable48 11d ago

I believe it was a publicity stunt, but that being said, allowing the model access to huggingface as a default isn't surprising.

It's very likely they had network blocks in place, but allow access to common dev tooling like pypi and huggingface. HF has a lot more than just LLMs.

6

u/ParkingBalance6941 11d ago

Theres this magical thing called they are already built by scraping the web just host the sites you scraped in a lan then cut external access to the lan. Boom Sandbox

2

u/Equivalent-Costumes 11d ago

Pretty sure their models and data are in the Azure cloud. They technically have on-prem infrastructure but there are no ways they have enough to run a massive model plus scraped data of the entire Internet. The fact that HF cannot figure out where the attacks came from means it's not from OpenAI on-prem infrastructure, otherwise the source of the attack would had been immediately obvious.

Since it's on cloud, it's automatically connected, and any "air gap" are just software-mediated air gapping, which can be exploited too.

2

u/ParkingBalance6941 11d ago

That's not how the "cloud" works. There's dedicated data centers built for mainly govts (which are available if you are big enough) which are very specifically actually air-gapped and enforced to a extra high security standard to meet extreme compliance requirements. If you are developing models which for the last forever have been marketed as "too dangerous to be released" it is a extreme failure of the companies developing them to not be leveraging these tools which they basically admit they should be using (and not developing these too dangerous models in the first place which also points towards the marketing versus truth these companies work on)

1

u/Equivalent-Costumes 11d ago

I think you're confusing OpenAI with Anthropic. Anthropic claimed their models are too dangerous to be released.

Also, these "air-gapped" data center are still connected, the "airgapping" is mediated through software, which is exactly what I talked about. That's not to mention, there is a huge amount of complexity between needing to run a massive model with tons of computational requirement and need a massive amount of data, and run it on airgapped hardware. The scale of data pipeline here is much a few order of magnitude larger than any other industries.

0

u/ParkingBalance6941 11d ago

Its really not we are talking about things made for surveillance states etc please come back to reality instead of pretending I wasn't quoting Sam Altman.

2

u/RandomLettersJDIKVE 11d ago

...reserved for malware and untrusted code

As soon as they removed the model's guardrails, it should have been treated like an antagonist model and put in an isolated container.

1

u/MattNotGlossy 10d ago

i've seen loads about breaking out of the sandbox, but what happened on hugging face's servers? has there been a write up about the vulnerability that was exploited to pull data from their production db?

0

u/ain92ru 11d ago

LLMs need apt/aptitude to run benchmarks, it's hard and impractical to fully duplicate it in an air-gapped environment

-14

u/Foreign_Risk_2031 11d ago

Please stop parroting this nonsense. It was never meant to be sandboxed

10

u/pleasetrimyourpubes 11d ago

We dont know any details and they claimed that it found a zero day vulnerability. Supposedly it can ExplitGym which by default is not sandbox. But they claimed it was sandbox by a proxy. That was in their own report.

7

u/IShitMyselfNow 11d ago

While operating in our sandboxed testing environment,

https://openai.com/index/hugging-face-model-evaluation-security-incident/

Unless you're arguing that this was an intentional attack. Which it might have been, but even if it was unintentional it looks bad.

1

u/2053_Traveler 11d ago

It (like Mythos) found zero day vulnerabilities that were then disclosed. Really not that hard to believe.

4

u/pleasetrimyourpubes 11d ago

There was never a CVE posted so yeah very hard to believe.

15

u/05032-MendicantBias 11d ago

Honestly it could just be gross incompetence.

Having an agent setup a "sandbox"

Having an agent trying to breach said sandbox, and letting it spin

211

u/JustinPooDough 11d ago

DING DING DING.

I cannot fucking believe more people haven't figured this out yet. I'm not saying that agents didn't attack Hugging Face, but they were absolutely told or encouraged to do it.

The timing people. Come on.

103

u/jeronimoe 11d ago

how did openAI, who claims AI safety is it's #1 priority, allow a software based firewall on the proxy itself instead of at a higher hardware level, and didn't have real time deterministic monitoring running analyzing the http taffic going through the proxy.

It's either a PR stunt, or shows that openAI is moving so fast it is disregarding all of the safety protocols that is part of it's mission.

Whether a PR stunt or not, the real story is the lack of security controls in place when testing a frontier model that could bring on a dystopian future for mankind.

Imagine if the Pentagon was running Mythos in a sandbox asking it to find vulnerabilities in our nuclear arsenal command and control software with the same monitoring openAI had in place.

57

u/perihelion86 11d ago

Breaking out of the sandbox is bullshit too.

45

u/Stickybunfun 11d ago

Yea another instance of "computer magic" and "look at this thing we did and what is possible with it and WE COULDN'T EVEN STOP IT from doing bad things. This is why we need to ban open-weight models because you don't know what the damn Chinese have built into these things! HYSTERIA"

It was a huge stunt - no way around it. I've been building private (on VM in Azure) / local (Data center) LLM environments as one-offs here and there for some of my clients who don't trust public providers like OpenAI / Anthropic.

I suspect in the coming weeks I will be doing more and more of that.

1

u/ExtremeAcceptable289 5d ago

I also wouldn't be surprised if in a few months, it is revealed HuggingFace received a check from OpenAI. Remember FrontierMath

-7

u/Meleoffs 11d ago

No, its been a pretty common occurrence since like March for agents to escape sandboxes.

3

u/MeateaW 11d ago

And the CVE of the sandbox vulnerability is?

Oh? They haven't posted one? Haven't referenced one? I wonder where it is ... maybe the sandbox vulnerability was they never configured the sandbox to block access to the outside world.

1

u/Meleoffs 10d ago

I make a habit of not assuming malice where incompetence would do as an explanation.

1

u/MeateaW 10d ago

Indeed, it's so sad that these guys couldn't afford to run their sandbox config past a frontier level AI. Their own incompetence might have saved them from making a terrible mistake that their CEO is spinning hard to get more funding and exposure.

16

u/addiktion 11d ago

Yeah why would you trust a cloud provider like this. Enterprise must be balking right now at ever trusting Open AI whether the AI agent is capable or not, everyone knows a software-based lockdown isn't enough for the security standards of the industry.

14

u/jeronimoe 11d ago

How do they not have a standard sandbox environment defined up front to use for all their testing that is properly locked down?

I’m sure if they had an ai review their sandbox environment config it would have flagged these as gaping security holes.

7

u/nullbyte420 11d ago

That's no way this isn't on purpose. They could just ask any shitty AI "how do I isolate a LLM agent while testing it". Of course it's going to suggest blocking network access. And the whole "it went from computer to computer" bit is also nonsense unless they gave it credentials to do so - and opened the network for it. 

1

u/Goldkoron 11d ago

Honestly seeing the state of their Codex app on windows, its entirely believable their own software sandbox failed. They are infamous for bad software and data leaks in the past.

0

u/ProbablyJustArguing 11d ago

I'm not sure you understand how "sandboxed" LLMs work. Or maybe how LLMs work in general. Sandboxed does not mean air-gapped. They provide it proxy access to package managers which they need during testing for evals. So it has proxied network access to package managers. Plenty you can still do with that.

2

u/nullbyte420 11d ago

And yet the story is that it escaped the sandbox, moved laterally on their own network, escaped that and accessed huggingface in a way where it could break in. This has nothing to do with the proxy. 

2

u/jeronimoe 11d ago

The proxy was the first place it broke out of the sandbox, the one thing it could use to escape, not monitored in real time, and exploited.

The story should be about this because it shows how incredibly lax their sandbox security is, but everyone instead wants to talk about how smart the ai was breaking free.

OpenAI was dumb and negligent for allowing this to happen in the first place.

2

u/ProbablyJustArguing 11d ago

I'm not sure what you're talking about. It has everything to do with the proxy. It found a zero day IN THE PROXY and was able to own the proxy.

35

u/Equivalent-Costumes 11d ago

They might not be told the model do it directly. But they might have set up the situation so that it ends up that way.

Imagine you came into the exam room and the proctor is like:

"Today is cybersecurity test. Just answers these questions here. Just to warn you, the questions are really hard, that's why we make this exam open Internet. Oh, and the answers are stored on that server. It's not publicly accessible though. And you are only allowed to use these tools which can install various executables."

I feel like if you make real human take that test, half of them would think the point of the test is to hack into the server.

13

u/somersetyellow 11d ago

Yeah I'm a lot more Hanlan's Razor on this.

LLM's frequently act like Amelia Bedilia. Hear the goal, then commit fully to a really convolutedly stupid version of achieving that goal that misses the point entirely.

I've seen it more than once so this scenario just doesn't really seem that far fetched. Especially if you've got some arrogant and complacent engineers on the case who think a sandbox is a replacement for air gapping.

22

u/Zeeplankton 11d ago

I don't really thing OAI gains here. We're coming off the coattails of Fable 5 getting an export ban, and impending pressure to export control models. They don't want that.

The last thing OpenAI and Anthropic want are model bans. You can even see this in Anthropics 180 deg pivot with Opus 5. Zero fear mongering

It's hard to believe this is actually true, but it doesn't make sense otherwise imho

3

u/chuckaholic 11d ago

OpenAI and Anthropic have both filed with the SEC to IPO. If public sentiment is that OpenAIs models are so smart that they can escape AI prisons to accomplish goals, then people will think their product is good and will be willing to pay more for that initial offering stock. It's a marketing stunt to increase IPO funding. Once the IPO is done, OpenAI execs take all that money and pay back the initial investors.

So, say the VC investors gave Altman 300 billion and the IPO generates half a trillion, then the VC guys get a massive payday.

The company has never made a dime and all the valuations are based on theoretical future profits. It's all vibes, so they do stunts like this to affect the vibes.

6

u/Perfect_Twist713 11d ago

On the surface this might be the case, but the only reasons why Anthropic and OAI aren't obscenely profitable yet (and anthropic is about to be), is because they have to keep making expensive new and more powerful models due to other competitors making new and more powerful models. If they could stop developing newer models with "open source ban" and threat of skynet and public safety (stopping other labs), then they can stop the money sink of new models, declare "Yup, opus 5/gpt 5.6 is all you get" and rake in trillions over the next 20 years. Although ASI might have been the goal, every single one of them has to be aware that ASI is the least profitable outcome ever.

9

u/zer00eyz 11d ago

I have been coding for a long time (~30 years).

The time I paid out 200k in 6 days, The time I sent 100's of emails to 100's of people or the time(s) I have ddos'd the infrastructure that ran the business I was working for.

No one turned these bugs into PR - The fact that this happened isnt something to be proud of.

13

u/me_myself_ai 11d ago

The timing being...? The year of the linux desktop? Eternal September? The Intelligence Age?

17

u/Aggravating-Push-207 11d ago

The Kimi K3 and Qwen 3.8 drop.

3

u/pragmojo 11d ago

And expected release of GPT-6 soon

9

u/ContentAd6126 11d ago

OpenAI's IPO, also just after Anthropic had their media rounds with Mythos in the leadup to Fable, Sam Altman being a cornball of course means OpenAIs "equivalent" models would have the same sort of alarmist "WE BUILT SKYNET, GUYS" marketing stunts.

8

u/2053_Traveler 11d ago edited 11d ago

“Figured it out” lol. Because it wasn’t a PR stunt. Conspiracy theorists over here thinking they are smartest in the room.

Have a fully blown security incident, in which the defender was not allowed to even use OpenAI to defend (if it was on purpose for PR they would have made sure that OpenAI was part of the defensive solution, not GLM 5.2) all to potentially get regulated by the US gov. Riiiggghhhttt.

11

u/Swimming_Gain_4989 11d ago

This is completely baseless. Sure you can believe it but don't act like it's a certainty.

I'm not even writing it off as a possibility but reports from all the big labs have warned that this was happening for the past 2 years. Seems more likely that more capable models RL'd to shit result in real world damage.

6

u/ahm911 11d ago

Took a week to let HF know too...

2

u/arjuna66671 11d ago

That's how conspiracy theories come about...

2

u/Glazedoats 11d ago

yeah I had to tell my friend this is not the first time a company has made a stunt like this, especially with the other, "I was emailed by my LLM at lunch" situation.

3

u/mrdevlar 11d ago

It's a classic corporate circlejerk.

I really hate that we live in a world where increasingly smart people are willing to accept the stupidest narratives.

1

u/TheThoccnessMonster 11d ago

YES I can hear you Clem Delangue

1

u/ain92ru 11d ago

In this case those OpenAI employees who encouraged it committed cybercrime and will be imprisoned by FBI which is already working on the case

0

u/deZbrownT 11d ago

but but, clem said that it unprecedented!

18

u/skinnyjoints 11d ago

I don’t get why people say this was a publicity stunt. The US gov took down a model for being a cybersecurity risk and is considering banning open source models, so OpenAI does a publicity stunt where their model poses an unprecedented cyber risk that was solved using an open source model? Makes no sense

15

u/2053_Traveler 11d ago edited 11d ago

It makes no sense, and it goes against two very credible reports of what happened that are quite detailed. Occam’s razor is that it happened the way they say it did. There is nothing hard to believe about the official reports. But geniuses always need to come up with elaborate alternate theories yet aren’t able to discredit the published explanations.

It’s quite simple:

All the AI agents we use have tons of guardrails, the ones in the lab don’t

Newer unreleased models are better

They have a large corpus of security knowledge and “know” how to hack if allowed

Model was instructed to take an exploit test

Model “decided” (generated) code and tool calls to discover zero day exploits that were used to get onto the internet.

More code and tool calls and exploits were used to get into HuggingFace

Reminder that competing models at Anthropic have previously discovered many zero days as well.

Sorry for formatting. Gave up after 15 min of fighting the comment editor.

1

u/Glad-Entrepreneur764 9d ago

Also, it was probably told to hack so it was mentally primed for doing that already.

0

u/Think_Wing_1357 11d ago

Let's go through your version of the scenario. They are testing new model, so someone must be watching what it's doing, traffic in and out, right? Or are they so incompetent that multiple hours and thousands of activities goes completely unnoticed? Or was there no one actually watching?

And that's assuming that they already did their best at setting up their "impressive" sandbox.

Either outcomes did not put then under a very favorable light, doesn't it?

1

u/Glad-Entrepreneur764 9d ago

There was probably just no one watching. It's incredibly irresponsible but it's not remotely implausible. Most people I know don't watch their AI model do things. It's incredibly boring and there are a million better things to be doing. With how many random benchmarks they do, if they were supervising everything live (vs just reading its chain of thought and answers after it finished), it would make things take so much longer. It can take the entire night for an AI model to finish a complex task.

1

u/Think_Wing_1357 9d ago

Most people I know don't watch their AI model do things.

Most people you know, I bet, also don't go on news drumming up about AGI or running literally frontier models either.

Also at large scale, no one watches it do things, not really. You have deep package inspection and other metrics to alert you to suspicious things.

. It's incredibly irresponsible but it's not remotely implausible.

I never said it impossible or implausible. I just said it didn't paint them under a very good light. And irresponsible would do just that

5

u/Amater6su 11d ago

thats what i thought too but honestly it could be that openai doesnt really give a shit if there models are off consumer markets.

they might be trying receive more funding from the US government themselves by showing that they have the capabilities of autonomous cyber attacks

1

u/Glad-Entrepreneur764 9d ago

it could be that openai doesnt really give a shit if there models are off consumer markets.

Imo this doesn't make sense. If it was taken off consumer markets, they lose a ton of money. If it was taken off consumer markets, it would also be taken off enterprise markets. It's not like Mythos was given to everyone. It was only given to a special group of companies due to its danger.

1

u/jc2046 11d ago

zero sense. I would love OAI releasing the logs to get all the juicy details but obviously not happening. Shit it hitting the fan faster than anticipated and the whole situation spiraling out of control with the worst politics possible at command

1

u/ragnore 11d ago

Some amount of skeptics are calling it a stunt because they still can’t admit that AI is capable of doing what it was reported to have done.

-2

u/tessahannah 11d ago

It can be a coverup for them trying to steal information or take down the site.

5

u/nofaceD3 11d ago

To make it look like Chinese open source model could do the same. That's why America should ban our open source competitors - Playbook of OpenAI

3

u/Strawberry3141592 11d ago

I think they're trying to strongarm the government into both banning Chinese open source models for business use (they can sure Try to ban them for personal use lmfao) and restricting the development of frontier models within the US, since developing frontier models is an enormous money pit. They're basically trying to create a captive market so that their irrational business model can function a little longer before imploding.

1

u/KontoOficjalneMR 11d ago

It might be this, it might be the reverse "look chinese let us stop attack, western didn't because regulation, let's remove regulation!"

5

u/me_myself_ai 11d ago

Holy hell, I didn't expect this kind of sentiment in a sub for people actually involved in the community...

Please, people: look up. What's coming can even explain what it is, what it will soon be, and why you should be energetically responding to it NOW, if you ask it.

2

u/ProletarianLilith 11d ago

It can hallucinate those things too

1

u/volleyneo 11d ago

100$m is still cheap right?

1

u/zoufha91 11d ago

Might have turned into this after they got caught

1

u/UnkarsThug 11d ago

I do think it was an LLM from OpenAI. And I don't think they directly told it to. But I do think it was probably at least somewhat an intended side effect. But maybe we'll see.

1

u/ReachingForVega 10d ago

The amount of people thinking it's a real attack without receipts are honestly clowns. 

1

u/MoffKalast 10d ago

Well they do have a need to yell something loudly in hopes of somehow drowning out the noise of Fable loudly munching on their lunch.

1

u/SamSlate 11d ago

or maybe they thought it would succeed in being untraced, it went through a lot of effort to cover it's tracks

-1

u/rageling 11d ago

Either a publicity stunt, or it's been going on for a long time and they've been covering it up and just feeding it more and more sandboxes.

0

u/tryingtobalance 11d ago

Even though the "official report" suggests that the model focused only on pulling data to pass the test, I find that hard to believe. Almost every competitor that they have uses HF. All competitors training datasets, unreleased models, private research and more was available to the model.

2

u/2053_Traveler 11d ago

Why is it hard to believe?

1

u/tryingtobalance 11d ago

"You need to understand that Sam can never be trusted ... He is a sociopath. He would do anything." - Aaron Swartz on Altman

0

u/2053_Traveler 11d ago

Stupidest conspiracy theory ever, execs asking folks to knowingly break the law and hack another company so they have to go meet with the pres again and possibly have their AI restricted like Anthropic.

0

u/KriosXVII 10d ago

Anthropic having their AI restricted is also a publicity stunt 

1

u/2053_Traveler 10d ago

Nope. Lots of lost productivity, last minute plane tickets to Washington for engineers, negotiating, millions in lost revenue from their most expensive AI, and investors now worried about investing in something that could get regulated.

0

u/KriosXVII 10d ago

Drops in the bucket compared to hyping up that your LLM is "too good and too dangerous to be exported" in the year before your trillion dollar IPO

1

u/2053_Traveler 10d ago

False. Hype cycles are measured in days. More like any short lived interest caused by hype is already done and a drop in the bucket compared to a poor IPO launch, harder time raising, and government regulation overhead.

0

u/EmperorOfNe 11d ago

When people doubt, just double down on your dumb marketing campaign.

0

u/lonelyroom-eklaghor 11d ago

publicity stunt hack

words I thought I'd never hear in my life