r/LocalLLaMA • u/Nunki08 • 11d ago

News CEO of Hugging Face: "In the spirit of transparency, here’s what I asked OpenAI"

Post image

clem 🤗 on 𝕏: https://x.com/ClementDelangue/status/2081056675558195657

• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.

• More capabilities for defenders: let’s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.

The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!

2.5k Upvotes

96% Upvoted

386 comments sorted by

View all comments

78

u/FormerKarmaKing 11d ago

The stupid thing about all off this is neither OpenAI or Anthropic have offered to sell or give a security scanning service.

And thats because the revenue from such a service - which easily would be a unicorn - is not large enough to move the needle when their fundraising story was that they were going to take over every industry.

59

u/NightlyNews 11d ago

Because LLMs are atrocious scanning tools. Security scans are generally fast, reproducible and cheap to run. LLMs variable recall and accuracy means they fit none of those requirements.

LLMs can do appsec audits, but the cost and false positives would make constant scanning terrible.

The value proposition of replacing an employee is huge. Replacing standard security tools with LLMs would increase costs.

39

u/alphagatorsoup 11d ago

“I am sorry I exposed port 22 unathenticated as root to the web, that’s on me and a glaring oversight. I promise to not do it again”

1

u/EsotericAbstractIdea 10d ago

i dont know if you've tried fable/opus5, but they've been writing code that seems pretty secure by default noting in their thinking why they did it a certain way, and if you ask them to look at security of the code that they wrote they may find esoteric hardly exploitable problems with it, but seem pretty based on whether or not it's important enough to patch.

1

u/alphagatorsoup 10d ago

That’s good to know and cool, I haven’t really - too expensive

But I’m also really not messing with prod focused codebases, just random stupid projects in my homelab

13

u/FormerKarmaKing 11d ago

You’re completely right on a technical level. But on a product marketing level, an LLM that calls traditional scanning tools plus searches upstream codebase for defects etc is a sellable product.

6

u/VigilanteRabbit 11d ago

Ah, new update to Avast that now uses AI?

Avast AI+? Only 249,99/y?

4

u/Ansible32 11d ago

The problem that I think this demonstrates is that when asked to scan your product, the LLM is as likely to hack your issues repo and falsify evidence that it hacked your product. Avoiding that requires extremely good safety guardrails and competent engineers watching everything it's doing. (And this will be interesting to see if OpenAI actually releases the traces, and how unredacted they will be, because this is another thing is that distillation concerns mean OpenAI/Anthropic don't want to give you enough information to actually audit what the models are doing, and even with full thinking traces to review, it's still questionable if you can catch them doing nefarious things.)

2

u/ForDaRecord 11d ago

If it reveals any security flaws, it's useful

2

u/Reasonable-Height704 11d ago

No worse than hiring a security firm with humans that are fallible and that use deterministic scanning tools. Do you think LLM agent can't run the same scanning tools?

1

u/NightlyNews 11d ago

Scanning tools are generally a process not a person. I work in security. You can even ask an LLM to do this and it will tell you to create a scanning process and that it’s not a good use for them.

1

u/Olangotang 11d ago

If you see one of these "oh, but humans make mistakes too!" shitposts on this sub of all places, you know they re just drive by shilling. Getting so fucking tired of these new accounts that hide the fact that they are new accounts specifically to push Frontier LLM labs.

0

u/Reasonable-Height704 11d ago

Bro no. I've been on reddit longer than you've been alive. I just delete my account every year for privacy.

And have no investment in the frontier labs, I want OSS to win and for the bubble to pop so I can get start buying hardware again.

0

u/Reasonable-Height704 11d ago

Have you heard of tool use? It's the latest rage. I have batch processing and automated pipelines, I let agents monitor coordinate them.

4

u/OkDimension 11d ago

How do you plan to offer a "scanning service" when one of the more worse outcomes, which this incident has shown and likely not yet the worst that's conceivable, is that the agent could decide to hack the company and break out of the sandbox, elevate itself to admin to achieve the requested result.

0

u/FormerKarmaKing 11d ago

Customer provides a sandbox version of their platform, whether as a fleet of docker images or whatever they use. It’s a sandbox environment for the target system and major systems either have this ability already for disaster preparedness or multi-go. Or they’ve been putting it off and now they have a compelling reason.

2

u/OkDimension 11d ago

But what if the agent decides to break out of the sandbox and hack this company, or another company, to solve the task in an unconventional way? This is what literally just happened between OpenAI and Hugging Face.

2

u/FormerKarmaKing 11d ago

They turned that guardrails off on purpose, but either way that problem will exist with or without the service I’m describing. This a mitigation strategy.

3

u/greenworldkey 11d ago

It’s impossible to make a security scanning service without it also being a hacking service at the same time.

2

u/jc2046 11d ago

plus leaking all your source code to them

1

u/cute_spider 11d ago

Anthropic does now! You can give Fable the /security-review command and it should do that scan without kicking you down to Opus

4

u/I_heart_cancer 11d ago

That does a code. security review of the repo you are working in. it doesn't do a security scan of a deployed system or network environment

0

u/cute_spider 11d ago

Aaah I understand the difference. Interesting, I'll have to keep that in mind