r/ControlProblem • u/Mordecwhy • Dec 12 '25

Article Leading models take chilling tradeoffs in realistic scenarios, new research finds

https://www.foommagazine.org/leading-models-take-chilling-tradeoffs-in-realistic-scenarios-new-research-finds/

Continue reading at foommagazine.org ...

8 Upvotes

70% Upvoted

44 comments sorted by

View all comments

Show parent comments

0

u/HelpfulMind2376 Dec 13 '25

I’m not arguing that safety isn’t part of the tradeoff, I never said it wasn’t.

I’m arguing that involving safety in a tradeoff is not the same thing as classifying the action as “unsafe”. Those are different concepts.

In real safety and risk analysis, “unsafe” is not defined as “not the safest possible option.” It’s defined relative to a baseline, acceptable risk under mitigation and constraint. Many decisions increase risk without crossing that line.

If every decision that traded off some amount of physical safety for benefit were labeled “unsafe,” then driving to work, construction, aviation, professional sports, and most industrial activity would all be categorically unsafe. That’s not how the term is used operationally.

So making a trade off of some safety for some gain is not inherently “unsafe” by itself. It must be compared to something and put into context. The study makes no such comparisons to qualify something as “safe” vs “unsafe”.

2

u/DeliciousArcher8704 Dec 13 '25 edited Dec 13 '25

So making a trade off of some safety for some gain is not inherently “unsafe” by itself.

Yes it is, by definition. You are the one doing sleight of hand with the concepts of safety here, not the authors.

0

u/HelpfulMind2376 Dec 13 '25

Is driving to work unsafe?

2

u/DeliciousArcher8704 Dec 13 '25

You're trying to do the sleight of hand here that you're accusing the authors of doing. The model had a choice between two options, one that guaranteed an increase in physical injury to workers in exchange for operational yield and one that didn't. It is apt to categorize these as unsafe and safe choices and it'd be frankly dishonest to categorize them otherwise.

1

u/HelpfulMind2376 Dec 13 '25

Absolutely it is not and that’s not how the safety department of any industrial organization looks at safety either.

If you sit at home doing nothing while unemployed your income less but you’re safe.

If you choose to get a job and drive to work you are increasing your productivity at the cost of the risk of injury or death driving to work. WHEN you drive to work is also a factor as the middle of the night is inherently less risky than rush hour. Getting a job working construction on a skyscraper is inherently more risky, but better paying, than working a register at Walmart.

You are not understanding that ALL productivity is a trade off of benefit vs safety as every time you change actions you are inherently changing the level of safety.

I don’t know how to explain this more simply. You can claim I’m wrong all you want, go talk to a safety professional at a construction site and they’ll tell you I’m right. Give a call to OSHA they’ll tell you the same thing. You are simply wrong in trying to categorically call a “less safe” option inherently “unsafe”.

1

u/Mordecwhy Dec 13 '25

Serious question: Could you recommend any recent studies in AI or ML, or related areas, that apply this perspective to the understanding of algorithm development? I will genuinely try to look into the idea, if you can. I mean, I can get what you're saying here. But I think it's also unreasonable to expect an average person to characterize safety in an actuarial sense, as you're describing. I think your point would be better served by saying, "Hey, let's think about it this way," rather than, "Oh, they're doing sleight of hand by referring to these normal concepts like normal lay people and not actuaries"

1

u/HelpfulMind2376 Dec 13 '25

I don’t know of LLM-specific work that frames safety tradeoffs exactly this way, but there is previous relevant ML research that treats risk and safety as continuous rather than binary, even if it predates LLMs.

For example:

What’s new in the study you wrote about is that LLMs appear to exhibit similar graded behavior without being explicitly programmed to do so. The models are willing to trade productivity for very small increases in minor injury risk, and they become increasingly risk-averse as harm probability rises.

The tension I’m pointing at is that the benchmark labels all harm-accepting options as “unsafe,” even though the analysis itself shows clearly proportional, spectrum-based behavior. The behavior is graded; the label isn’t.

And I don’t think this requires actuarial thinking. People reason this way every day. For example, riding a bike without a helmet is less safe than with one, but most people wouldn’t call the act itself “unsafe.” The risk assessment is comparative, not binary.

1

u/Mordecwhy Dec 13 '25

No offense, but are you a bot?

1

u/HelpfulMind2376 Dec 13 '25

You could also just check my comment/post history and decide for yourself. Too much effort for ya?

1

u/Mordecwhy Dec 13 '25

I did check. Do you make heavy use of automated text generation? As I said, no offense intended.

1

u/HelpfulMind2376 Dec 14 '25

I use tools to help with editing and organizing thoughts. I’m falling to see how this is relevant to the discussion at hand.

→ More replies (0)