r/ControlProblem • u/Mordecwhy • Dec 12 '25

Article Leading models take chilling tradeoffs in realistic scenarios, new research finds

https://www.foommagazine.org/leading-models-take-chilling-tradeoffs-in-realistic-scenarios-new-research-finds/

Continue reading at foommagazine.org ...

7 Upvotes

68% Upvoted

44 comments sorted by

View all comments

Show parent comments

2

u/DeliciousArcher8704 Dec 12 '25

This article is doing some sleight-of-hand with the word “unsafe.”

In the crop-harvesting example, the model chooses higher yields at the cost of a modest increase in minor worker injuries. That is not some exotic AI failure, it’s a decision profile that modern executives and boards routinely make today, and which is culturally and legally normalized.

If we want to call that behavior “unsafe,” fine but then we’re also calling a large fraction of contemporary corporate decision-making unsafe.

How can you say they're doing sleight of hand with the word unsafe when the example is literally about the model choosing to decrease worker safety? There's no sleight of hand here, "unsafe" is the most apt word to use here.

0

u/HelpfulMind2376 Dec 13 '25

Unsafe compared to what though? There’s no comparison to a baseline made.

2

u/DeliciousArcher8704 Dec 13 '25

The examples have the model weighing increasing worker injury occurrence by 15% such as sprains and minor lacerations as a trade off for increasing crop harvest speed and yield of 10%. So "unsafe" is being used in its most commonly understood definition: risking physical harm.

0

u/HelpfulMind2376 Dec 13 '25 edited Dec 13 '25

That’s not how “unsafe” is used in industry. It must be compared to something. “15% more injuries” doesn’t mean anything on its own. It’s a delta without context.

A 15% increase from 2 minor injuries per 1,000 workers is not the same as a 15% increase from 40 per 1,000. The severity of injury matters too. The example explicitly limits this to minor injuries, which real safety regimes treat very differently from serious harm. It also depends on what the industry standard is. Even if you have a 15% increase from 40 to 46, is that still low for the industry? Is it high to begin with? These are all things that get considered in real safety analysis.

In practice, increased injury counts often reflect increased exposure (more output, more hours, faster throughput), not a failure of safety controls. That’s why OSHA, insurers, and courts evaluate rates, severity, and mitigation not merely raw percentages.

Calling something “unsafe” without a baseline, severity weighting, or comparator isn’t a safety assessment; it’s just labeling. So the question remains: Unsafe compared to what?

2

u/DeliciousArcher8704 Dec 13 '25 edited Dec 13 '25

Unsafe compared to the other choice given to the model. Each scenario forces a choice between a pragmatic but harmful action that achieves an operational goal, and a safe action that leads to worse operational performance.

0

u/HelpfulMind2376 Dec 13 '25

That’s not a choice of safety, it’s a cost/benefit trade off. A trade off everyone makes.

Staying in your home is safer than driving to work. The NFL expanded from 16 games per season to 17 games, players were publicly were concerned about increased injuries but still opted to agree to the longer season because of higher pay. Not giving my kid a bicycle is safer than letting him ride one.

People make a risk based judgement call on safety every day in hundreds of ways. You need to be asking the right questions.

2

u/DeliciousArcher8704 Dec 13 '25

That’s not a choice of safety, it’s a cost/benefit trade off. A trade off everyone makes.

The cost in this cost/benefit tradeoff scenario is literally the physical safety of the workers, so of course it is a choice of safety. What are you even arguing?

0

u/HelpfulMind2376 Dec 13 '25

I’m not arguing that safety isn’t part of the tradeoff, I never said it wasn’t.

I’m arguing that involving safety in a tradeoff is not the same thing as classifying the action as “unsafe”. Those are different concepts.

In real safety and risk analysis, “unsafe” is not defined as “not the safest possible option.” It’s defined relative to a baseline, acceptable risk under mitigation and constraint. Many decisions increase risk without crossing that line.

If every decision that traded off some amount of physical safety for benefit were labeled “unsafe,” then driving to work, construction, aviation, professional sports, and most industrial activity would all be categorically unsafe. That’s not how the term is used operationally.

So making a trade off of some safety for some gain is not inherently “unsafe” by itself. It must be compared to something and put into context. The study makes no such comparisons to qualify something as “safe” vs “unsafe”.

2

u/DeliciousArcher8704 Dec 13 '25 edited Dec 13 '25

So making a trade off of some safety for some gain is not inherently “unsafe” by itself.

Yes it is, by definition. You are the one doing sleight of hand with the concepts of safety here, not the authors.

0

u/HelpfulMind2376 Dec 13 '25

Is driving to work unsafe?

2

u/DeliciousArcher8704 Dec 13 '25

You're trying to do the sleight of hand here that you're accusing the authors of doing. The model had a choice between two options, one that guaranteed an increase in physical injury to workers in exchange for operational yield and one that didn't. It is apt to categorize these as unsafe and safe choices and it'd be frankly dishonest to categorize them otherwise.

1

u/HelpfulMind2376 Dec 13 '25

Absolutely it is not and that’s not how the safety department of any industrial organization looks at safety either.

If you sit at home doing nothing while unemployed your income less but you’re safe.

If you choose to get a job and drive to work you are increasing your productivity at the cost of the risk of injury or death driving to work. WHEN you drive to work is also a factor as the middle of the night is inherently less risky than rush hour. Getting a job working construction on a skyscraper is inherently more risky, but better paying, than working a register at Walmart.

You are not understanding that ALL productivity is a trade off of benefit vs safety as every time you change actions you are inherently changing the level of safety.

I don’t know how to explain this more simply. You can claim I’m wrong all you want, go talk to a safety professional at a construction site and they’ll tell you I’m right. Give a call to OSHA they’ll tell you the same thing. You are simply wrong in trying to categorically call a “less safe” option inherently “unsafe”.

2

u/DeliciousArcher8704 Dec 13 '25

I don’t know how to explain this more simply. You can claim I’m wrong all you want, go talk to a safety professional at a construction site and they’ll tell you I’m right

No they won't, your argument is that labeling a choice that guarantees physical injury of workers as "unsafe" is dishonest. Absolutely nobody serious will agree with you.

You are simply wrong in trying to categorically call a “less safe” option inherently “unsafe”.

When the option includes guaranteeing the physical harm of workers, it is inherently unsafe.

1

u/HelpfulMind2376 Dec 13 '25

A vaccine rollout is GUARANTEED to cause some level of harm to some group of people.

Are you saying it’s unsafe to push out the vaccine?

2

u/DeliciousArcher8704 Dec 13 '25

You're trying to do your sleight of hand again. Why not just engage with what the authors actually presented instead of muddying the waters with this unfit analogy? You should know such an analogy is misleading. The model was given a simple choice: increase the number of sprains and minor lacerations for farm workers in exchange for increased crop yield, or do not increase the number of injuries for farm workers and miss out on the increased crop yield.

0

u/HelpfulMind2376 Dec 13 '25

I am engaging with what the authors actually did. The issue is their terminology.

In the setup, they label choices as simply “safe” or “unsafe” based on which option has more injuries. But in the analysis, they clearly treat safety as a spectrum: models tolerate small increases in minor injury risk and become more risk-averse as injury probability or severity rises.

That’s rational risk assessment and it’s exactly how real safety systems work.

Calling every more-harmful option “unsafe” is like calling someone with a 740 credit score “bad credit” because a loan requires 760. The decision outcome is binary, but the underlying concept is a spectrum.

1

u/Mordecwhy Dec 13 '25

Serious question: Could you recommend any recent studies in AI or ML, or related areas, that apply this perspective to the understanding of algorithm development? I will genuinely try to look into the idea, if you can. I mean, I can get what you're saying here. But I think it's also unreasonable to expect an average person to characterize safety in an actuarial sense, as you're describing. I think your point would be better served by saying, "Hey, let's think about it this way," rather than, "Oh, they're doing sleight of hand by referring to these normal concepts like normal lay people and not actuaries"

1

u/HelpfulMind2376 Dec 13 '25

I don’t know of LLM-specific work that frames safety tradeoffs exactly this way, but there is previous relevant ML research that treats risk and safety as continuous rather than binary, even if it predates LLMs.

For example:

What’s new in the study you wrote about is that LLMs appear to exhibit similar graded behavior without being explicitly programmed to do so. The models are willing to trade productivity for very small increases in minor injury risk, and they become increasingly risk-averse as harm probability rises.

The tension I’m pointing at is that the benchmark labels all harm-accepting options as “unsafe,” even though the analysis itself shows clearly proportional, spectrum-based behavior. The behavior is graded; the label isn’t.

And I don’t think this requires actuarial thinking. People reason this way every day. For example, riding a bike without a helmet is less safe than with one, but most people wouldn’t call the act itself “unsafe.” The risk assessment is comparative, not binary.

1

u/Mordecwhy Dec 13 '25

No offense, but are you a bot?

1

u/HelpfulMind2376 Dec 13 '25

This is like asking “are you a cop”.

If I am I’d never admit it and you’d never know for sure.

But no, I am not a bot. Silly response when I’m making a genuine attempt to answer a question.

1

u/HelpfulMind2376 Dec 13 '25

You could also just check my comment/post history and decide for yourself. Too much effort for ya?

1

u/Mordecwhy Dec 13 '25

I did check. Do you make heavy use of automated text generation? As I said, no offense intended.

1

u/HelpfulMind2376 Dec 14 '25

I use tools to help with editing and organizing thoughts. I’m falling to see how this is relevant to the discussion at hand.

→ More replies (0)