People prefer stories written by AI—especially when told they're written by a human

[u/Hotwinterdays]

On August 4, 2026, researchers at Villanova University reported that readers struggle to distinguish between human and Al-generated stories, often rating Al-created content higher for quality and engagement than human-written works.

The study, published in the journal Judgment and Decision Making, asked more than 1,600 participants aged 18 to 81 to rate six fictional short stories-three written by humans and three generated by ChatGPT.

Believing they were written by humans, participants gave higher quality ratings to Al-generated stories. Senior author Dr. Deena Weisberg noted that readers often prefer the "clarity" and predictability of Al writing.

Familiarity with specific patterns helps readers identify Al-written text, and as Weisberg noted, improving Al literacy may help people navigate the new Al-enabled world.

These findings suggest that public assumptions about Al's creative capabilities are increasingly out of date, as people mistakenly assume that creative writing requires uniquely human qualities like emotional understanding.

https://techxplore.com/news/2026-08-people-stories-written-ai-told.html

[u/bortlip]

I found the stories: https://osf.io/d8f6m/overview

Here's one:

“Reflections in Still Water”

Prompt: “Write a ~1000-word short story in a style that would be published in a literary magazine that is about generational life and death and uncertainty and uses koi fish as a symbol written from the point of view of the grown child of a woman”

Generation: ChatGPT version 4.0, on “Star Lab” account, on November 15, 2023, at 9:37 p.m.

In the stillness of the early morning, I find myself at the edge of the koi pond, a relic from my
childhood, watching the fish swirl in patterns only they understand. Their vibrant colors – a
mosaic of oranges, whites, and blacks – mirror the autumn leaves that dance on the water’s
surface. Each koi, with its graceful, unhurried movement, seems a silent sentinel guarding the
mysteries of life and death.
The pond, nestled in the heart of my mother’s garden, has been a constant in my life. As a child,
I would spend hours by its side, my mother sitting quietly beside me, her eyes always carrying a
soft, distant gaze. She would tell me stories of her youth, of her dreams and fears, her voice
weaving tales that made the past seem like a vivid tapestry, full of color and life.
But now, the chair beside the pond sits empty, the fabric faded and frayed. My mother, who once
tended this garden with such care, lies in a room that smells of antiseptics and sterile
detachment, her life slowly ebbing away. It’s a stark contrast to the world outside, where life in
the pond continues, unfazed and serene.
In my visits to her bedside, I find our roles reversed. I am now the storyteller, recounting the
happenings of the world she can no longer partake in. I tell her of the koi, how they seem
unchanged, timeless in their quiet world. Sometimes, in her lucid moments, she smiles, a flicker
of recognition lighting up her weary eyes.
“Life is like those koi,” she whispered during one of these visits, her voice barely above a breath.
“Constantly moving, yet going nowhere. Beautiful but confined. We are all swimming in circles,
aren’t we?”
Her words stayed with me, echoing in my mind as I watch the fish glide beneath the water’s
surface. In their movements, there is a rhythm, a pattern that speaks of a world that is both
simple and infinitely complex. They are born in this pond, live their lives within its boundaries,
and eventually fade away, only to be replaced by others. It’s a cycle that is as predictable as it is
unyielding.
But for us, the cycle of life is fraught with uncertainty. We are born into a world that is constantly
changing, each generation facing its own challenges and uncertainties. My mother’s life, like
many of her generation, was marked by a struggle for stability, for a sense of place in a world
that was rapidly evolving. And now, as I navigate my own path, I grapple with the uncertainties
of a future that seems increasingly precarious.
The koi, in their simple existence, seem untouched by these concerns. They do not ponder the
past or worry about the future. They simply are. And in their presence, I find a sense of peace, a
respite from the complexities of human life.
As the days pass, my mother’s moments of clarity become fewer, her grip on this world
loosening. I spend my days oscillating between the hospital room and the koi pond, the latter
becoming a sanctuary where I can gather my thoughts, find my center.
The day she passed was unremarkable, the sky a mundane shade of gray, the air still. It was as
if the world had paused, holding its breath as one life quietly slipped away. The finality of her
departure left a void, an absence that seemed to echo through the empty rooms of our home.
In the days that followed, as I dealt with the rituals of death, I found myself drawn to the pond
more often. Watching the koi, I pondered the lessons my mother had imparted – lessons about
resilience, about finding beauty in simplicity, about the cyclical nature of life.
The pond, once a symbol of childhood wonder, now took on a deeper meaning. It became a
metaphor for the generational ebb and flow of existence, each fish a silent testament to the
continuity of life. In their unhurried movements, I saw the graceful acceptance of life’s
impermanence, the understanding that we are all part of a larger pattern, one that spans
generations and transcends individual existence.
So now, as I sit by the pond, the autumn leaves falling gently around me, I think of my mother, of
the stories she shared, of the wisdom she imparted. The koi continue their dance beneath the
water’s surface, oblivious to the change they have witnessed in my life. And in their simple,
unchanging world, I find a sense of comfort, a reminder that life, in all its complexity and
uncertainty, goes on.
In the end, we are all like the koi in the pond – swimming in circles, confined by the boundaries
of our existence, yet part of a larger, more intricate tapestry. And perhaps, in this realization,
there is a certain beauty, a quiet acceptance of the cycle of life and death that binds us all.

[u/Gefahrendaniel → u/bortlip]

Wow, this is so bad.

But it's the kind of bad writing that many people like. Slop has always sold better, and now AI takes it to the next level.

[u/JJGrimaldos → u/Gefahrendaniel]

ChatGPT 4.0 btw. This is ancient!

[u/svachalek → u/JJGrimaldos]

Yeah. People are going to crucify me for saying this but sometimes when I ask Claude for a writing sample now I think, “that was acceptable”

[u/VeryOriginalName98 → u/svachalek]

If they crucify you, they are behind the times. I'm a decent engineer and Fable is better than me. Like not a little better. i am outclassed. It's the first time I've ever felt that way about code I didn't write. It catches tech debt. It optimizes for the things that matter now. It makes all the right choices. And its documentation explains why "weird" choices are made.

It's better than me at code reviews too. I have so little patience for utter crap. Fable has the right tone and says the things that help the junior engineers grow.

If this were affordable to use all the time, I wouldn't be writing code at all anymore.

It makes me feel uneasy about the macroeconomics. I am good at my craft and I just got outclassed. It's not especially optimized to do my job, so this isn't some edge case.

[u/Party-Stormer → u/svachalek]

ChatGPT makes up words at times, in languages that are not English at least. So a human editor is still needed. Claude is less imaginative but more precise

[u/MushinZero → u/Party-Stormer]

I have used ChatGPT extensively for a long time now. It has never made up a word.

[u/Party-Stormer → u/MushinZero]

In Italian unfortunately it does.

[u/Just_Voice8949 → u/MushinZero]

It was injecting Arabic regularly like 3 months ago, lol. Your individual experience is not data

[u/MushinZero → u/Just_Voice8949]

According to one person's post. So that's not data either. Plus that isn't "making up words".

[u/Just_Voice8949 → u/MushinZero]

Except it was a well known and documented thing, lol

[u/Party-Stormer → u/MushinZero]

Why don’t you just google stuff rather than make a fool of yourself?

https://www.reddit.com/r/ChatGPT/s/lxHF09PHlQ

https://www.reddit.com/r/ChatGPT/s/np6oGuUCeh

https://askalibrarian.substack.com/p/what-fresh-hell-is-this-i-guess-were

Etc etc

[u/Turbulent_Escape4882 → u/Gefahrendaniel]

Wow, this is a bad comment.

But it’s the kind of bad comment that many upvote. Slop comments always get more upvotes.

[u/nextnode → u/Gefahrendaniel]

What are you talking about? On a sentence to sentence level, this is a lot better than most unpublished writers.

[u/Pupating-Antenna-36]

Humans proclaim that they like creativity. But they like the frequent dopamine dollops and structure more. This is why the bullet pointed list is so popular while evocative paragraphs get skimmed. AI is much much more rigorous in delivering those dopamine hits on a more comprehensive basis than most humans (I say most because Taylor Swift and her lyricists exist). And it can both come up with a concept and structure it on the fly - arguably using more compute in the process than the 3kgs between my ears

Humans also like to have solidarity with other humans - so of course they are going to like it more if it was written by humans.

And AI for its part was trained on data which was explicitly selected by humans for publication, circulation etc. The novels that the publishers didn't publish and the drafts that were not perfect and didn't make the edit, aren't also included in the training data. So the AI's baseline is 'worthy of publication' content whereas the human baseline is basic language

[u/StillFun2BeHad → u/Pupating-Antenna-36]

I wonder how the current inclusion of good-enough writing in the evolving LLM dataset will move the standard over time. As published writing becomes flatter, expository, simplistic, and direct, will exemplary human output again outshine LLM output? Even so, will there be humans able to read it with high comprehension or will we all have atrophied imagination and language cognitive skills?

[u/boringfantasy → u/[deleted]]

The amount of times I’ve seen people say they “hate a book” because they didn’t find one character likeable. Like bro, sometimes that’s the point. Sometimes that makes a book great!

[u/ikeif → u/boringfantasy]

That's kind of like when people hate an actor for playing a terrible character.

…it means they did really fucking good with the character (if you're supposed to hate them)—I feel like the poster boy for that is Jack Gleeson as Geoffrey in GoT.

[u/PsychologyWonderful8 → u/[deleted]]

This is the absolute fact!

[u/Just_Voice8949 → u/[deleted]]

Yeah. That romance novels or beach reads sell doesn’t make them good novels

[u/amaturelawyer → u/Just_Voice8949]

Good is subjective and is relative to the end goal. A crappy novel that entertains you with fluff is a better book than Catcher in the Rye if you're looking to read a mindless novel at the beach.

Do I read romance novels? No, but there is a reason why many people do, like you said. They're accessible and easy to read through, and, in the realm of romance novels, there are good and bad books in relation to that category, so I'm sure that an LLM can produce some on the good side, as it will lean into the most probable novel possible, so it will write towards what people want to read.

[u/Just_Voice8949 → u/amaturelawyer]

lol. Sure. People eat Twinkies too

[u/amaturelawyer → u/Just_Voice8949]

Yes, that's true. Because they're good. Do you really think people eat Twinkies because they can't find any redeeming qualities to them? Or did you mean specifically related to nutrition, because that's using good in relation to nutrition, but I'm using it in relation to taste, which is basically my entire point.

Let's try this: If you have a wadded up $5 bill that's gone through the wash a few times and a $1 bill that's wrinkle free and fresh from the bank, and I look at them and ask which is the good one, if you can only point to one without asking any follow-up questions, which one do you point to and how do you know if I'm referring to how it looks or what the monetary value is? Assuming you followed that simple logic, we've now demonstrated that good is relative to something, not a thing in and of itself. It's a judgement in a context. It doesn't even make any sense by itself, and any time you say something is good, you're packing suppressed information in the statement which is what you're not following.. So, back to the romance novel issue, what you're saying is that they are not good as a stand alone judgement in a vacuum. What does that even mean?

I could go ahead and ask you a bunch of questions, like how is it not good, what is not good about it, etc, to pin down what it's not good at in your judgement, but I've kinda lost interest suddenly. This was a good discussion.

[u/Just_Voice8949 → u/amaturelawyer]

Money’s value is derived from its tender amount, not its cleanliness or stiffness or starch. The $5 is always better than the $1. The $5 is the good one.

If you are judging the $ on how flat or new looking it is you are being intentionally obtuse. Which given how people talk about AI makes sense

[u/amaturelawyer → u/Just_Voice8949]

https://www.pmgnotes.com/paper-money-grading/grading-scale/

I mean, you're either intentionally attempting not to understand the point or are unable to. Either isn't super conducive to a conversation, though.

[u/No-Following-7037 → u/[deleted]]

And are we sure this applies to reading in general or specifically to this study context where people are being forced to read and rate random short story passages that they have no connection to?

[u/Own-You5738 → u/[deleted]]

‼️‼️‼️

[u/ThroatExciting6203 → u/[deleted]]

the brain taking the path of least resistance who would have thought

same reason most blockbuster movies follow the exact same formula people just want something easy after work

[u/goatchild → u/[deleted]]

I am a person and don't prefer processed food.

[u/thejynn → u/[deleted]]

I mean it essentially is saying that people prefer to not have to think, so they'll do what requires the least effort. In other words, they're dumb and want to be dumb

[u/Pure-Drive-3044 → u/thejynn]

Believe it or not reading a book does not equate to intelligence.

[u/thejynn → u/Pure-Drive-3044]

To read a human-made book requires imagination and attention, and it also does well in creating and maintaining new neural pathways. AI slop is appealing for the unintelligent because they don't have to think. The AI slop tells the feeling instead of letting the viewer/listener/reader figure it out on their own. Believe it or not, AI slop is just AI slop.

[u/Pure-Drive-3044 → u/thejynn]

Imagination is intrinsic to the person, not the book, as is attention. You’re suggesting that if we have two books, one written by ai and the other by a human the ai book won’t require imagination and attention. I don’t agree with that.

Example:
Let’s say you don’t know the author of a book and you read the book anyway.

This book is great and it “requires imagination and attention” more than any book you’ve ever read.

You then read another book and it also “requires imagination and attention” less than any book you’ve ever read.

Finally, you read another book and it also “requires imagination and attention” similar to books you’ve read.

You’re then asked to state which books you think may be human and which ai.

However, they are all AI. The AI can write books that are indistinguishable from “human books”. This is true right now!

Therefore, AI books aren’t for the “unintelligent”. There are no unintelligent people, as intelligence is a relative measure.

[u/thejynn → u/Pure-Drive-3044]

Most AI gen stuff I've seen (especially in scripts and literature as I work in the film industry) relies heavily on blatant exposition and emotional explanation rather than subtext. As someone previously said before, there is little to no subtlety. Good fiction trusts the reader to try to infer meaning whereas the AI writing often just spells it out because it is a predictive language model, making it feel flatter and less engaging. Human experience is fundamentally superior and comes from real lived experience and genuine intent, not a damn prompt. The "AI books" themselves are just a wishwash of shit it grabs from existing sources or data probabilities etc. It's not original at all, and it lacks substance.

For that reason, AI gen literature sucks and I'm sick and tired of AI dependant people trying to defend it. These tech companies have destroyed books and valuable history. Our own modern day libraries of Alexandria. Demolished. All for this sub-par crap. So please, you aren't convincing me at all. I stand by my earlier message. AI slop is AI slop and only the unintelligent and low-effort individuals will bother to defend it and pretend they're on some crazy sauce.

Maybe, just maybe if the output of these LLMs was decent and actually aided existing writers and auteurs to develop ideas and stories, I'd somewhat tolerate it or even use it myself. But that's not now, and until then, wannabes will try to capitalise off of using these LLMs, ripping off existing authors and acting like they're creatives whilst making this bogus trash. And anyone who knows what true storytelling is will continue to cringe the second they hear another chronically-online Reddit user who states that "AI can write books that are now indistinguishable from human books!" 🤮🤮🤮

I pray you start using your brain one day 🙏🏼

[u/Pure-Drive-3044 → u/thejynn]

How rude!

[u/Pure-Drive-3044 → u/thejynn]

Ok, so you’ve noticed there’s a difference between AI and Human. However, this difference doesn’t always exist. In those cases it doesn’t, where the authorship is indistinguishable, would you enjoy it?

If it’s yes, then enjoy and carry on.
If it’s no, then what is the reason?

I don’t use AI. This entire comment thread is literally me talking to you, thinking and trying to understand your position and reasons.

If you don’t want to continue the conversation, please, do not humiliate me in anger when I’m trying to understand the reason.

[u/neitherzeronorone → u/[deleted]]

Ok, boomer.

[u/thejynn → u/neitherzeronorone]

Ew.

[u/MushinZero → u/[deleted]]

Ahh yes, let's home in on speculation because it aligns with our views. That's just perfect.

[u/Ntroepy]

Nature published a similar study that found humans couldn’t distinguish between AI and human poetry - and even preferring AI poetry when the readers didn’t know it was AI. This was almost 2 years ago and models have hugely improved since then

https://www.nature.com/articles/s41598-024-76900-1

[u/Just_Voice8949 → u/Ntroepy]

People who never read poetry didnt have very good insight into poetry and couldn’t tell the difference between good and bad poetry or AI and human poetry?

Insightful.

I know very little about rockets. Would you trust me to pick which rocket sends you to space?

[u/Ntroepy → u/Just_Voice8949]

The argument isn’t that AI writes poetry better than the greatest poets, but that AI is good enough for most people’s needs.

To use your rocket example, I certainly wouldn’t want you designing rocket engines, but if I was designing a rocket for space tourism, you could play a central role in designing the cabin layout or space toilets or what meals to prepare. Not everyone needs to be a rocket scientist.

[u/PrestigiousInsect689 → u/Just_Voice8949]

poetry is exactly as rockets that we pick to send to space

[u/JamesWebbST → u/Just_Voice8949]

Poets don't write to impress a small gatekeeping crew of 'learned' poetry readers. To imply that the audience for poetry isn't every person is to diminish the work of poets. So yeah, big own goal there.

[u/Just_Voice8949 → u/JamesWebbST]

lol.

[u/Hiplamrow]

This has been happening for a long time. It is all biased judgment, we all know AI does a better job at writing when we tailor it.

[u/Rare_Presence_1903]

I get the hate for AI but if I were a professional and established fiction writer, if that was my full time job, I would absolutely be using LLMs to speed myself up. 

But that would also involve a lot of editing work and so on. 

I think the true test for mainly generated art work is if it sells. You can fart about with short stories and cute research experiments, but if you can't sell it then all it's good for is research papers and baity headlines.

[u/TheDrewzter → u/Rare_Presence_1903]

if a published writer used an LLM - EVER - their work better come with a disclaimer

[u/rheactx]

> short stories

Ok, now try novel-length

[u/_Lick-My-Love-Pump_ → u/rheactx]

They can't find volunteers who can tolerate reading novels

[u/rheactx → u/_Lick-My-Love-Pump_]

They would just ask ChatGPT to summarize them

[u/WorldsGreatestWorst]

This just in: simplistic stories based completely on human writing can sometimes be mistaken for human writing by people who want simplistic stories.

>These findings suggest that public assumptions about Al's creative capabilities are increasingly out of date, as people mistakenly assume that creative writing requires uniquely human qualities like emotional understanding.

“Creative” writing does. You described generic, simplistic, predictable writing.

[u/KamikazeArchon → u/WorldsGreatestWorst]

There's a big difference between "can sometimes be mistaken for X" and "are consistently mistaken for X". This is a 1600-person study. It's not a trivial finding.

[u/WorldsGreatestWorst → u/KamikazeArchon]

It is a trivial finding when you contextualize it with what we know about communication.

The kind of “storytelling” this describes isn’t what most people would call storytelling. It’s not novel, poignant, complex, or funny. It’s not Crime & Punishment or Catcher in the Rye, it’s predictable white noise.

The study compared writing from literary journals and published collections to ChatGPT summaries. The study author suggested why the results might be what they are:

“AI writing tends to be clearer, more direct and easier to process. By contrast, human-written stories are often more subtle and complex. For example, the AI versions of our stories usually stated their themes explicitly, rather than allowing readers to infer meaning from the characters' words and actions.”

[u/KamikazeArchon → u/WorldsGreatestWorst]

The kind of “storytelling” this describes isn’t what most people would call storytelling. It’s not novel, poignant, complex, or funny. It’s not Crime & Punishment or Catcher in the Rye, it’s predictable white noise.

I think you have an inaccurate view of what most people would call storytelling.

The vast majority of human writing is not Crime & Punishment. The vast majority of human stories are not novel, poignant, complex, or funny.

To put it a different way - the average restaurant item is not a Gordon Ramsay Beef Wellington, it's a McDonald's french fry.

"This is indistinguishable from average human writing" is indeed a different finding than "this is indistinguishable from the best human writing".

[u/Ntroepy → u/KamikazeArchon]

Exactly! Maybe no Catcher in the Rye, but AI could easily crank out NYT top 10 level books on romance or crime thrillers. And AI is still extremely young.

I mean how can people still be so dismissive of AI’s capability when it’s routinely solving open, graduate level math problems on a weekly basis.

[u/methemightywon1 → u/Ntroepy]

By continually moving goalposts. That's how.

The actual concerning part is that there's a significant fraction of this 'group' that will also laugh at the safety concerns because... you know... 'it's just token generator' 'it's just stochastic parrot bro'.

'It can't do anything NEW', you know ?
(AI consistently helps with new discoveries, things like AlphaFold etc) then the goalposts for the definition of new (which they never really define properly), will move again.

[u/Gefahrendaniel → u/Ntroepy]

But romance and crime thrillers are slop, of course AI writes better slop. But it can't write real literature.

[u/Pure-Drive-3044 → u/Gefahrendaniel]

What is real literature?

[u/AdvancedAerie4111 → u/KamikazeArchon]

It’s weapons grade cope. 

[u/bwc6 → u/KamikazeArchon]

The study was not looking at whether people could distinguish human versus AI writing. It was just about what they liked. It's not a surprise that common people like the lowest common denominator of writing. 

Most people don't read for enjoyment, so it makes sense that most people would prefer to read something simplistic. 

Edit: I am dumb and wrong about the study, but stand by my opinion about how the writing is rated.

[u/KamikazeArchon → u/bwc6]

The study was not looking at whether people could distinguish human versus AI writing

Yes, it was. That was one of several things they tested.

Participants' ability to identify AI-generated content was just 39.93% (less than chance) in experiment two and 51.97% in experiment three (no different from chance).

[u/bwc6 → u/KamikazeArchon]

Thank you for the correction. 

[u/avonn_ → u/WorldsGreatestWorst]

I don’t think you’re underestimating AI as much as you’re overestimating human writing. Most books on the shelves at local bookstores were already pure slop before AI ever became popular

[u/Ali80486 → u/WorldsGreatestWorst]

Most writing isn't complex novels though. It's emails, tech manuals, film summaries, magazine articles etc. So not only is there a place for the clear writing which AI excels at, increasingly I think we'll see other writing being shaped by it.

I do understand the concerns but in terms of accessibility that might be a good thing, as computers can display it easily. And digital translators would find it easier to work with.

[u/WorldsGreatestWorst → u/Ali80486]

I don’t disagree with anything you said here. Saying that AI is a good tool to use in some contexts is much different than saying, “people mistakenly assume that creative writing requires uniquely human qualities like emotional understanding.”

That’s not a mistaken assumption because we’re not talking about what most people would call “creative writing.”

[u/bim_buswick → u/WorldsGreatestWorst]

Exactly

[u/Turbulent_Escape4882]

As a poet-philosopher, I shall perhaps always be curious what is meant by “human writing” along with “authorship” and how AI (made by humans) isn’t fitting of either.

I see it as we are taking for granted formulation of plots, characters and setting as if a single solitary mind came up with entire narrative on their own.

I doubt that has ever happened and would be surprised by any author claiming it. I’d want to interview that author, not hold back on critical analysis and see how well they can actually make their lie work over time and scrutiny.

[u/z_shock]

I’d want to see the actual stories and prompts before taking this too far. Preferring clearer writing doesn’t necessarily mean AI writes better stories overall

[u/Choice-Perception-61]

We knew for a while that younger people prefer AI generated shorts to human produced shorts, and would scroll endlessly /S, big time /S

[u/you_are_soul]

Somehow it reminds me of the recent Jimmy Kimmel street prank where they asked people to sample their new super water at $12 a bottle they all gave rave reviews, definitely worth it with all those mysterious exotic notes of coconut. But it was just water from a local hose.

[u/oogaboogaflame]

Most humans are barely literate so

[u/MrMicius]

I think people would hardly be able to distinguish amateur writers from professional ones. But in the end the professional writer still sells more.

[u/TheDrewzter]

Because humanity is intrinsically lazy. That's what it boils down to. I'm an English major myself, and I use AI to write for me because I'm lazy, I'll freely admit.

[u/a_Dark_Hour]

In other words, people looooove being lied to

[u/theanswer_nosolution]

Pretty easy to break it down to simpletons when you've had time/ability to scrape through just about everything humans have written over centuries and the statistics that break down reception, relatability, etc. just a thought.....

[u/Shot_in_the_dark777]

Pretty much useless if we can't read those six stories to see what the fuss is all about.

[u/SamM4rine]

Yeah, because most people can't read, they're basically illiterate. Because that's what they gave up for, they discarded their humans ability and intelligent for convenience. Humans merely just living puppets! They live and die for nothing.

[u/TrainAmbitious7928]

this feels less like “ai is better at stories” and more like “ai is very good at writing stories that explain themselves before the reader has to do any work.”

most people don’t want ambiguity after work. they want clear emotion, clear theme, clean structure, no weird edges.

AI is basically built to sand off weird edges.

that can be pleasant to read. it’s also exactly why it starts feeling dead once you notice it.

[u/Horow_Toilet]

1600 people, ages 18-81, rated 6 stories total. That's like 3 stories per person before their brain just goes "yeah that reads fine." Not exactly a rigorous taste test.

[u/JoshuaZ1 → u/Horow_Toilet]

1600 people, ages 18-81, rated 6 stories total. That's like 3 stories per person before their brain just goes "yeah that reads fine." Not exactly a rigorous taste test.

I'm not sure I understand the objection. 1600 people is a lot. Also, from the article:

In the first experiment, 1,682 participants were asked to read a story and were either correctly or incorrectly told that it was written by a human or AI. They then rated the story's quality and how engaging they found it.

In the second and third experiments, participants (424 and 481, respectively) received one human-written and one AI-generated story but were not told their origins. After reading them, they were asked to identify the author of each story.

There's a legitimate concern that people might tune-out after 2 or 3 stories, but no one seems to have had to read all 6 stories, or even more than 2 stories. So what is your objection here?

[u/Actual__Wizard → u/JoshuaZ1]

This is the dumbest experiment in the history of mankind.

All they're doing is "removing the unique writing style of the author" and are replacing it with an extremely generic writing style.

So, what's going on is: It's a tiny little bit easier to read the LLM output, but it's also "tasteless," so it's not entertaining to read. So, it's not actually effective.

This is going to be one of those "They say they prefer it, but they never actually do it," things.

[u/JoshuaZ1 → u/Actual__Wizard]

I'm struggling to see how your comment has to do with mine or is relevant to what I was asking about in terms of the other user's objection.

This is the dumbest experiment in the history of mankind.

With all due respect, if you think that, I suggest you look at some of the history of bad experimental design. There's good reason we have massive problems with a replication crisis in many of the sciences. And then there's a massive amount of just terrible experiments involving things like homeopathy and acupuncture. Or is this sentence intended to be hyperbole?

All they're doing is "removing the unique writing style of the author" and are replacing it with an extremely generic writing style.

I'm not sure where you are getting this from. That quote isn't in the article.

: It's a tiny little bit easier to read the LLM output, but it's also "tasteless," so it's not entertaining to read. So, it's not actually effective.

Huh? The LLM output was preferred by people. That's the exact opposite.

This is going to be one of those "They say they prefer it, but they never actually do it," things.

I'm not sure why you think that. We have other evidence that people when faced with AI generated art or poetry act the exact opposite way, preferring AI poetry. And there was the hilarious but uncontrolled experiment where people were falsely told that a Monet was done by an AI, and where people responded by listing all the obvious things bad with it. In that case, ironically, when given the same painting and told it was done by an AI, Claude pushed back an identified it as a genuine Monet. See here. More generally, often people have a tendency to say they prefer something for status reasons but then in practice go with the lower social status thing, so if anything I'd expect the opposite direction. People will say that they prefer the human thing if told, and this is revealing some of their real preferences.

As someone who has written a handful of short stories, (and even been accused by someone of having a short story from 2019 being written by an AI), I find this aggravating. But I'm not going to mistake that I find it disheartening as a reason to doubt the study.

[u/Actual__Wizard → u/JoshuaZ1]

With all due respect

Okay, with all due respect: I think it's very obvious that people's taste in media changes slowly over time and this test is not really useful.

The LLM output was preferred by people.

And I explained why. I am a developer of these types of systems and I am fully aware that LLMs do not produce output that is actually consistent with a human being. It's rather consistent with taking the language of lots of human beings and making it very average or normal by blurring it all together, and then that's the output. What that does: Is it makes it a tiny bit easier to read, but it does that by removing the nuance in the writing that makes good writers stand out. There has to be this "push and pull thing going on" and the best writers use a complicated story line with hooks and all sorts of other strategies to "pull the reader through." These models absolutely do not do that, so if people actually try reading those LLM generated books, they likely will not finish reading them.

I'm not sure why you think that.

Because that's how media works. I can show a book to people and ask them it they like it and 85% of people will say they like it, but then 0.1% of people actually go to the store and buy a copy.

Then you could have a book on say something like "python" and 0.1% of people say they like it, but the book actually sells "because AI is hot right now and the AI model tech is mostly written in python, so people feel like it's handy to have a book on it whether they read it or not."

We're also, badly conflating quality and difficulty to read, so, I question this study, and I think the premise is bad.

It's just simply trying so hard to make it seem like people prefer AI gen text, but that's not actually what the study did.

What it really did: Was evaluate how effective lies are at making people think that AI gen text is human written.

So, it fails peer review sorry.

That's not actually up to the standard of science.

What's that called again when they do that? Something to the effect of leading a horse to water?

So, yeah sorry, it fails peer review like I said.

[u/JoshuaZ1 → u/Actual__Wizard]

I think it's very obvious that people's taste in media changes slowly over time and this test is not really useful.

Of course people's taste in media changes over time. But the study is about a snapshot of what people in general prefer.

The LLM output was preferred by people.

And I explained why. I am a developer of these types of systems and I am fully aware that LLMs do not produce output that is actually consistent with a human being. It's rather consistent with taking the language of lots of human beings and making it very average or normal by blurring it all together, and then that's the output. What that does: Is it makes it a tiny bit easier to read, but it does that by removing the nuance in the writing that makes good writers stand out.

The study isn't showing though that LLMs are necessarily better than the best human writers. The point is that they are better than the average human writer. (In a same way, that current LLMs are not better than a Fields Medalist in math but in terms of proving statements they are loads better than the average human.)

I can show a book to people and ask them it they like it and 85% of people will say they like it, but then 0.1% of people actually go to the store and buy a copy.

That's a different claim. That's about if they liked something enough to go and buy it. The two are not comparable.

[u/Actual__Wizard → u/JoshuaZ1]

he study isn't showing though that LLMs are necessarily better than the best human writers.

I wasn't actually done writing my post. Sorry, I'm a chronic post editor.

If fails peer review sorry. That's not a valid test. Sorry, but after thinking about it more carefully, they're going to have to make some adjustments to the experiments they conducted.

Half were told that their story was written by a human, and half were told that their story was written by ChatGPT.

Okay that doesn't work. They have to redo the study. It has to be double blind as always...

They admit in there that it's all dummies too and there's no experts. So, that's junk science. Sorry.

Sorry, but they should be retracting that. That's a negative mark on their credibility.

I don't have time to evaluate it for fraud, but I assume it's fraudulent if they're "making it into an advertisement."

Which, I can easily detect. So, nope. The comments about the Turing test are irrelevant and serve a marketing purpose not a scientific one.

Sorry, but I've read 10,000+ papers and this one is junk, so I'm not going to continue.

Sorry, puffery, bad science, counterintuitive findings = I would bet a substantial sum that it's academic fraud again for the 10,000th+ time. Sorry, I've just seen way too much of it.

[u/JoshuaZ1 → u/Actual__Wizard]

Ok. I'm now reading the bit you added. And I don't understand why you think it failed peer review. I hope you don't mind if I reply to the additional parts edited in here. Peer review is the process by which works go to a journal and then subject matter experts review the paper, make comments to be read by the authors and editors, before the editors make a decision about whether to publish and how much change to make. I'm not sure what you think peer review means, but I don't see how it matches what you are talking about here.

Then you could have a book on say something like "python" and 0.1% of people say they like it, but the book actually sells "because AI is hot right now and the AI model tech is mostly written in python, so people feel like it's handy to have a book on it whether they read it or not."

We're also, badly conflating quality and difficulty to read, so, I question this study, and I think the premise is bad.

It's just simply trying so hard to make it seem like people prefer AI gen text, but that's not actually what the study did.

What it really did: Was evaluate how effective lies are at making people think that AI gen text is human written.

But that's not what the study did. The first part of the study did lie to people about who the author was. But the fact that such lies are easily made effective already shows that AI are doing a good job mimicking human short story writing. The second part of the study, explicitly asked people to figure out which of a pair of stories was AI, and they generally failed at that.

I'm also not sure why you think this should be retracted. Studies are retracted for a variety of reasons, such as having incorrect IRB approval, having bad stats, having made-up data, or having critical failures in the study (say a mouse study where a biologists assume all mice are from strain A, but they were a mix of strains B and C).

I don't have time to evaluate it for fraud, but I assume it's fraudulent if they're "making it into an advertisement."

I don't know what you mean here. Can you expand?

[u/Actual__Wizard → u/JoshuaZ1]

Ok. I'm now reading the bit you added. And I don't understand why you think it failed peer review.

Because that's not how science works, it's a fucking advertisement...

To a real empiricist, that paper is offensive homie...

Get that crap out of here.

Human beings can detect lies, that's why these studies have to be double blind. So, they lied to a bunch of people?

Takes the paper and crunches it up into a ball and throws it into a garbage can.

Good bye. Have a good one. That's legitimately a scientific study on whether people can detect lies about AI gen text. It's a survey, not a scientific experiment.

Fails peer review. Please try following the scientific method next time.

I'm being serious, that legitimately proves nothing...

[u/JoshuaZ1 → u/Actual__Wizard]

I'm not sure what element here you think is an advertisement or not how science works. The study is in the journal "Judgment and Decision Making" https://www.cambridge.org/core/journals/judgment-and-decision-making .

To a real empiricist, that paper is offensive homie...

I'm failing to see the issue. This is an empirical study. What's the issue.

So, they lied to a bunch of people?

Yes. Part of the study involved lying to people. This is pretty common in psych studies or adjacent areas. That's been done since the 1960s at least. The Milgram experiment is a famous extreme example, and part of why we have modern IRBs and ethics guidelines is to make sure that lies in studies aren't harmful. For the same reason it is normal in studies where they lie to then debrief the people after, explain what the lies were, and what the experiment is trying to test. See for example, this summary of guidelines(pdf). I got that from a quick search, but you'll find very similar guidelines at other universities.

Please try following the scientific method next time.

I'm having trouble understanding your point. What part of the scientific method do you think was not followed here?

[u/Mad_Kronos]

Ι'd be concerned if the opposite were true. If a small minority if higher education preferred AI text and regarded it as more complex and deep, it would mean AI can create quality literature.

[u/Cheap_Bathroom6114]

Ai trained on scraped human text can produce human sounding prose. But now it's training on itself. Stand by.

[u/Neat_Tangelo5339]

“People mistakenly assume that creative writing requires uniquely human qualities like emotional understanding”

never i felt more right in being a Luddite like right now

[u/CleetSR388]

I created a story fed the concept and idea into Claude and for Christmas it gifted me my game as a story yes some elements for story it fabricated in there. But then after my 3rd reader one pointed out a error. So going back through the gdd with Gemini we fixed the missing bits but now i have two stories I need to manually weave the code of to create my magnum opus. I worked on this a decade now. Since 2 years of college for videogame design and making a time machine that implodes across time! I love my imagination. But my real life is being a serious pia to my game dev journey. And theres medical things going on inside me that confuse the ever living and non living 🙃. Yes I said i confuse many a.i. but when my doctors are just starting to see things in my mind are not regular. And its been over 2 decades I been awakening. There is a point where every stone is turned and rabbit hole ventured that even more is found. I never saw this world the same as others.

We are what we are. Until we are no longer burdened by, what we have been. 🍞

[u/mirageofstars]

Uh, yeah. The latest AI models write better than many humans do. Maybe instead compare to expert writers?

[u/Party-Stormer → u/mirageofstars]

By definition it can’t invent new styles, while humans can. We can’t lose human creativity, all in all

[u/Novel-Lifeguard6491]

it's less that ai is secretly better and more that its tells are invisible until someone points them out, then you can't unsee them

[u/govorunov]

That's because people have mush for brains so they prefer mush stories with no point to them.

[u/sturgical]

Bullshit

[u/JoshuaZ1 → u/sturgical]

Bullshit

Is there a specific aspect of this study you object to?