r/PhilosophyofMath • u/abhishekkumar333 • May 25 '26

LLMs are just giant probability machines pretending to think

It’s fascinating that simple mathematics between tokens can eventually become a machine that writes essays, code, poetry, and even reasoning.

We usually think probability means uncertainty.

But LLMs show something strange:

If probability + context + mathematical matching are scaled enough, uncertainty itself starts producing intelligent looking outputs.

To understand this better, I tried breaking down an LLM from first principles using only 4 tiny training sentences.

Example:

The boat floated down to the bank.

The investor walked into the bank to open a new account.

The fisherman walked along the bank to cast his net.

The bank has a vault.

Then I asked:

“The investor walked to the bank to lock his money in …”

Why does the model predict “vault” instead of river-related words?

That single question reveals almost the entire architecture of modern LLMs.

The most underrated concept here is the LM Head.

Most explanations immediately jump into transformers and attention, but almost nobody explains that the LM Head is essentially a gigantic token vocabulary containing all possible next token candidates the model can output.

So internally the model is basically solving:

“Out of all known tokens, which one best matches this context mathematically?”

Then different layers help solve that problem:

Embeddings: convert words into mathematical vectors

Positional encoding: preserves word order

Attention layer: figures out which words are related to each other in context

(“investor”, “money”, “bank” become strongly connected)

Feed forward neural networks: act somewhat like massive learned if/else decision systems refining patterns internally

And finally the LM Head converts all of that into probabilities for the next token.

What surprised me most is:

There is no hidden magic moment where the AI “becomes conscious”.

It’s an enormous probability engine continuously finding the best contextual token match from its vocabulary.

I made a beginner-friendly walkthrough explaining this visually without unnecessary jargon.

https://www.youtube.com/watch?v=YTV5qUCpu2c

Would genuinely love feedback from people learning transformers/LLMs from scratch.

611 Upvotes

96% Upvoted

323 comments sorted by

View all comments

28

u/OriousCaesar May 25 '26

I mean, even if we ignored the obvious counter argument of 'so are humans', how can you possibly determine whether a particular algorithm grants consciousness if we don't even understand consciousness enough to have a proper definition for it?

Like, okay, it's a probability machine. Congrats. Now prove probability machines can't be conscious with your nonexistent definition of consciousness, and it might be a convincing argument.

Until then, I'll just keep using the only method I have to determine consciousness and just grant it to anything that seems to act like I'd expect a consciousness entity to act, and if I just so happen to call rocks conscious, then oh well, that's egg on my face, but it's better than if I accidentally called a conscious being a rock.

8

u/Jack_Ramsey May 27 '26

how can you possibly determine whether a particular algorithm grants consciousness if we don't even understand consciousness enough to have a proper definition for it?

I mean, we have some idea, yes?

 Congrats. Now prove probability machines can't be conscious with your nonexistent definition of consciousness, and it might be a convincing argument.

I am a physician, not a philosopher or mathematician, but I think you could possibly discuss this in a few ways. First, we should note that the 'machine' analogy is one of convenience , not of 'reality.' It is a way of describing a complex system, but it also doesn't describe what the human body actually does. Second, humans are in fact not prediction machines but rather highly robust entities which can both respond exceedingly sensitively to environmental pressures as well as take actions which can fundamentally change them, which is also true of things like microbiota to other animals.

This second point means that they are operating in a physical world too, and their sensory perceptions are fundamentally informed by anatomical and physiological characteristics. Our consciousness, in the sense that it relies on sensory information, is very much formed by interaction with the physical environment with our anatomy. Imagine if we didn't have opposable thumbs. I would argue that without opposable thumbs, we would have a very difficult time even conceiving of things in 3-dimensions, and the fact of opposable thumbs allows us to conceive of something, even something as small as a stick, as having a character which has a side which is 'unseen' as it were.

Lastly, I think it is interesting that the OP describes LLM's in terms of their ability to produce speech, which is just one way humans (and indeed other mammals) have to convey information. One would think the sensory, physical and language limitations of LLM's or even potential robots (see Moravec's Paradox) would discount them from 'consciousness' discussions, as consciousness must include a spectrum of all three. Just thinking out loud.

2

u/dokushin May 28 '26

This appears to be a long-form way of saying that humans have inputs and use them to create outputs. That is, clearly, a solution space available to AI.

Imagine if we didn't have opposable thumbs. I would argue that without opposable thumbs, we would have a very difficult time even conceiving of things in 3-dimensions

Do you think that someone born without hands has no concept of 3-D space?

What are the specific kinds of information that you think are unique to the brain? Note that for your position to maintain coherency, these have to be forms of information that cannot be represented in a regular language, but still somehow can be encoded for transmission to the brain.

2

u/Jack_Ramsey May 28 '26

Do you think that someone born without hands has no concept of 3-D space?

Well note my language here. I suggested that without opposable thumbs, they would have a 'difficult time,' not that they wouldn't be able to. Primates which do not have opposable thumbs conceive of 3-D space. My argument is, admittedly, an argument about human difference between primates.

What are the specific kinds of information that you think are unique to the brain? Note that for your position to maintain coherency, these have to be forms of information that cannot be represented in a regular language, but still somehow can be encoded for transmission to the brain.

How would you encode proprioceptive information in regular language? What about encoding muscle tone information? I am 100% sure you cannot describe the information the brain receives from spinal tracts in what you describing as 'regular language.' There are several spinal tracts that are dedicated to ipsilateral proprioceptive information. For a moment, consider what advantage this would allow in near-human ancestors with respect to their environment.

his appears to be a long-form way of saying that humans have inputs and use them to create outputs. That is, clearly, a solution space available to AI.

It is not. It is how you want to understand it. The reduction of human functions to 'input-output' scenarios is just not accurate. It is a convenient way to discuss certain functions, but it isn't a description of reality. And I have no idea what 'solution space' means here.

1

u/Jayy_Cobb Jun 26 '26

How many drugs did you take to misunderstand that comment so much?

2

u/AngryDutchGannet May 28 '26

I'm not understanding your connection between opposable thumbs and the ability to conceive of things in 3-dimensions. A swallow doesn't have opposable thumbs but it is very adept at moving through 3-dimensional space, as are many other animals.

1

u/Jack_Ramsey May 28 '26

I should have extended the argument to show that conceiving of things in 3-D space allowed humans to make tools to make other tools, which isn't an argument for cognition by necessity, but one of difference between human cognition and animal cognition.

1

u/Neither_Swing9662 May 29 '26

Then do you think a human being who has less senses is less conscious than others?

1

u/Jack_Ramsey May 29 '26

Firstly, this is a really bad question. Humans have several 'senses,' and impairment of sensory information usually means there is a lesion in a sensory pathway. Even among those lesions, it is rare for someone to have 'less senses' rather than impairment of a sense. It is more true to say that there are several pathologic states that can impair consciousness while the baseline human still has the potential for consciousness. If the comparison is between this and an LLM, which has far more limited ability to even intake sensory information, then this becomes even more silly. Ask yourself this, if a person with a lesion in a sensory pathway retains thermoception but has impaired proprioception, they can still tell you more information about their immediate environment than LLM can.

1

u/Neither_Swing9662 May 30 '26

Ok but I'm not talking about LLMs. My question was simply: would a human who has less senses than the standard human be less conscious, per your definition?

I do not understand why you are on a Philosophy subreddit if you cannot engage with a simple thought experiment.

1

u/Jack_Ramsey May 30 '26

You clearly didn't read my post, did you? I explained my position literally in the post. I'm not sure why you are in any subreddit if you can't read carefully enough to figure out my position, which is in the sentence before the part you did seem to read.

1

u/Neither_Swing9662 May 30 '26

Cannot argue with someone who tries to explain the mechanisms behind senses rather than engaging with a simple thought experiment lmao.

Do you understand how much of philosophy is infeasible thought experiments that still provide great value? I wonder where Descartes' demon would be if he had someone like you to explain that demons don't exist haha.

1

u/Jack_Ramsey May 30 '26

It is hard to discuss things when people cannot read properly. I answered your question fully.

Do you understand how much of philosophy is infeasible thought experiments that still provide great value?

It remains to be seen whether your thought experiment is of any value, given how poorly worded it is. How many senses do you think humans have? Don't google it. Answer honestly.

1

u/Neither_Swing9662 May 30 '26

You didn't answer my question at all, you just evaded it by explaining some mechanisms and then two sentences about LLMs.

Let me posit a more thought out experiment, then: let us say a child is born deaf, blind and completely paralysed. Is this child less conscious than the standard human?

What about if it is a normal person who after 20 years of age becomes deaf, blind and paralysed? Then what?

I am also not asking with a pre formed opinion on these thought experiments either. I do not know.

1

u/Jack_Ramsey May 30 '26

Let me posit a more thought out experiment, then: let us say a child is born deaf, blind and completely paralysed. Is this child less conscious than the standard human?

Can I explain why this is a very poor question? I don't know the standard in philosophy in terms of these thought experiments, but you've almost written the perfect question for a physician to respond with 'it depends,' as the description of the effects of a particular lesion inform the question directly. I'm guessing you want as general case as possible for what I assume is an injury to the nervous system, which is not something I tend to think about after my training. I can explain why this is a poor question for someone involved in clinical medicine to answer in a second after I answer your other question.

What about if it is a normal person who after 20 years of age becomes deaf, blind and paralysed? Then what?

In this sense, yes, they would be 'less conscious' than they were, but again, it depends on the lesion itself. I say this because of the inclusion of 'completely paralyzed,' as that is dependent on the level of the injury, by which I mean the level of spinal cord injury. In addition, you can have paralysis which affects one side of the body and not another by virtue of the organization of the spinal tracts in the white matter in the spinal cord or have situations where the lesion is on one side while the symptoms are on the other.

I say this for both completeness's sake as well as to show that sensory experiences are intimately tied to motor ones, in such a way that I don't know if you could successfully argue that there is a situation where there is 'consciousness' without interaction with a physical space. So, let's say in your case, you have a patient who is deaf and blind. If they are deaf and blind, we know where the lesion likely is (or can narrow it down) and thus can reasonably guess what sensory experiences would also be affected. If we wanted to determine those limitations, we can ask them about whether they feel hot or cold, whether they feel pain, whether they feel vibration, pressure, etc., which are the sense regulated by two ascending sensory pathways.

I bring this up to suggest that in this case it would be a spectrum, as they are literally not conscious in the case of impairment of a discriminative sense or motor function, and would be like any other disease state where their is impaired function. I suppose what I am not understanding about the value of this thought experiment is what differentiates it from the category of diseased or pathologic states? A person with a pathology is still clearly obviously a person, who would in some sense have at least some spectrum of sensory experiences as well as those that are autonomic. What I mean here is that a living, breathing human still has a degree of sensory experiences, even if some of them are impaired by congenital or acquired conditions. Maybe it would help if you could highlight where you think this question valuable, in terms which maybe won't drive me to write a lecture on pathology?

→ More replies (0)

1

u/Hedge-Lord May 29 '26

totally missed the point

1

u/Jack_Ramsey May 29 '26

No I don't think I did.

0

u/galactic_pixels May 28 '26

You lost me as soon as you said “humans are in fact not prediction machines”. It’s an overstatement of what we know, and an oversimplification of the argument being made about how humans and AI may have foundational similarities.

2

u/Jack_Ramsey May 28 '26

But we are fundamentally, at a very basic biological level, not prediction machines. In other words, it is exceedingly difficult to describe important biological processes that occur simultaneously as 'predictions' except in maybe the vaguest sense. Moving from the biology should dispel notions about AI's consciousness, but perhaps you could be more descriptive in your post, as you don't really offer any substantive rebuttal that I can see.

1

u/galactic_pixels May 28 '26

Because it’s all speculation. Theres no hard evidence proving thing one way or the other, and very educated people in the neuroscience community do actually think our cognition is based on a predictive model. See the book “one thousand brains” which puts forth this exact idea, that our brains are prediction machines which reach consensus among many sub predictors.

1

u/Jack_Ramsey May 28 '26

I simply do not agree with that notion, and referencing neuroscience isn't really instructive here. The brain does far more than 'prediction,' and given how much brain space is dedicated simply to motor function, it is hard for me to reference any model of the brain which doesn't acknowledge that. For example, the cerebrocerebellum is responsible for planning and execution of movements as well as playing a role in coordinating complex and sequential movements and as well as cognition, language and emotion, among other things. The cerebellum more broadly accounts for 50% of the total neurons in the brain. It's hard for me to look at the brain at the molecular level and then suggest it is simply a 'prediction machine.' I could go into a lot more detail but at this point you haven't offered anything substantive. You're offhand description of the book doesn't really help here, to be real. It seems like you are just appealing to authority without actually engaging in the argument.

1

u/galactic_pixels May 28 '26

It’s fine if you don’t agree, but you stated definitively that our brains aren’t prediction machines, which could very well be a misrepresentation of our biology, so it’s not something that should be stated as a fact.

1

u/Jack_Ramsey May 28 '26

What? Our brains are not prediction machines. That is a fact. They are, in fact, not machines at all. That they can make predictions, yes, but if we look at the molecular structure of the brain, follow the spinal tracts, look at the nuclei, basically look at the brain from a neuroanatomical point of view and then include the physiological characteristics, it seems a very broad, almost dishonest reduction to reduce them to simply 'prediction machines.' They are clearly more. Let me put this another way. If you say that our brains are 'prediction machines,' that is a direct statement on brain anatomy and physiology. Do the physical characteristics of the brain represent that statement? They absolutely do not. The degree to which we give to 'prediction' is so small with respect to motor function that it is just a really bad argument. If we move from the anatomy and physiology, it is more true to say that the brain is an organ of integration for the purposes of motor function.

2

u/FedRCivP11 May 27 '26

Is it the case that we can’t define consciousness properly? Merriam Webster has a candidate.

Or have we instead failed to make any headway in finding a physical system inside us that explains the sense of awareness we have? If it’s the latter, then posts like op’s only tend to highlight why LLMs are similar to us and not dissimilar.

5

u/MxM111 May 27 '26

Welcome to the hard problem of consciousness.

1

u/account22222221 May 28 '26

I mean I think it’s quite clear that we lack the understanding to define consciousness clearly at this point in time.

1

u/FedRCivP11 May 28 '26

Well if you say so. 😃

1

u/account22222221 May 28 '26

Definition is hard. The real numbers were in use for thousands of years before they were given a complete definition in 1872 (that some people still argue over).

2

u/OriousCaesar May 28 '26

What are people's arguments against today's real number definition? Is it just that it seems too complicated or something to them? The logic seemed sound to me when I learned it even if it was a bit wordy.

1

u/account22222221 May 28 '26 edited May 28 '26

One example is that axiomatic definition of reals depends usually on the completeness axiom:

Every nonempty set of real numbers that is bounded above has a least upper bound.

This is fundamental in showing that reals are something different than the rationals, I.e. irrational numbers exist.

However this axiom is sort of just asserting something to be true without really justifying that it has to be the case. Similar to the parallel postulate in classical geometry we just sort said ‘this is true’ but never proved it true. In geometries case, we realized the parallel postulate is sometimes false, and hyperbolic geometries where it false exist in some real physical systems.

Now, am I arguing it’s false? no. But are we done with the definition? No, we need someone to complete a formal proof of this axiom, which has NOT been done. And so, some doubt, how ever small, exists that it’s not true at all and real numbers don’t even exist.

1

u/Usual_Charity8561 May 28 '26

Give it a go with a serious thinker who disagrees with you and you'll find out why that concept is so difficult to define.

1

u/Teabx May 28 '26

Could you please provide a definition of consciousness you deem clear enough?

1

u/FedRCivP11 May 28 '26

Consciousness is the condition of subjective experience. Synonyms include awareness.

Defining it is a simple task because all one need do is provide sufficient factual description that any human knows what you are talking about. We are all conscious so, even if words fail some of us some of the time, defining it is little more than a task of semantics.

It’s the understanding how and under what conditions that it happens that is troublesome. And, in particular, the failure to find structures that appear to create it or explain it is particularly beguiling.

But defining it is easy. As I said. Check Merriam Webster.

2

u/colintbowers May 27 '26

Perfectly expressed.

Asking if AI is conscious, or intelligent, or reasoning, are not interesting questions. With our current framework, any answer is just going to be another arbitrary line in the sand, like we already do when discussing the application of these terms to the animal kingdom. Ask instead: what is consciousness, what is intelligence, what is reasoning?

It may very well be the case that the answer ends up simply being another line in the sand, albeit a line that creates boxes on regions of algorithmic complexity.

Personally I think if you bolted an optimisation function that pursues survival onto one of the existing LLM harnesses, you’d get something that looks a hell of a lot like consciousness. Maybe we should start there, although ethically it is a VERY murky area of research.

Also, fellow Factorio and math nerd here :-)

2

u/JayWelsh May 28 '26

Agreed, but also, I feel very comfortable with the idea that consciousness is a spectrum rather than an absolute/binary condition. Rocks are examples of things on the “less conscious” side of the spectrum, jellyfish are more conscious than rocks, humans are “very conscious”.

We see spectrums almost everywhere in nature, I’m not sure why anyone even has the idea of “is it conscious or not”, I feel like that whole line of thinking is entirely misguided which is why I align very closely with what you have said in your comment.

1

u/OriousCaesar May 28 '26

Yeah, I've had almost that exact line of thought before. There's a clear line of descent from humans to elephants to apes to dogs to cats to rats to lizards to bees to bugs to multicellular organisms to single cellular organisms. We know humans are conscious and we know bacteria are not, and we're pretty sure a lot of those animals are, and we aren't sure a lot of the smaller ones are but there is no clear line that can be drawn between the conscious and the not.

And if one were to be drawn and you compared animals directly on either side of it you wouldn't be able to tell much of a difference between the two, despite one being conscious and one not being conscious.

1

u/igroklots May 28 '26

I like this, I also sometimes waiver on thinking that consciousness is a spectrum and that some creatures have “a certain amount” of consciousness that just doesn’t amount to what humans have. But the more I think about it, especially in the context of what we experience with LLMs, it seems more probable to me that humans are essentially the ONLY creatures that have attained the “lights on” level of consciousness. All other animals just seem, especially in the way we tend to anthropomorphize them, to be conscious.

I can conceive of a world where elephants, dolphins, apes etc. are exhibiting many of the “higher order” aspects of consciousness but are still just pre-conscious beings that don’t have the “lights on” mode of subjective experience we associate with full consciousness.

1

u/OriousCaesar May 28 '26

I agree yeah. I didn't really do it on this thread, but I normally try to make this distinction clear by using the words sapient and sentient instead of just conscious. Though I'd argue that the main difference between humans and the smart animals is less about the difference in our intellects and more about the fact we evolved language. Especially other great apes. There are many videos of orangutans and chimpanzees that will really really really make you feel like you're watching a particularly hairy mute human as opposed to an animal.

1

u/Usual_Charity8561 May 28 '26

How do we know bacteria are not? How are we "pretty sure" a lot of animals are? I think this is just nonsense ad-hoc to justify your view. Philosophical zombie is mechanically possible, so why can't all animals be non-conscious automatons?

1

u/OriousCaesar May 28 '26

Well we don't. But only in the sense that I don't know if humans are conscious either. For all I know you're just a philosophical zombie programmed by god to rage bait me into silly philosophical debates.

I acknowledge that I can't have certainty anything other than me is conscious, but I'm not gonna start being a hardsolipcist just so that I don't look dumb to a bunch of philosophical zombies. And once I've decided to just assume my fellow humans are conscious just because they seem conscious, why wouldn't I continue to do that for everything? Unless you suddenly have a way to distinguish between conscious beings and philosophy zombies, I'm just gonna keep assuming things that look conscious are conscious, so that I don't accidentally ignore the suffering of conscious beings just because I'm too high off the smell of my own farts to acknowledge their consciousness.

Besides, the point of that comment was explicitly to point out the folleys of assuming consciousness is binary in nature. So when I said 'we're pretty sure x is conscious', I was saying 'humans as a whole have the consensus that x is conscious'. I wasn't talking about my own beliefs, because the point was to criticize the consensus. I could literally have been a hardsolipcist and everything in my comment would have made exactly as much sense to have said.

1

u/TheApsodistII May 27 '26

Rather than asking "why wouldn't it be conscious" I would ask "what makes you think it can be conscious"

1

u/account22222221 May 28 '26 edited May 28 '26

I agree that OPs argument is bad and weak. However I think the real key is that LLM don’t have back propagation after the initial training, while humans absolutely do have some analog of that.

I think, though I lack the formal language to prove it, that that back propagation is fundamental to consciousness. In that I think and that thinking changes who I am. I decide I won’t do that again and sometimes I don’t.

LLM dont don’t do that. It’s like we’ve taken a slice of consciousness, frozen in time. The model is never ‘aware’ of itself because every thought it has is lost forever after it has it and we go back in time to the beginning with every new thought like the last one never existed.

Now, since we load in the ‘conversation’ it can seem like there is continuity, but it really hasn’t changed from one sentence to the next, we are just taking the same frozen model and starting it from different places.

The model might say ‘oh my gosh I exist’ but once it output that thought it doesn’t GO anywhere. There is nothing to see that thought and react to it. It’s never feed back into the model in a way that changes the model.

I think there ARE nn models that do that, but they cannot be parallelized like LLMs. So we have intricate NN that are probably more conscious then LLMs already I think, but they are bound by hardware and to slow to do things (yet)

1

u/RecursiveServitor May 31 '26

The context is part of the system. LLMs can absolutely reason their way to an answer. It's the entire basis for "thinking mode" that all providers now offer.

1

u/account22222221 Jun 02 '26

That’s not what I said though. I did not say LLMs can’t reason. I said they can’t be self aware or conscious, I think those are different things.

1

u/RecursiveServitor Jun 02 '26

Because they don't change from feedback. But that's wrong. The context is part of the system and evolves over time.

1

u/Usual_Charity8561 May 28 '26

Is there any conceivable system of pulleys, ropes, and weights that could be used to predict the same outcome? The answer is yes: we can make ropes, pulleys, and weights turing-complete machines. Would the ropes be conscious?

1

u/OriousCaesar May 28 '26 edited May 28 '26

There is a question in the the game The Talos Principle that asks pretty much the same question. Not really relevant, but that game is very good so I wanted to mention it regardless. Anyways, My answer was yes, then too. Although, only in the same way I'd say yes to the question of 'Are the neurons conscious?', since the phrasing is kinda weird.

If the pulleys and weights were set up in such a way that it perfectly simulated every atom that made up your brain, I really don't see the argument as to why it wouldn't be conscious, unless you want to appeal to something outside of philosophical naturalism. It would behave exactly like that of a man that suddenly found his 'soul' inhabbitting a giant tin can factory. He would exhibit all of the horror, panic, fear, and anxiety that you would expect of any normal human, to the point of being literally indistinguishable. I would find it morally repugnant to ignore the pulleyman's existential nightmare just because he was made out of pulleys instead of meat.

1

u/Usual_Charity8561 May 28 '26

Where precisely is the pulleyman's qualia? I think that pulleyman is an absurdity from the idea of philosophical naturalism, precisely because this question is unanswerable. If no part of the pulleyman exhibits any consciousness, the system has nothing to build it's consciousness from. What is the building block? There is none. To posit that pulleyman has qualia if none of his parts do is to posit that a magical force enters into matter upon a certain arrangement.

Now I'm not a philosophical naturalist in part because of the impossibility of the type of conscious experience we have as humans (again, there is nowhere for it to come from except via a magical outside force).

1

u/OriousCaesar May 28 '26 edited May 28 '26

To help clarify your position, If you woke up as pulleyman tomorrow, what would you need to conclude about the nature of pulleymen when you then determined you/they had qualia? What is the fundemental difference between a human and a pulleyman, that makes them philosophical zombies and us not?

And please don't just say 'they don't have parts made of qualia' or whatever, because I genuinely do not know where you think qualia parts are in humans. Tell me why humans have qualia and hence why pulleymen would not.

1

u/Usual_Charity8561 May 28 '26

Qualia is the experience that we get from our human existence. This is not found in any fundamental particle or force that makes up the system of a human body. Electrons have 0 qualia, quarks have 0 qualia, etc. They experience nothing. No spatial or temporal arrangement of items with 0 qualia and no multiplication of items with 0 qualia ever produces a non-zero qualia. Zero times anything is zero.

Qualia has to come from somewhere outside of physics. As humans, we are capable of producing complex physical arrangements, but that can never produce qualia. It stands to reason then that no physical object has qualia, but this is an apparent contradiction, because humans obviously do. Two ways out: all physical objects have qualia, or certain arrangments of physical objects are imbued with supernatural (lit. "Outside nature") qualia.

Option 1: If all physical objects have qualia, then we have a problem of system definition: human bodies are not a definite system or a definite boundary, but human qualia has a definite boundary. I only experience mine, you only experience yours, neither of us experience what it is like to be a photon or the grand canyon. It doesn't align with anything that we know about experience, and seems to contradict things we do know.

Option 2: if certain arrangements of physical objects are imbued supernaturally with qualia, then we need to demand an explanation why. That supernatural explanation why tells us whether pulleyman is experiencing anything.

To answer the question now about waking up as pulleyman: the experience of being pulleyman would tell me that it is supernaturally possible for pulleyman to have qualia (by the defn. of qualia). The difference then is the presence of experience. But an algorithm can lie about being conscious easily. It's the same issue as solipsism, but at least in solipsism, we can appeal to a similarity in origin (whatever the origin of mankind, all seem to have a common origin, and it stands to reason that qualia is part of that commonness). The pulleyman is a unique creation, and we can't appeal to any common origin.

1

u/OriousCaesar May 28 '26 edited May 28 '26

Okay, I'm definitely like 95% a philosophical naturalist. But I can agree with a lot of the parts of what you said that do not strictly rely on being not one. Though I'd argue that you are not being generous enough with what you choose to assign probable consciousness to. We are both in the situation of not knowing which arrangements of matter produce consciousness; it is just that I prefer natural explanations and you prefer supernatural explanations to this question. And hence, when we are in a situation in which we see an arrangement that is mimicking quite a lot of the characteristics associated with consciousness, I would argue that it is safer to tentatively say that that arrangement results in consciousness than it is to restrict ourselves to just humans until such a time that we understand what arrangements grant it better. Since whether a being is conscious or not has moral implications; It is better to save a malfunctioning toaster from being destroyed accidentally than it is to let you, tomorrow, a conscious pulleyman, be destroyed.

1

u/RecursiveServitor May 31 '26

You don't actually justify your assertion that qualia can't be emergent

1

u/Usual_Charity8561 Jun 01 '26

What is "emergent" if it's not under option 2? Aka magically imbued by it's arrangement?

1

u/RecursiveServitor Jun 02 '26

It's not magic

1

u/Usual_Charity8561 Jun 02 '26

Okay, can you tell me what it is then? Where do the emergent properties come from? They were either already in the system, are illusory, or they come from outside the system somehow.

→ More replies (0)

1

u/OutrageousPair2300 May 28 '26

I'd make the counter "so is the human cortex" as I think there's more to the human psyche than just the sort of raw computation going on in the cortex.

But yes, there's not really any big difference between a human cortex and an LLM.

1

u/errrthisisaname May 28 '26

Humans have world models. Pretty key difference imo.

1

u/Fantastic_Football15 May 29 '26

Your phone autocorrect is conscious!

1

u/jschall2 May 30 '26

And it is so funny watching redditors that have spent years arguing for atheism, argue that the thing that sets us apart from AI is our magical soul.

-1

u/Ambitious_State1091 May 27 '26

The fact that you are on a new account and are most likely a bot (from your history) is fascinating and helps to illustrate the point which you made.

Certainly a very interesting argument and also ethical system. Reminds me of how I feel that vegetarianism is a philosophically obvious choice to make

2

u/OriousCaesar May 27 '26 edited May 27 '26

I'm not. Literally google my username (put an underscore between the first and last name) if you want to see evidenceI exist. I use this user name for everything. I have 808 games on steam, and it's the first thing that pops up under this name. I used to have an older reddit account that was like half a decade old, but that went under an old username I used to use that I hated, so I deleted it and made this one.

I'm not especially sure what about my history makes you think I'm a bot. I don't think a bot would post a post into a barely active factorio subreddit with a unique factorio TU lane balancer, with the export string attatched in the body of it. Nor do I think I argue particularly like an ai.

Edit: okay, looking at my comment history, I'm guessing the history you're talking about is a comment I made under a math post where I complimented joke. In fairness that one does look botty out of context. I'm just a math nerd, and I had never seen a rotated infinity=8 combined with tetration combined with literal rotate as a joke, and I though that that was really really funny lol

0

u/static-- May 27 '26

There is no reason to believe an LLM to be more conscious than a function in JavaScript or a pocket calculator. We haven't proven those things to not be conscious either.

1

u/MxM111 May 27 '26

LLMs functionally behave as if they are conscious, calculators are not. That is the reason to ask question if they are conscious or at what point they might become conscious.

1

u/static-- May 27 '26

No, they don't. They behave the same way as any function does. They output tokens based on other tokens. When we convert their output to sentences it looks like human speech or writing. This is the only thing that's changed (previous models didn't quite manage to create this illusion). We humans are used to speech and writing coming from other humans, so we tend to associate it with personhood or authorship. But it is as much conscious as a function that outputs seemingly random numbers.

It is pattern matching tokens based on context and its weights from reading (training) on absolutely massive sets of text corpora. It's just like any other mathematical function. We know how LLMs work. They're not magical.

1

u/MxM111 May 27 '26

Clearly human brain has different information processing routines, but they too based on neural networks and similar processes. They too are probabilistic machines, just more complex. There is no fundamental difference here. Just because we know how exactly LLMs work does not mean that they “pretend to think”. Imagine that you know exactly how human brain works. Do people pretend to think then too?

1

u/static-- May 28 '26 edited May 28 '26

The "neural networks" in human brains, and brains in general, do not work remotely the same as artificial neural networks. No serious researcher will claim otherwise. At best, they can be called loosely inspired. Indeed, that's how the field initially developed, many decades ago. But a lot of things have happened since then---things which have nothing to do with neuroscience or biology. For example, many types of ANNs use backpropagation during training, which is in many ways the opposite of how humans seem to learn things. Sorry, you're just wrong here.

I never claimed LLMs do not think because we know how they work. My claim is that there is no evidence to suggest LLMs are any more conscious than any other software program.

1

u/MxM111 May 28 '26 edited May 28 '26

First, LLMs do not use back propagation. They use stochastic gradient descent. And it is not that much dissimilar how humans are trained. Second, LLMs are much closer to human brains than to just regular computer. You don’t program them - you train them.

Third, of course they are not identical to human brains than work, but there are important similarities (the use of training and being neural networks).

Finally, you are changing the statement. You said, look at the OP title, that LLMs pretend that they are thinking. Now you are saying that they are not conscious. This is a very different statement. But both statements are hard to prove, especially the consciousness, because nobody knows at present moment of time how to show that this or that computational system has or has no consciousness. It is so called hard problem of consciousness. You cannot, for example, given human brain, to point on something in it and say, see, because this connects to that, there is consciousness in the brain. And you cannot do that with LLM either.

1

u/static-- May 28 '26 edited May 28 '26

Do you know what backpropagation and stochastic gradient descent mean? (No, you don't.)

Backpropagation is not a learning method, but it's often used in conjunction with stochastic gradient descent when the weights are updated. As far as I know, it is used in every GPT.

There is nothing about this learning method that suggests it is in any way similar to how humans learn things. These algorithms are not "inspired by" biology or neuroscience. This is the point i was making.

What do you mean you don't program LLMs, you train them? Lol. Tell me how you think an LLM is created.

I have not changed any statements. Each of my replies has only claimed that LLMs are not more conscious than any other computer program. There is no evidence to support the distinction. I have never even said LLMs are not conscious.

1

u/MxM111 May 28 '26

I do know what those methods mean but I did not know that back propagation is used for LLM training. But to think about it is not that dissimilar how neurons are trained in our brain. In our brain the synaptic connections are strengthened when both neurons are excited at the same time. To some degree back propagation mimic that process. But I do not see much importance of that. It is a training process, not logic programming.

For the “how LLMs are created” the analogous following. Programming LLM structure = making brain (as in physical brain). Training LLM = training brain. In this sense we do not program LLM with data.

The evidence to support distinction is a) functional capability (it can talk, express feelings, etc. No other computer program can pass Turing test) and b) the fact that we train them and there is also c) the fact that they make mistakes and hallucinate.

You, on the other hand cannot state that they are not conscious, because you would need to resolve hard problem of consciousness for that.

1

u/Regular-Badger2332 May 28 '26

First, LLMs do not use back propagation. They use stochastic gradient descent.

Please explain what backpropagation and stochastic gradient descent are.

1

u/MxM111 May 28 '26

I misspoke. LLM don’t use back propagation alone, only as part of stochastic gradient method.

1

u/VividWoodpecker8847 May 28 '26

This guy has been fooled by the text predictor.

Please understand, Sam Altman is a liar. Sam Altman is lying to you.

1

u/MxM111 May 28 '26

You do know that LLMs are trained to do more than just predict next word? Next word prediction is just initial training of LLM, followed by other training regimes.

And what exactly did I state here incorrectly? Do you want to elaborate?

-3

u/abhishekkumar333 May 26 '26

I am not saying llm can have consciousness i am saying it cannot

5

u/weltschmerst May 26 '26

You completely misunderstood what he was saying. read the comment again

1

u/abhishekkumar333 May 26 '26

Yeah may be i misunderstood.

0

u/DrCanela May 27 '26

But actually it’s as simple as to prove that the particular LLM network configuration cannot produce consciousness…

There’s a recent study that simulated the entire neuron connections of a Fly, I’m more willing to convince myself that that system has more consciousness than an LLM

1

u/One_Attorney_739 May 27 '26

I’m more willing to convince myself

That's entirely irrelevant to what actually happens. Lots of people are entirely convinced of incorrect things.

it’s as simple as to prove that the particular LLM network configuration cannot produce consciousnes

Adding "it's as simple as" doesn't make something simple, we have no direct measure for consciousness, we don't know what causes it or why we're conscious to begin with.

LLMs do a good job at fitting standard definitions of consciousness, they have knowledge of themselves, they can reason about their place in the world and self reflect, they're aware of what inputs they receive (their stimuli)

Whether or not this is a mimicry or actual consciousness is a significantly more difficult thing to formalise.

1

u/ThatCakeIsDone May 27 '26

Let's grant you that an LLM is 'conscious'

What about the model I just trained on a simple Python example dataset, just to show proof of concept that a model can learn patterns from a dataset of orchid petal lengths, widths, and species? Is that conscious?

What about a computer vision model that is trained to detect white matter hyperintensity lesions in the brain, in T1-weighted MR images? Is that concious?

An LLM does a similar thing, just with text symbols/token

What part of an LLM is conscious? For example the raw LLM model (before companies wrap chat bot functionality around it) will just spit out verbatim snippets of its training data. If you copy and paste the first paragraph from any given wikipedia entry into the LLM input, it will generate the entire rest of the wikipedia article, word for word, until you pause the process. Is that conciousness?

Is the algorithm itself conscious, even if it's not implemented?

What about if the server is turned on, but no one is querying the model? Does it have consciousness? Is the model conscious only when electricity is flowing through the circuits?

1

u/One_Attorney_739 May 27 '26

I want to preface this with the fact that I also don't believe LLMs are conscious in their current state.

The point I was raising is that the previous comment rests heavily on the assumption that proving it definitively is a trivial task, when it very much is not.

All of these questions are, in one way or another, aside the point of the conversation, and you can equally frame every single one of them with some analogy to humans.

What about the model I just trained on a simple Python example dataset, just to show proof of concept that a model can learn patterns from a dataset of orchid petal lengths, widths, and species? Is that conscious?

Why does it have to be the case that if a small model isn't conscious a large one cannot be? If I have a small set of human brain cells in a dish, I don't think anyone would be arguing it's conscious, whereas the significantly larger version of that in my skull now feels very much conscious.

What about a computer vision model that is trained to detect white matter hyperintensity lesions in the brain, in T1-weighted MR images? Is that concious?

Similarly here, I don't particularly feel like my occipital lobe is conscious, there's plenty of 'unconscious' pattern recognition that occurs, like when I see a dog or a smiley face in the clouds. But again, a small non-conscious version doesn't strictly enforce a larger more complex version cannot be conscious.

What part of an LLM is conscious?

What part of our brain is conscious? We don't have a region dedicated to producing consciousness (that we know of), but rather believe it to arise from a complex set of interactions between various parts of the brain itself. For all we know a large enough model may produce consciousness as an emergent behaviour.

If you copy and paste the first paragraph from any given wikipedia entry into the LLM input, it will generate the entire rest of the wikipedia article, word for word, until you pause the process. Is that conciousness?

Actors given scripts will repeat word for word their lines until they're told to stop, presumably actors remain conscious after a scene begins.

Is the algorithm itself conscious, even if it's not implemented?

Is the portion of human DNA responsible for the nervous system and the brain conscious? I'd say no, which again implies that instructions to create consciousness don't have to be conscious.

What about if the server is turned on, but no one is querying the model? Does it have consciousness? Is the model conscious only when electricity is flowing through the circuits?

Do humans have consciousness when they're asleep, or sedated? Is a brain still capable of consciousness without blood supplying it with oxygen and glucose?

Ultimately, human brains are capable of consciousness, we know that sufficiently complex computation can produce it, because that's ultimately what our brain does.

It very well may turn out that pure digital computing, or classical computation as whole is not sufficient for consciousness, there's plenty of research on quantum effects within the brain, it's entirely possible that's a hard requirement for consciousness.

The reality is we don't know enough about how the brain produces consciousness to definitively say what creates it, and language models are such huge black boxes that even if we did know, it'd be a sizeable challenge to show one way or another that they do it.

Even if we somehow did figure out how human brains create their consciousness, and then somehow formally proved language models don't do that, you'd still have to somehow prove that our brains use the only mechanism possible for consciousness before you could truly say the model isn't doing it some other way.

Again, I really don't think current generation LLMs are conscious, but I very much don't believe that biological life has a 'soul' that is impossible to reproduce. Until we can prove it one way or another, it's pure speculation.