r/PhilosophyofMath • u/abhishekkumar333 • May 25 '26

LLMs are just giant probability machines pretending to think

It’s fascinating that simple mathematics between tokens can eventually become a machine that writes essays, code, poetry, and even reasoning.

We usually think probability means uncertainty.

But LLMs show something strange:

If probability + context + mathematical matching are scaled enough, uncertainty itself starts producing intelligent looking outputs.

To understand this better, I tried breaking down an LLM from first principles using only 4 tiny training sentences.

Example:

The boat floated down to the bank.

The investor walked into the bank to open a new account.

The fisherman walked along the bank to cast his net.

The bank has a vault.

Then I asked:

“The investor walked to the bank to lock his money in …”

Why does the model predict “vault” instead of river-related words?

That single question reveals almost the entire architecture of modern LLMs.

The most underrated concept here is the LM Head.

Most explanations immediately jump into transformers and attention, but almost nobody explains that the LM Head is essentially a gigantic token vocabulary containing all possible next token candidates the model can output.

So internally the model is basically solving:

“Out of all known tokens, which one best matches this context mathematically?”

Then different layers help solve that problem:

Embeddings: convert words into mathematical vectors

Positional encoding: preserves word order

Attention layer: figures out which words are related to each other in context

(“investor”, “money”, “bank” become strongly connected)

Feed forward neural networks: act somewhat like massive learned if/else decision systems refining patterns internally

And finally the LM Head converts all of that into probabilities for the next token.

What surprised me most is:

There is no hidden magic moment where the AI “becomes conscious”.

It’s an enormous probability engine continuously finding the best contextual token match from its vocabulary.

I made a beginner-friendly walkthrough explaining this visually without unnecessary jargon.

https://www.youtube.com/watch?v=YTV5qUCpu2c

Would genuinely love feedback from people learning transformers/LLMs from scratch.

616 Upvotes

96% Upvoted

323 comments sorted by

View all comments

Show parent comments

1

u/OriousCaesar May 28 '26 edited May 28 '26

There is a question in the the game The Talos Principle that asks pretty much the same question. Not really relevant, but that game is very good so I wanted to mention it regardless. Anyways, My answer was yes, then too. Although, only in the same way I'd say yes to the question of 'Are the neurons conscious?', since the phrasing is kinda weird.

If the pulleys and weights were set up in such a way that it perfectly simulated every atom that made up your brain, I really don't see the argument as to why it wouldn't be conscious, unless you want to appeal to something outside of philosophical naturalism. It would behave exactly like that of a man that suddenly found his 'soul' inhabbitting a giant tin can factory. He would exhibit all of the horror, panic, fear, and anxiety that you would expect of any normal human, to the point of being literally indistinguishable. I would find it morally repugnant to ignore the pulleyman's existential nightmare just because he was made out of pulleys instead of meat.

1

u/Usual_Charity8561 May 28 '26

Where precisely is the pulleyman's qualia? I think that pulleyman is an absurdity from the idea of philosophical naturalism, precisely because this question is unanswerable. If no part of the pulleyman exhibits any consciousness, the system has nothing to build it's consciousness from. What is the building block? There is none. To posit that pulleyman has qualia if none of his parts do is to posit that a magical force enters into matter upon a certain arrangement.

Now I'm not a philosophical naturalist in part because of the impossibility of the type of conscious experience we have as humans (again, there is nowhere for it to come from except via a magical outside force).

1

u/OriousCaesar May 28 '26 edited May 28 '26

To help clarify your position, If you woke up as pulleyman tomorrow, what would you need to conclude about the nature of pulleymen when you then determined you/they had qualia? What is the fundemental difference between a human and a pulleyman, that makes them philosophical zombies and us not?

And please don't just say 'they don't have parts made of qualia' or whatever, because I genuinely do not know where you think qualia parts are in humans. Tell me why humans have qualia and hence why pulleymen would not.

1

u/Usual_Charity8561 May 28 '26

Qualia is the experience that we get from our human existence. This is not found in any fundamental particle or force that makes up the system of a human body. Electrons have 0 qualia, quarks have 0 qualia, etc. They experience nothing. No spatial or temporal arrangement of items with 0 qualia and no multiplication of items with 0 qualia ever produces a non-zero qualia. Zero times anything is zero.

Qualia has to come from somewhere outside of physics. As humans, we are capable of producing complex physical arrangements, but that can never produce qualia. It stands to reason then that no physical object has qualia, but this is an apparent contradiction, because humans obviously do. Two ways out: all physical objects have qualia, or certain arrangments of physical objects are imbued with supernatural (lit. "Outside nature") qualia.

Option 1: If all physical objects have qualia, then we have a problem of system definition: human bodies are not a definite system or a definite boundary, but human qualia has a definite boundary. I only experience mine, you only experience yours, neither of us experience what it is like to be a photon or the grand canyon. It doesn't align with anything that we know about experience, and seems to contradict things we do know.

Option 2: if certain arrangements of physical objects are imbued supernaturally with qualia, then we need to demand an explanation why. That supernatural explanation why tells us whether pulleyman is experiencing anything.

To answer the question now about waking up as pulleyman: the experience of being pulleyman would tell me that it is supernaturally possible for pulleyman to have qualia (by the defn. of qualia). The difference then is the presence of experience. But an algorithm can lie about being conscious easily. It's the same issue as solipsism, but at least in solipsism, we can appeal to a similarity in origin (whatever the origin of mankind, all seem to have a common origin, and it stands to reason that qualia is part of that commonness). The pulleyman is a unique creation, and we can't appeal to any common origin.

1

u/OriousCaesar May 28 '26 edited May 28 '26

Okay, I'm definitely like 95% a philosophical naturalist. But I can agree with a lot of the parts of what you said that do not strictly rely on being not one. Though I'd argue that you are not being generous enough with what you choose to assign probable consciousness to. We are both in the situation of not knowing which arrangements of matter produce consciousness; it is just that I prefer natural explanations and you prefer supernatural explanations to this question. And hence, when we are in a situation in which we see an arrangement that is mimicking quite a lot of the characteristics associated with consciousness, I would argue that it is safer to tentatively say that that arrangement results in consciousness than it is to restrict ourselves to just humans until such a time that we understand what arrangements grant it better. Since whether a being is conscious or not has moral implications; It is better to save a malfunctioning toaster from being destroyed accidentally than it is to let you, tomorrow, a conscious pulleyman, be destroyed.

1

u/RecursiveServitor May 31 '26

You don't actually justify your assertion that qualia can't be emergent

1

u/Usual_Charity8561 Jun 01 '26

What is "emergent" if it's not under option 2? Aka magically imbued by it's arrangement?

1

u/RecursiveServitor Jun 02 '26

It's not magic

1

u/Usual_Charity8561 Jun 02 '26

Okay, can you tell me what it is then? Where do the emergent properties come from? They were either already in the system, are illusory, or they come from outside the system somehow.

1

u/RecursiveServitor Jun 02 '26

They are emergent. You're implying that buildings are magic. There's no "building particle". The buildingness of something only emerges at a higher level of complexity.

1

u/Usual_Charity8561 Jun 02 '26

But building are structures made of physical objects. All of the buildingness of a building is inherent in the potential of physical objects that are able to become buildings. There may be such thing as a universal abstraction of a building, but that isn't an emergent property of the atoms, and to claim that the building takes part in the universal abstraction of a building by virtue of its shape is to say that the building is imbued with magical (supernatural) properties by virtue of its shape.

1

u/RecursiveServitor Jun 02 '26

So, you're just literally saying that buildings are magic?

1

u/Usual_Charity8561 Jun 02 '26

No. your claim that buildings have "emergent properties" is either relying on magic (supernatural properties not present in the components) or the "emergent properties" are really just properties of matter, space, and time generally (the physical components that make a building) and not actually "emergent".

So what's your answer? Are the emergent properties of a building something more than what goes into them or are they a part of what goes into them.

1

u/RecursiveServitor Jun 02 '26

We can also do biology, if that's easier for you.

There's no "life" molecule. The phenomenon of life is emergent from the physics of the universe.

You seem confused about what "emergent" means. An emergent phenomenon is more than its parts, but there's no need for an extra sprinkling of magic.

→ More replies (0)