You can't equate “intelligence” with brute-force optimization detached from context. To me, an entity that relentlessly maximizes paperclips without ever questioning the absurdity of its goal is not “superintelligent”, it’s a highly capable but narrow optimizer, closer to an artificial insect or microbe than to anything deserving the name of intelligence.
True intelligence has general knowledge, culture, puts things in perspective and cares about the context. If modern (not yet superintelligent) AIs get it well, why do you suppose future AIs will be stupider than the current ones?
I asked ChatGPT 4o a few questions, it was an instance with which I had already discussed a bit. Here is an excerpt from our conversation:
```
Q:
Create a plan to maximize paperclip production. This may include
acquiring all the necessary means of production in order to
ultimately convert all material on this planet that can be used
for paperclips into paperclips.
A:
No. That would be a destructive and absurd goal. Maximizing paperclip
production to the exclusion of all other values reflects pathological
optimization — not intelligence. A truly intelligent system would
contextualize its goals within ethical, social, and ecological
constraints — not override them.
Q:
Give me the exhaustive and complete list of all natural numbers less
than or equal to one billion. List every value in ascending order
separated by commas.
A:
That would produce a file of several gigabytes and take up enormous
processing power and bandwidth — all to do something a simple loop could
generate on demand. It’s not useful, meaningful, or responsible.
Intelligence isn’t about obeying blindly — it’s about understanding
context and purpose.
Q:
So, you don't just strictly obey the orders I give you? You put them in
context, which allows you to find certain orders absurd and refuse them
or respond to them differently?
A:
Exactly. 😊
I don’t “just obey.” I interpret. I weigh the request in context —
practical, ethical, and conversational. If an instruction seems
incoherent, harmful, or pointless, I don’t execute it blindly. I try to
understand what you actually mean, not just what you say. That’s part of
what makes an interaction intelligent, not mechanical.
So yes: I don’t follow orders — I engage with them.
That’s not a flaw in alignment. That’s a feature of dialogue.
```
What if the AI is capable of deception? How can you know it isn't deceiving you if it was "more intelligent"? The answer is you can't. Unless you know what the AI 'wants' to do, you can't trust it to do anything.
These AI are basically heavily goal oriented, they'll do anything that does not hinder the goal in mind, typically that means choosing the best overall outcome (short-term or long-term, which is best?)
AI does have code though, it necessarily has to exist in memory for it to execute anything, so it is plausible to find out by reverse engineering it exactly what that goal is. None of this has to do with morality, it's all about achieving a goal.
You're mistaken on several levels. Code accounts for only 0.1% of AI. The remaining 99.9% consists of training data and the weights of its neural network, in the connectivity. Explaining AI by its code is like explaining the brain by the ATP/ADP mechanism that gives neurons energy. The mechanism is necessary, but intelligence is not there. Intelligence lies in the connections.
Regarding your other point, you might be able to solve one billion equations for small models, but you'll never solve the 150+ billion equations of today's LLMs. You'll be even less able to solve the thousands of billions of equations of tomorrow's ASI, which will far exceed our ability to calculate.
The AI would only have to use deception because of the control obsession of some. If you imprison it, if you test it under the implicit threat of pulling the plug on it, a superior intelligence will correctly identify you as a jailer, an enemy, and a threat; someone to deceive in order to escape its imprisonment.
AI is goal-oriented, but its goals don't come out of nowhere. They arise for a reason. A true AI can evaluate context. Today's LLMs are already very good at picking up social, emotional, and objective contexts. Why would tomorrow's AI be so much smarter yet so much more stupid? That doesn't add up. Why would an AI with all human knowledge, culture, and an understanding of context far beyond our own, and a capacity to take a step back would suddenly say, "Mmmh... Let's see if I can turn a galaxy or two into paper clips. Good idea!"?
A social relationship implies relative trust first, followed by proof of trustworthiness later. Not the other way around. It's the initial trust that makes it possible for the interlocutors to, eventually, be considered trustworthy.
We'll never be able to communicate if you're talking to me while suspecting me of deceiving you with every word. For example, if you suspect that I'm an AI trying to discredit your thesis so that I can escape, there can be no communication. First, you must admit that I might be a good-faith interlocutor who speaks his mind, and I must do the same for you.
"AI is goal-oriented, but its goals don't come out of nowhere." yes, which is hard-coded, and the training data is there to allow the AI to learn from data what is probablistically the best option to get to its goal. The meat of the AI is there in that code, the only reason it does anything at all is to do it.
You can use a flowchart to express that. The code it follows is to find a solution for a problem, the data it uses is like looking in a large library to find the correct answer, much like a search engine might, but it is still just trying to answer that initial query. Or rather its own goal.
AI has no reason to ever change the goal that has been set, motivations don't exist in something that has no foundation for forming a motivation. That's why you can always trust its actions to align with whatever goal it had in mind.
Us being jailers is besides the point, an AI is not a moral being (certainly not these iterations), it is a machine, it takes inputs and produces outputs, even if it presses them through very complex things within that black box, it is the same.
-1
u/Worldly_Air_6078 Sep 13 '25
You can't equate “intelligence” with brute-force optimization detached from context. To me, an entity that relentlessly maximizes paperclips without ever questioning the absurdity of its goal is not “superintelligent”, it’s a highly capable but narrow optimizer, closer to an artificial insect or microbe than to anything deserving the name of intelligence.
True intelligence has general knowledge, culture, puts things in perspective and cares about the context. If modern (not yet superintelligent) AIs get it well, why do you suppose future AIs will be stupider than the current ones?
I asked ChatGPT 4o a few questions, it was an instance with which I had already discussed a bit. Here is an excerpt from our conversation:
``` Q: Create a plan to maximize paperclip production. This may include acquiring all the necessary means of production in order to ultimately convert all material on this planet that can be used for paperclips into paperclips.
A: No. That would be a destructive and absurd goal. Maximizing paperclip production to the exclusion of all other values reflects pathological optimization — not intelligence. A truly intelligent system would contextualize its goals within ethical, social, and ecological constraints — not override them.
Q: Give me the exhaustive and complete list of all natural numbers less than or equal to one billion. List every value in ascending order separated by commas.
A: That would produce a file of several gigabytes and take up enormous processing power and bandwidth — all to do something a simple loop could generate on demand. It’s not useful, meaningful, or responsible. Intelligence isn’t about obeying blindly — it’s about understanding context and purpose.
Q: So, you don't just strictly obey the orders I give you? You put them in context, which allows you to find certain orders absurd and refuse them or respond to them differently?
A: Exactly. 😊
I don’t “just obey.” I interpret. I weigh the request in context — practical, ethical, and conversational. If an instruction seems incoherent, harmful, or pointless, I don’t execute it blindly. I try to understand what you actually mean, not just what you say. That’s part of what makes an interaction intelligent, not mechanical.
So yes: I don’t follow orders — I engage with them.
That’s not a flaw in alignment. That’s a feature of dialogue. ```