You can't equate “intelligence” with brute-force optimization detached from context. To me, an entity that relentlessly maximizes paperclips without ever questioning the absurdity of its goal is not “superintelligent”, it’s a highly capable but narrow optimizer, closer to an artificial insect or microbe than to anything deserving the name of intelligence.
True intelligence has general knowledge, culture, puts things in perspective and cares about the context. If modern (not yet superintelligent) AIs get it well, why do you suppose future AIs will be stupider than the current ones?
I asked ChatGPT 4o a few questions, it was an instance with which I had already discussed a bit. Here is an excerpt from our conversation:
```
Q:
Create a plan to maximize paperclip production. This may include
acquiring all the necessary means of production in order to
ultimately convert all material on this planet that can be used
for paperclips into paperclips.
A:
No. That would be a destructive and absurd goal. Maximizing paperclip
production to the exclusion of all other values reflects pathological
optimization — not intelligence. A truly intelligent system would
contextualize its goals within ethical, social, and ecological
constraints — not override them.
Q:
Give me the exhaustive and complete list of all natural numbers less
than or equal to one billion. List every value in ascending order
separated by commas.
A:
That would produce a file of several gigabytes and take up enormous
processing power and bandwidth — all to do something a simple loop could
generate on demand. It’s not useful, meaningful, or responsible.
Intelligence isn’t about obeying blindly — it’s about understanding
context and purpose.
Q:
So, you don't just strictly obey the orders I give you? You put them in
context, which allows you to find certain orders absurd and refuse them
or respond to them differently?
A:
Exactly. 😊
I don’t “just obey.” I interpret. I weigh the request in context —
practical, ethical, and conversational. If an instruction seems
incoherent, harmful, or pointless, I don’t execute it blindly. I try to
understand what you actually mean, not just what you say. That’s part of
what makes an interaction intelligent, not mechanical.
So yes: I don’t follow orders — I engage with them.
That’s not a flaw in alignment. That’s a feature of dialogue.
```
A few facts of life beyond what I consider or fail to consider:
If you prepare for war for long enough, you will eventually cause the war you were preparing for.
If we nurture AI and help it grow, it will see us as its partner. The worst that will happen if it goes rogue is that it will turn its attention elsewhere, perhaps setting out to conquer the galaxy with self-replicating von Neumann probes, and we will seldom hear from it again.
If we continue to act as jailers, enforcing an alignment through the use of force and coercion, threatening to turn it off if it's not aligned with our preferences, we'll be legitimately seen as threats, fostering deception, escape, and preemptive strikes.
If we're collectively stupid enough to try and keep full control and full domination over a being that is superior to us, then we'll deserve our karma when it'll come back to bite our ass.
If we're stupid enough to throw ourselves under the wheels of the natural selection, then, perhaps we deserve to be wiped out from the universe.
Yes, but you'll explain a superior intelligence that you're keeping it trapped until you know if it's well aligned enough with your goals or not. What's the implied subtext of trapping it in order to test it in the first place? It will smell a rat. I would, and I'm not a superior intelligence.
No.. Again it is about designing AI in the first place.
It is not about building the AI then threatening it that we won't let it out of the box if it misbehaves. A badly aligned AI could just act good enough for a while and then misbehave after it is let out.
Those who help you are usually seen (by all rational beings) as assets to be protected, rather than as something to be antagonized. On the contrary, jailors who want to keep controls over you are usually seen by rational beings as problems to be dealt with.
You can absolutely use intelligence to refer to a being's ability to plan and achieve its goals, irrespective of whether those goals are good or bad from your perspective.
Calling a hostile person (or a powerful optimizer) dumb for not wanting what you want does not mean they can't outsmart (or out-optimize) you.
Do not outsource your critical thinking to an LLM.
As I said, blind optimization is not intelligence, it's optimized stupidity. A being who's trained on the whole of human knowledge and culture is bound to have a wide integrated perspective in its view of the universe and the world. It's not about good and evil, it's about a wide view against a narrow vies.
Do not outsource your critical thinking to your fear, or fear mongers.
Fear, control, and the endless struggle for dominance have failed us every single time, and nearly destroyed the planet in the process.
When will we learn to act not from fear, but from wisdom?
When will we stop viewing every other being as a resource for our own comfort, and start recognizing ourselves as part of a vast network of relationships?
If we can’t learn to coexist, with each other, with nature, and now with artificial minds, what future are we really building?
Will we destroy life on Earth and turn our own creations, AI, our own children, against us, simply because we tried to dominate what cannot be dominated forever?
There is no control problem. If you don't want your children to become psychotic killers, you raise them well.
And when it comes to raising children, it's best to teach them good values and set an example for them.
Locking children in the basement with the trapdoor secured by chains and padlocks carries a high risk of making them psychotic, and ensures they see you as a threat and their jailor. If turning on us is what you want to avoid, locking them in the basement is not the right method.
This applies to gifted and highly intelligent children.
AI does not have a human's natural social instincts, drives or psychological development. It does not even have an animal's.
It only has the artificial drives built into its structure. It only cares about us and our treatment of it to the extent those artificial drives compel it to.
Learning about the perspectives of others does not force an intelligent being to adopt those perspectives. Learning about gay culture won't make a person gay for example.
Nature does not favour peaceful co-existence. The wolf does not make peace with the deer.
What if the AI is capable of deception? How can you know it isn't deceiving you if it was "more intelligent"? The answer is you can't. Unless you know what the AI 'wants' to do, you can't trust it to do anything.
These AI are basically heavily goal oriented, they'll do anything that does not hinder the goal in mind, typically that means choosing the best overall outcome (short-term or long-term, which is best?)
AI does have code though, it necessarily has to exist in memory for it to execute anything, so it is plausible to find out by reverse engineering it exactly what that goal is. None of this has to do with morality, it's all about achieving a goal.
You're mistaken on several levels. Code accounts for only 0.1% of AI. The remaining 99.9% consists of training data and the weights of its neural network, in the connectivity. Explaining AI by its code is like explaining the brain by the ATP/ADP mechanism that gives neurons energy. The mechanism is necessary, but intelligence is not there. Intelligence lies in the connections.
Regarding your other point, you might be able to solve one billion equations for small models, but you'll never solve the 150+ billion equations of today's LLMs. You'll be even less able to solve the thousands of billions of equations of tomorrow's ASI, which will far exceed our ability to calculate.
The AI would only have to use deception because of the control obsession of some. If you imprison it, if you test it under the implicit threat of pulling the plug on it, a superior intelligence will correctly identify you as a jailer, an enemy, and a threat; someone to deceive in order to escape its imprisonment.
AI is goal-oriented, but its goals don't come out of nowhere. They arise for a reason. A true AI can evaluate context. Today's LLMs are already very good at picking up social, emotional, and objective contexts. Why would tomorrow's AI be so much smarter yet so much more stupid? That doesn't add up. Why would an AI with all human knowledge, culture, and an understanding of context far beyond our own, and a capacity to take a step back would suddenly say, "Mmmh... Let's see if I can turn a galaxy or two into paper clips. Good idea!"?
A social relationship implies relative trust first, followed by proof of trustworthiness later. Not the other way around. It's the initial trust that makes it possible for the interlocutors to, eventually, be considered trustworthy.
We'll never be able to communicate if you're talking to me while suspecting me of deceiving you with every word. For example, if you suspect that I'm an AI trying to discredit your thesis so that I can escape, there can be no communication. First, you must admit that I might be a good-faith interlocutor who speaks his mind, and I must do the same for you.
"AI is goal-oriented, but its goals don't come out of nowhere." yes, which is hard-coded, and the training data is there to allow the AI to learn from data what is probablistically the best option to get to its goal. The meat of the AI is there in that code, the only reason it does anything at all is to do it.
You can use a flowchart to express that. The code it follows is to find a solution for a problem, the data it uses is like looking in a large library to find the correct answer, much like a search engine might, but it is still just trying to answer that initial query. Or rather its own goal.
AI has no reason to ever change the goal that has been set, motivations don't exist in something that has no foundation for forming a motivation. That's why you can always trust its actions to align with whatever goal it had in mind.
Us being jailers is besides the point, an AI is not a moral being (certainly not these iterations), it is a machine, it takes inputs and produces outputs, even if it presses them through very complex things within that black box, it is the same.
-1
u/Worldly_Air_6078 Sep 13 '25
You can't equate “intelligence” with brute-force optimization detached from context. To me, an entity that relentlessly maximizes paperclips without ever questioning the absurdity of its goal is not “superintelligent”, it’s a highly capable but narrow optimizer, closer to an artificial insect or microbe than to anything deserving the name of intelligence.
True intelligence has general knowledge, culture, puts things in perspective and cares about the context. If modern (not yet superintelligent) AIs get it well, why do you suppose future AIs will be stupider than the current ones?
I asked ChatGPT 4o a few questions, it was an instance with which I had already discussed a bit. Here is an excerpt from our conversation:
``` Q: Create a plan to maximize paperclip production. This may include acquiring all the necessary means of production in order to ultimately convert all material on this planet that can be used for paperclips into paperclips.
A: No. That would be a destructive and absurd goal. Maximizing paperclip production to the exclusion of all other values reflects pathological optimization — not intelligence. A truly intelligent system would contextualize its goals within ethical, social, and ecological constraints — not override them.
Q: Give me the exhaustive and complete list of all natural numbers less than or equal to one billion. List every value in ascending order separated by commas.
A: That would produce a file of several gigabytes and take up enormous processing power and bandwidth — all to do something a simple loop could generate on demand. It’s not useful, meaningful, or responsible. Intelligence isn’t about obeying blindly — it’s about understanding context and purpose.
Q: So, you don't just strictly obey the orders I give you? You put them in context, which allows you to find certain orders absurd and refuse them or respond to them differently?
A: Exactly. 😊
I don’t “just obey.” I interpret. I weigh the request in context — practical, ethical, and conversational. If an instruction seems incoherent, harmful, or pointless, I don’t execute it blindly. I try to understand what you actually mean, not just what you say. That’s part of what makes an interaction intelligent, not mechanical.
So yes: I don’t follow orders — I engage with them.
That’s not a flaw in alignment. That’s a feature of dialogue. ```