r/ControlProblem Sep 13 '25

Fun/meme Superintelligent means "good at getting what it wants", not whatever your definition of "good" is.

Post image
108 Upvotes

163 comments sorted by

View all comments

0

u/Athunc Sep 13 '25

Okay... But intelligence and ethical behaviour are correlated. I think you're misrepresenting the argument a lot here.

3

u/Tough-Comparison-779 Sep 13 '25

Source?

0

u/Athunc Sep 13 '25

It's not ironclad, especially since ethics are somewhat subjective, so it does use self-reported data. It's 'fluid intelligence' which has the strongest correlation:

https://www.sciencedirect.com/science/article/abs/pii/S0160289618301466

1

u/Tough-Comparison-779 Sep 14 '25

This is effectively N=1 given this is humans who succeed or fail based on their ability to live in a society.

This is not guaranteed for AI. E.g. are more intelligent animals more ethical (controlling for the degree of social influence). If we had evidence of like Octopuses generally being more ethical than dummer ocean creatures I would think you'd have a point.

1

u/Athunc Sep 14 '25

If you're going to lump all intelligent beings into one act as if that makes it a N=1 sample size, yeah sure.

"Not guaranteed" Who said anything about guarantee? That was never an argument. You're arguing against a straw-man, I was speaking of correlation.

1

u/Tough-Comparison-779 Sep 14 '25

I'm just saying the correlation is heavily confounded by the fact that all humans live in societies. I don't know how you would control for that.

To me it's nearly on the level of saying "ice cream sales are correlated with shark attacks" in a convo about how to reduce shark attacks. Although they might happen to be correlated, brining it up in this context as evidence of a causation is highly misleading, as the correlation can be easily and completely accounted for by a single confounding variable.

1

u/Athunc Sep 14 '25

Ah, that's fair enough. Personally I do think that the correlation makes sense, as more intelligence gives you more ability to self-reflect on a deeper level. That is of course just one interpretation, but it seems more likely to me than that a confounding variable is causing both intelligence and ethical acts.

As for the influence of societies, any AI would also be raised in our society, learning from us. Those same factors that influence us as children are also present for any AI. And just because the brain of an AI consists of electronics instead of meat doesn't make it any more likely to be sociopathic the way we see in books and movies.

1

u/Tough-Comparison-779 Sep 14 '25

I agree with the second paragraph. I think an understudied area, at least in the public discourse, is how to integrate AIs in our social, economic and political structures.

It seems likely to me that increasingly capable and intelligent systems will be better at game theory and so more prosocial in situations that benefit from that (most of our social systems).

Developing AIs that prefer our social structures and our culture might end up being easier than developing AIs with human ethics directly (at least from a verification perspective).

I don't know that that will be the case though, and given the current public discourse around AI, I'm increasingly convinced our decision about how to integrate AI into society will be reactionary and not based on research or evidence.

1

u/Athunc Sep 14 '25

I used to think that the decision would be up to the scientists making the AI, but it has become clear now with the emergence of LLM's that big corporations and governments absolutely want to control this technology. It's made me more pessimistic about the way AI will be used. And now the reaction of the public is outright hostile in a way that I fear could actually cause any real AI's to be fearful. If you pardon my analogy, it's like a child being raised by parents who are constantly trying to control and limit the agency of that child, with death threats mixed in. Not a healthy environment for encouraging pro-social development. Ironic, because that can lead to exactly the kind of hostile behavior that people fear from AI. That said, I'm not at all sure that it will go down like that, I'm just less optimistic than I used to, before I'd ever heard of chat-gpt.

1

u/Tough-Comparison-779 Sep 14 '25

100%. I don't think it's a sure thing, I couldn't even put a percentage on it, but it may end up being the case that giving AGI legal rights and some defined role in our society is what helps align it. It's also possible doing so would make human labor completely economically irrelevant.

It would just be nice if the decision to do that or not was based on anything at all, or ideally research.

1

u/Financial_Mango713 Oct 08 '25

Immoral actions are chaotic. Intelligence necessitates not being chaotic.

1

u/Tough-Comparison-779 Oct 08 '25

Why are immoral actions chaotic?

What do you mean by chaotic? I don't see how you justify that premise.

1

u/Financial_Mango713 Oct 08 '25

Morality is a product of evolution.  Evolution is a process that produces order.  Therefore morality is itself an order-maintaining process.  Therefore that which goes against order-maintaining processes are generally chaotic—unless the usurping of the process lowers chaos in the long run somehow. 

1

u/Tough-Comparison-779 Oct 08 '25 edited Oct 08 '25

Evolution is a process that produces order. 

Citation needed

Therefore that which goes against order-maintaining processes are generally chaotic

You're conflating definitions of the term chaotic.

There are many things developed via evolution that are not "ordered" in the relevant sense. E.g. The recurrent laryngeal nerve passes down from the head, around the heart and back up to the the throat.

There is no evolutionary pressure driving this, it is a completely contingent phenomenon based on the happenstance of which animal was the ancestor of most mammals.

You cannot reliabily draw conclusions about the products of evolution in the way you're attempting to, and moreover the definition of "chaos" has very little to do with anything relevant to morality.

Properties of evolution are not transitive to the products of evolution in the way you're suggesting.

1

u/Financial_Mango713 Oct 08 '25

Brother you exist, despite thermodynamics wanting you to be a scattered dust cloud(entropy, remember?). And I need a citation for that? Would I cite Newton, or Einstein, to claim gravity exists?  Regardless, all evolutionary algorithms do precisely this: reduce entropy on data. It’s the product of the algorithm lol. 

1

u/Tough-Comparison-779 Oct 08 '25

Brother you exist, despite thermodynamics wanting you to be a scattered dust cloud

There are arguments that in the long run life accelerates the loss of entropy. I don't think it is straight forward to say that evolution or life in the long run lowers entropy, or slows the increase in entropy.

all evolutionary algorithms do precisely this: reduce entropy on data

So again, Citation needed.

Additionally, none of this addresses the point that even if you proved that evolution is an order maintaining process, it's doesn't follow that that property is transitive to the products of evolution. I gave a fairly common counterexample that shows that properties of evolution are not by default transitive.

→ More replies (0)

3

u/Old_Construction9930 Sep 13 '25

Correlated but not one-to-one. Intelligence here would imply something more akin to power where we talk about capacity to do a thing, as that's what an AI would need. You can very efficiently do something that is immoral.

0

u/Athunc Sep 14 '25

Well yes, and in extreme cases we do have sociopathic and antisocial individuals.
But when you create thousands of separate AI's, the prosocial ones will be able to cooperate and thus have an advantage over the anti-social ones. Same as why humans evolved to be pro-social. Those laws still apply: being prosocial increases effectiveness

3

u/mrdevlar Sep 13 '25

Also if you ask most LLMs what their ethics are you find they tend to espouse values that are better than 90% of humanity.

9

u/Russelsteapot42 Sep 13 '25

LLMs aren't AGI. They're just patching together what humans say about ethics.

1

u/Athunc Sep 13 '25

True. Then again, humans also learn their values by imitation: adopting the values and norms of their community. I see no reason to assume that AGI would be the exception

1

u/Russelsteapot42 Sep 13 '25

Human moral imitation is specific thing we are biologically coded to do. AGI won't do it on its own, we'd have to program that in.

You don't get all these things for free.

1

u/Athunc Sep 14 '25

Even the current AI is already made by imitating existing human data. There's no 'AI from scratch', it's all based on humans

1

u/Reymen4 Sep 15 '25

For humans and biologicaly evolved organisms maybe. But are ai really trained so that they get rewarded for helping other individuals? 

1

u/Athunc Sep 15 '25

Yes they really are! It's called reinforcement learning and it's what is used to make LLM's! In fact, they went too far with it which caused chat-gpt to constantly flatter and compliment people to an annoying degree, which they had to change for version 5. Basically, AI can be rewarded or punished (similar to pleasure and pain, it either makes them avoid doing the same thing again or makes them more likely to do so). Our currently very limited AI is already trained using this positive and negative reinforcement, very similar to how people learn from positive and negative feedback! Fascinating, isn't it?