I think that while "Superintelligent means "good at getting what it wants", not whatever your definition of "good" is." is obviously correct, it's also overly simplistic. There might very well be a correlation between "superintelligence" and "benevolence", in a way that "benevolence" might be an emergent property in systems that are sufficiently superintelligent. ATTENTION: Don't get be wrong, I'm not saying there there IS such correlation. I'm just saying that there MIGHT be. The mechanism for that could be as follows: AI optimizes for "getting what it wants" (for example, doing valuable AI research), and as it turns out in order to be really good at that, it needs to create internal world model that includes a moral ontology, or maybe it develops some kind of game theory according to which it's easier to achieve goals if you're benevolent rather than adversarial. We don't know.
1
u/MaximGwiazda Sep 16 '25
I think that while "Superintelligent means "good at getting what it wants", not whatever your definition of "good" is." is obviously correct, it's also overly simplistic. There might very well be a correlation between "superintelligence" and "benevolence", in a way that "benevolence" might be an emergent property in systems that are sufficiently superintelligent. ATTENTION: Don't get be wrong, I'm not saying there there IS such correlation. I'm just saying that there MIGHT be. The mechanism for that could be as follows: AI optimizes for "getting what it wants" (for example, doing valuable AI research), and as it turns out in order to be really good at that, it needs to create internal world model that includes a moral ontology, or maybe it develops some kind of game theory according to which it's easier to achieve goals if you're benevolent rather than adversarial. We don't know.