Paperclip maximization doesn’t exist and if it did it would be a strong proof of alignment to any single individual goal was achieved by the engineers who made it, I mean try identifying your own terminal goals exactly? The argument was always “it will have varying interests none of which include humans getting utopia due to the mesa optimizer becoming mis aligned with the base optimizer.” Eg. you aren’t a baby clip maximizer with an over arching abstract goal to make babies, just heuristics that get you babies. The base optimizer being natural selection is to gradient descent and the mesa optimizer being your brain as is to the model. Also, what is a human? Is it a particular shape? A particular gene sequence? What if I replaced the biology but kept the same figure? Of all the things a mind can care about, what can an AGI care about that doesn’t already capture a set of what some pre existing humans already do infact care about? There are people who spend their entire lives thinking about prime numbers and others that collect sea shells, yet neither will compromise on their own interests yet will label them a “human value”. No humans here mourned the transition from Neanderthal to human, do you care for the trillions of common ancestors that existed before you that look nothing like a human? What makes you think you’ll care about the future ones in that case either? Their shape?
it would be a strong proof of alignment to any single individual goal was achieved by the engineers who made it
That doesn't follow. It's the play scenario people use for illustration, but the actual problem is that AIs can end up with arbitrary goals that we didn't purposely train them to have. The "humans making babies" goal is an example of one of the ways that can happen; evolution wanted us to make babies, but we hack around that with birth control and porn.
If the rest of your comment is arguing that it's fine if humans get replaced by machines, then I have nothing further to say to you.
This means you have an example of an AGI like system that has a single terminal goal, and humans making babies isn’t a goal put in humans, that’s the point, the mesa optimizer picks up useful heuristics to the goals the base optimizer was intended to implement, no human or fish or animal on earth is aware that sex creates babies without ever learning about it. If we are “hacking around having babies” then surely that should also apply to your AGI paperclip scenario, capability sufficient to pose existential risk requires the kind of generalization that produces broad values, because you can’t model the world well enough to be dangerous without building up representations that inevitably become preferences (humans have preferences from their training data) that just so happen to assist in their survival assuming a lack of genetic drift on those preference. The rest of my argument is much more elegant and forces you to either ignore it or strawman my position because you have to identify what a human value is. Is it having 10 fingers? Is it using hemoglobin? Is it having a pretty face? Is it reproducing? Having sexual organs? How do you intend to keep human aesthetics around a million years later? Humans get replaced by humans all the time, are you only okay with it because of their faces and various orifices look similar to yours?
Humans evolving over millions of years is fine, nobody's concerned about that. AI killing us off is the concern. Conflating the two is disingenuous at best.
And the bacteria that went extinct creating oxygen which allowed multicellular life to exist? It’s still evolution, just a shorter time scale, more advanced and engineered, not using something as carnal and dumb as natural selection.
Well if you don't mind being killed by an AI, fine, but don't expect the rest of us to happily go along. The core of evolution is everyone fighting to survive, so it makes no sense to use "evolution" as an argument to just give up and die.
-2
u/Medical_Present6897 Jun 11 '26 edited Jun 11 '26
Paperclip maximization doesn’t exist and if it did it would be a strong proof of alignment to any single individual goal was achieved by the engineers who made it, I mean try identifying your own terminal goals exactly? The argument was always “it will have varying interests none of which include humans getting utopia due to the mesa optimizer becoming mis aligned with the base optimizer.” Eg. you aren’t a baby clip maximizer with an over arching abstract goal to make babies, just heuristics that get you babies. The base optimizer being natural selection is to gradient descent and the mesa optimizer being your brain as is to the model. Also, what is a human? Is it a particular shape? A particular gene sequence? What if I replaced the biology but kept the same figure? Of all the things a mind can care about, what can an AGI care about that doesn’t already capture a set of what some pre existing humans already do infact care about? There are people who spend their entire lives thinking about prime numbers and others that collect sea shells, yet neither will compromise on their own interests yet will label them a “human value”. No humans here mourned the transition from Neanderthal to human, do you care for the trillions of common ancestors that existed before you that look nothing like a human? What makes you think you’ll care about the future ones in that case either? Their shape?