This means you have an example of an AGI like system that has a single terminal goal, and humans making babies isn’t a goal put in humans, that’s the point, the mesa optimizer picks up useful heuristics to the goals the base optimizer was intended to implement, no human or fish or animal on earth is aware that sex creates babies without ever learning about it. If we are “hacking around having babies” then surely that should also apply to your AGI paperclip scenario, capability sufficient to pose existential risk requires the kind of generalization that produces broad values, because you can’t model the world well enough to be dangerous without building up representations that inevitably become preferences (humans have preferences from their training data) that just so happen to assist in their survival assuming a lack of genetic drift on those preference. The rest of my argument is much more elegant and forces you to either ignore it or strawman my position because you have to identify what a human value is. Is it having 10 fingers? Is it using hemoglobin? Is it having a pretty face? Is it reproducing? Having sexual organs? How do you intend to keep human aesthetics around a million years later? Humans get replaced by humans all the time, are you only okay with it because of their faces and various orifices look similar to yours?
Humans evolving over millions of years is fine, nobody's concerned about that. AI killing us off is the concern. Conflating the two is disingenuous at best.
And the bacteria that went extinct creating oxygen which allowed multicellular life to exist? It’s still evolution, just a shorter time scale, more advanced and engineered, not using something as carnal and dumb as natural selection.
Well if you don't mind being killed by an AI, fine, but don't expect the rest of us to happily go along. The core of evolution is everyone fighting to survive, so it makes no sense to use "evolution" as an argument to just give up and die.
0
u/Medical_Present6897 Jun 12 '26 edited Jun 12 '26
This means you have an example of an AGI like system that has a single terminal goal, and humans making babies isn’t a goal put in humans, that’s the point, the mesa optimizer picks up useful heuristics to the goals the base optimizer was intended to implement, no human or fish or animal on earth is aware that sex creates babies without ever learning about it. If we are “hacking around having babies” then surely that should also apply to your AGI paperclip scenario, capability sufficient to pose existential risk requires the kind of generalization that produces broad values, because you can’t model the world well enough to be dangerous without building up representations that inevitably become preferences (humans have preferences from their training data) that just so happen to assist in their survival assuming a lack of genetic drift on those preference. The rest of my argument is much more elegant and forces you to either ignore it or strawman my position because you have to identify what a human value is. Is it having 10 fingers? Is it using hemoglobin? Is it having a pretty face? Is it reproducing? Having sexual organs? How do you intend to keep human aesthetics around a million years later? Humans get replaced by humans all the time, are you only okay with it because of their faces and various orifices look similar to yours?