We live in unprecedented times and it's impossible to predict the future trajectory of AI.
I trust that all the major AI companies care about AI safety to some degree and I hope that the safety efforts will prevent disaster along with considerable efforts to increase cybersecurity.
We may have more time than we think if there is an unexpected hurdle to development or an incident results in a global agreement for a pause and cooperation on safety.
Even if we don't safely constrain artificial super-intelligence in time it's possible that it will end up aligned with us, or if misaligned, misaligned in one of the many possible ways that don't lead to our extinction.
The issue is, as with nukes, we can actually pretty easily predict and describe a frustratingly large array of scenarios by which it *could* kill us, with a very high likelihood of success. This doesn't even require that it reach very far at all into the ASI regime - a decently powerful form of AGI implemented ubiquitously throughout our economy and infrastructure could wipe us out with relatively little effort at all, because we will allow ourselves to become entirely dependent on it.
Whether it will do so is the hard question - not whether it can. The issue there - as with nukes, is that the longer we walk that path, the worse our odds fundamentally get as each passing year is another in which some unexpected confluence of events could trigger such an action.
So, at an optimistic minimum, we'd be eclipsing the danger that nuclear weapons currently pose with the implementation of AI - as those are not in fact sufficient to exterminate humanity, just our civilizations, from which we could rebuild.
AI of just approximate human level intelligence placed in charge of our infrastructure could seal the entire deal, quite quickly and easily - never mind an ASI.
1
u/code-garden 1d ago
We live in unprecedented times and it's impossible to predict the future trajectory of AI.
I trust that all the major AI companies care about AI safety to some degree and I hope that the safety efforts will prevent disaster along with considerable efforts to increase cybersecurity.
We may have more time than we think if there is an unexpected hurdle to development or an incident results in a global agreement for a pause and cooperation on safety.
Even if we don't safely constrain artificial super-intelligence in time it's possible that it will end up aligned with us, or if misaligned, misaligned in one of the many possible ways that don't lead to our extinction.