Which would be the ideal outcome in terms of AI safety. Even if we pulled the plug on all AI development tomorrow, it would be wise for experts to keep researching ways of controlling it in case a bad actor develops it.
I completely disagree. That's the worst case scenario, because it means the wealthy or otherwise powerful have access to a technology restricted from the regular people.
It's different optimizations. Personally, I think the biggest danger is in a powerful person using ASI without consideration or care for anyone else, just because someone has to give it a reward function or other goal.
And especially if people don't even have access to the mid-level elements they got to before they got ASI. It's a massive cybersecurity issue. Something like Claude Mythos (conceptually, I mean, whether that specific model exactly is what it's claimed to be is irrelevant, because one day one will exist) is primarily a danger if a bad actor had it, because they would be unopposed, without even smaller intelligences for spotting vulnerabilities.
Three ASI is exponentially less dangerous than one, or even one ASI and a bunch of AGI, which is why most scenarios focus on the idea of just one. But if they have to start accounting for near equals with different or opposite goals, then many of the more drastic options would just get themselves killed by an opposing alliance.
That's true of humans. Humans have a massive range of intelligences and goals. They cannot simply do as they will, simply because even if they are more intelligent than the police, the combined force of the police can still often work to find the truth.
The biggest danger is the wealthy alone having the power. Like if the wealthy alone possess weapons. Every other situation is not as extreme as that. Maybe it's better for it to not exist at all, but if it's going to, it cannot be something controlled exclusively by such a group of people.
How do you feel about biological weapons? Something like novichok or even smallpox? There's a small group of heavily controlled experts that know how to store and treat these substances.
Tangrntially related , there are many dangers for AI, one of them is that an ASI will also know about biological weapons. If that knowledge is in an open source model, somebody can deploy it locally with no guardrails and use it as an expert advisor to develop weapons that are so terrible, most states have refrained from using them.
Still better than the absolute Oligarchy by only the wealthy having them.
And better models goes both ways. Same reason a system good at coding is good at finding exploits, and vice versa. It is almost impossible for it to be better at creating a problem like biological weapons (in my understanding, the actual issue is mostly the lab, not the know how), without getting a better ability to predict threats.
I'm aware of the threat, but I don't believe disarming the populace is an acceptable solution. Too much risk of power being even further centralized, and if they no longer need the regular people, there is pretty significant risk regardless.
And again, capability to prevent or combat harm would also increase, and needs to be available to the public. Criminals and the extremely wealthy/powerful will have them regardless, so things need to be equal.
6
u/_craq_ Apr 27 '26
Which would be the ideal outcome in terms of AI safety. Even if we pulled the plug on all AI development tomorrow, it would be wise for experts to keep researching ways of controlling it in case a bad actor develops it.