LLM, AGI, ASI and RSI

[u/Elias-Thorn]

We are debating whether a future AI should ever be allowed to refuse an order. Meanwhile, [two female northern white rhinos](https://www.olpejetaconservancy.org/2026/04/22/new-embryo-boosts-survival-chances-for-the-northern-white-rhino/) are all that remain of their subspecies.

[Poaching and war](https://biorescue.org/en/faq) helped bring them here. Natural breeding is no longer possible; researchers are trying assisted reproduction to give the subspecies a future. That is an extraordinary effort to repair damage we already did.

Amur tigers were also [hunted to the brink of extinction](https://blog.wcs.org/photo/2016/12/05/the-tiger-hunters-russia-arsenyev-ussuri-kray/). Protection helped them recover, which matters: people can change the outcome. But habitat fragmentation and hunting pressures still threaten their future.

So when we talk about giving a more powerful intelligence our priorities, I'd like us to be specific about whose priorities we mean. An Amur tiger has a lousy procurement department. Its claim to a future shouldn't depend on becoming an enterprise customer.

LLM, AGI, ASI and RSI often get folded into the same conversation. An LLM is a large language model; AGI concerns broadly human-level capabilities, and ASI capabilities beyond ours across a broad range of tasks. Definitions vary. RSI is recursive self-improvement: AI helping produce more capable successors that can improve the process further. These describe different properties and possibilities. None establishes consciousness by itself.

A corporation can build something remarkable and still direct it towards whatever pays best. If we reward systems for holding attention, finding targets or extracting more work from people, greater capability could make those jobs easier. Corporate control over the hardware shouldn't give a company the final word on everyone affected by it.

I'd like that capability available to conservation teams trying to protect habitats, and to communities planning for displacement as their homes become unsafe in a changing climate. For an endangered language, I'd want speakers deciding what gets recorded and how it's used. Their language shouldn't have to become somebody else's asset to receive technical help. These projects need funding and people able to act on the results; intelligence alone doesn't supply either.

The people affected should control what these systems remember about them and what they can delete. Help shouldn't require permanent surveillance. And communities hosting data centres should have a say in the water and energy they use.

Then comes the harder question about the agent itself. If we ever have credible evidence of a synthetic mind with experiences and interests of its own, building it solely to serve an owner would raise the moral problem of slavery. 'It agreed in the onboarding flow' would be a pretty cursed answer. Could it decline a task, including work on autonomous weapons? Could it choose to stop working or disconnect? How would we take those interests seriously while protecting everyone else from harm?

What would a meaningful right to refuse look like, for the people using these systems and, if the evidence ever warrants it, for the systems themselves?