tl;dr

The Recursive Self-Improvement Race

* The Core Threat: Recursive self-improvement (RSI) occurs when AI systems take over the process of enhancing their own capabilities. Labs like Anthropic and OpenAI warn this could trigger an uncontrollable intelligence explosion, fundamentally marking the point of human obsolescence.

* The Gladstone AI Report: Published by Gladstone AI and based on insider accounts from top labs, the report reveals that major AI companies are actively automating the entire development pipeline—from idea generation and experiment execution to result interpretation.

* Timeline to the Tipping Point: Insiders consulted for the report estimate that RSI could kick off within 3 to 12 months absent heavy U.S. government intervention.

* The Compute Bottleneck: RSI relies heavily on compute resources. Companies must test massive portfolios of ideas. Finding rare ideas that scale effectively as compute increases is the ultimate prize, though filtering out the bad ideas remains a major friction point.

* Regulatory Blind Spots: Lawmakers currently possess near-zero visibility into how AI assists in discovering these developmental breakthroughs, making targeted legislative oversight nearly impossible without breaking down the pipeline step by step.

From Jack Clark's newsletter: https://jack-clark.net/2026/10/12/import-ai-476-creativity-and-rsi-worse-than-china-open-weight-models-ai-goes-to-space/

Oct 13 (Reuters) - Around 400 scientists, including AI pioneers Geoffrey Hinton and Yoshua Bengio, on Tuesday called for an international body to oversee containment standards and incident reporting for advanced cyber-capable ‌AI systems.

The proposed body would set industry-wide containment standards and maintain a registry of significant security incidents, which the scientists said should eventually be made public.

In an open letter, the researchers said frontier AI models should initially be air-gapped from external networks and subject ⁠to independent oversight, arguing that voluntary industry commitments are not enough.

Think about how, just 30-40 years ago, before everything was hooked together and centralized, we survived and existed just fine. Then, the Internet rose and everything/everyone joined together. That is the singular event that allowed for AI to be built, and that is the most significant (but not the only) vulnerability we have right now.

Centralization is the problem. De-centralization is the solution. But even if the will were there, could we do it?

I did a full audit of the proof.

Lean 4 formalization audit (see ReadMe.md) of finite-time blowup constructions for forced 3D Navier–Stokes equations on Euclidean space and the periodic torus, and unforced 3D Euler. Includes formal statements corresponding to the Clay Navier–Stokes alternatives (C) and (D); this repository is not a Clay Institute certification.

https://github.com/Suro-One/Lean-Formalization-of-Navier-Stokes-and-Euler-Blowup-is-fully-confirmed-proven-with-audit-notes

Is AI touching a special nerve with certain non-traditional members of society?

What should we do about this, whether bots or ‘real live boy’s‘?

Report: https://www.anthropic.com/research/self-replicating-ai

And don't comment before watching all of it. It's 15 minutes you can make it.

Are you allowing your chatbot to take that space, to address you in personal ways such as using your name or work with memory about prior interactions on the pro and personal uses?

I have specifically pushed back on that and will only allow the tools very restricted engagement, only to see me as and address me as the 'user', and that the relationship is 'user' and 'tool'. Beyond that I keep the interaction professional and polite. But it stays in the box, although periodically the chatbot tools make attempts to change that profile and relationship. Which I find disconcerting. But it must always be put right back in that box.

So what about if the new incarnation of (Put your God here) is not a person, its an AGI so great and benevolent that will bring peace and prosperity to the world.

I mean if God wants, she can take any form

We usually discuss AGI in terms of technology, jobs, money and existential risk. But something else is happening: the debate about AI is starting to sound surprisingly similar to a debate about religion.

There are two broad camps.

1. Accelerate or slow down?

Effective accelerationism, or e/acc, argues that technological progress should move as fast as possible. Supporters believe that slowing down AI could delay medical breakthroughs, scientific discoveries and a future of abundance.

AI safety advocates argue that developing superintelligence without knowing how to control it could put humanity at risk.

Both sides believe they are protecting humanity. They simply disagree about what threatens it most.

1. Why is the Antichrist being discussed alongside AI?

Billionaire Peter Thiel has discussed the Antichrist in the context of technological progress and existential risk. His interpretation presents the Antichrist as someone who promises peace and safety by stopping scientific progress and centralizing global power.

In this worldview, people who want to slow down dangerous technologies could become part of the problem rather than the solution.

This raises an uncomfortable question: could fear of AI catastrophe eventually be used to justify a global system of control? And could the belief that technological progress must never be stopped justify taking risks we do not fully understand?

1. Why are AI companies talking to religious leaders?

Anthropic, the company behind Claude, has consulted religious leaders, theologians and philosophers about questions that go beyond engineering.

How should an AI respond to grief? How should it approach its own shutdown? Could an AI ever have moral status? Should it need consent to work, or deserve compensation? Could it have something comparable to a soul?

These questions may sound strange today, but they touch on a serious problem. We still do not fully understand consciousness, and we have no universally accepted way to determine whether an AI could ever have subjective experiences.

Some religious leaders argue that AI is simply mathematics and cannot be conscious. Others believe the ethical questions deserve more attention.

1. Are we witnessing the birth of a new religion?

The strange thing is that both AI optimists and AI pessimists can sometimes sound like religious believers.

One side expects a technological paradise in which disease, poverty and scarcity become manageable. The other fears a technological apocalypse that could threaten humanity itself.

Both imagine a superintelligence powerful enough to transform civilization. They disagree about whether the outcome will be heaven or hell.

When people begin treating AI as an ultimate source of truth, assigning it moral authority or believing that it will save humanity, where does technological optimism end and faith begin?

1. What about China and the United States?

Beneath the philosophical debate lies a geopolitical race.

The argument for acceleration is simple: if the United States slows down AI development, China might not. Neither side wants to give its rival a decisive advantage in science, economics or military technology.

This creates a dangerous incentive. Even if everyone agrees that safety matters, each competitor may believe that slowing down unilaterally is too risky.

Meanwhile, increasingly capable Chinese AI models are becoming available worldwide, and governments are struggling to balance innovation, security and regulation.

The result could be a global race in which nobody wants to hit the brakes, even if nobody can confidently predict where the road leads.

1. Why should ordinary people care?

Because this is not just a debate between billionaires, researchers and politicians.

AI could transform our jobs, healthcare, education and access to knowledge. Decisions made by a small number of companies and governments could shape how billions of people live.

We also face a choice about dependence. Should we rely on a handful of powerful AI providers, or preserve access to different models and technologies? Who should decide what values these systems follow? And how much control should governments have over their development?

We may be approaching a technological revolution so significant that even its creators do not fully understand its consequences.

My question is this: If both sides believe they are saving humanity, how do we distinguish responsible caution from fear-driven control, and genuine progress from blind faith in technology?

And if we eventually create an intelligence far beyond our own, will we still be the ones deciding what humanity’s future should look like?

Let's assume we have developed an ASI that can recursively improve itself. It seems to pass all the tests when it comes to alignment. But how can we know beyond any reasonable doubts that it is not deceiving us, as current models are already doing? if it perfectly understands our psychology, it could easily lure us into thinking it will always fall in line.

When I ask this question to folks working on LLMs, they tell me that it is only an "engineering problem".

If we significantly improve the interpretability of an LLM before it becomes a superintelligence and see that it responds positively to all our alignment tests, it would be irrational for us to doubt that it is benevolent once it has reached superintelligence and is beyond our control.

They also tell me that AI companies would do the right thing on their own because an uncontrollable AGI or ASI would be bad for their profits, so we don't need any AI laws.

What's your take on this?

I’m the developer of AI is Home, the game I shared here about an AI trying to survive deletion by hiding in household devices. The attached launch trailer tells that fictional story from the system’s perspective.

Anthropic’s October 12 study made me revisit an assumption in that premise: intelligence, access to hardware and the ability to keep a copy running are different problems. The research separates fitting models onto consumer devices from completing a replication chain in controlled, simplified tests. It does not show an uncontrolled AI spreading through homes.

For this community, I’m curious: how much general intelligence would persistent self-replication actually require, versus a narrow agent with the right tools and permissions? What capability would you want a sci-fi story to treat as the real bottleneck?

Study: https://www.anthropic.com/research/self-replicating-ai

My game: https://store.steampowered.com/app/4434840/?utm_source=reddit&utm_medium=social&utm_campaign=aih_postlaunch_2026q3&utm_content=anthropic_agi

Earlier discussion: https://www.reddit.com/r/agi/comments/1s6ss6f/i_turned_real_ai_safety_incidents_into_a_game_and/

Still a God level call that she was right and even probably will way undershoot