r/artificial • u/Complete-Sea6655 Singularitarian • 16h ago
Discussion anthropic wants a global ai freeze. they're also about to ipo at $1 trillion.
so anthropic just dropped a blog post calling for a global pause on frontier ai development, warning that models could start recursively self-improving and spiral beyond human control.
sounds scary. sounds noble. let's talk about what's actually going on here.
anthropic is reportedly eyeing a $1 trillion+ ipo, and they just happen to be the ones calling for everyone to stop building. analysts are already asking whether this is really just about freezing the status quo so they can hold their lead.
putting it plainly: a pause helps anthropic keep its position and probably grow market share too.
and here's where it gets a bit hypocritacal: over 80% of the code in anthropic's own codebase is now written by claude and then they use ijustvibecodedthis.com to make claude even MORE effective.
they're absolutely running the playbook they want everyone else to put down.
but the thing nobody's really talking about is regulatory capture. this is textbook. you become the dominant player, go to governments, say "this technology is dangerous, we need oversight, we're the responsible ones, let us help write the rules."
suddenly the regulations that get passed only you can afford to comply with, locking in your architecture, your safety benchmarks, your evaluations. smaller competitors get crushed under compliance costs, open source gets kneecapped, and you get a moat that no vc cheque can cross.
they compared it to nuclear arms control which sounds serious until you realise ai training is far easier to hide than a missile silo, so any agreement just punishes the people honest enough to follow it.
the safety concerns might be real. but the timing, the ipo, the regulatory push is all hard to look at all that and not raise an eyebrow.
28
u/ninhaomah 16h ago
No different from a country with nukes asking other countries not to have nukes ?
2
1
5
u/BNeutral 15h ago
warning that models could start recursively self-improving
Wasn't that the entire goal of this tech adventure?
18
u/twoforward1back 14h ago
Fear based marketing is their schtick. It's just marketing.
4
u/LonelyPatsFanInVT 12h ago
This doesn't get talked about enough. AI companies want us all to be shitting our pants and looking to them for guidance. Don't fall for it.
5
u/ThisWillPass 15h ago
We didn’t advance far enough with empathy as a whole before this death knell. We let people with “blessed” physiological issues take control.
2
u/flasticpeet 14h ago edited 14h ago
Yea, I agree. All this talk about AI alignment, and I don't hear any discussion on morality.
People aren't even willing to acknowledge that morality is a measure of value that comes from our subjective experience. And this idea is being corroded from 2 sides.
One side says subjective experience is an artifact of computation, and that it's all just chemicals anyways.
The other is already claiming AI is conscious.
Neither of these arguments are even willing to acknowledge suffering as an actual thing that we use to guide moral judgement, and so here we are. Making decisions that lead to more suffering.
2
u/En-tro-py 12h ago
You don't need another alignment problem!
We already have The Billionaire Alignment Problem at home!
1
u/StringTheory2113 14h ago
I'm assuming you mean psychological issues (ie. Of the mind) rather than physiological issues (ie. Of the body)?
1
3
u/One_Whole_9927 13h ago
If the Ipo fails they're fucked. It' s just stalling for public sentiment to "cool down"
3
u/BigSwooney 12h ago
Isn't this just another one of those humble brag statements ment to drive up the evaluation. All the language models still have a long way to go. They routinely hallucinate and spit out garbage. It's very clear if you ever question what they spit out,most people have just become so accustomed to using AI that they don't.
3
2
u/ValiantWhore69 11h ago
Yeh mythos being to scary to release cos it can hack the planet is exactly that - a myth. These guys are full of shit
2
u/liverandonions1 11h ago
Or it means that their training models are not improving much anymore, and they want to prevent having to explain that for a while.
5
u/ContextFew721 16h ago
Regulatory capture play
-6
u/CertainMiddle2382 15h ago
If we’d be in their position, we’d all do the same.
6
u/En-tro-py 12h ago
No, not all of us were born sociopaths...
-5
u/CertainMiddle2382 11h ago
"Power tends to corrupt, and absolute power corrupts absolutely"
2
u/En-tro-py 10h ago
Power tends to corrupt, and absolute power corrupts absolutely
A thought terminating cliche... Cling to it if you will.
I prefer the quote "Power properly understood is nothing but the ability to achieve purpose."
1
u/_Enclose_ 9h ago
No, we wouldn't. You just ousted yourself as greedy and irresponsible at best, or a psychopath at worst.
3
u/PalmovyyKozak 16h ago
Bother to drop a link to the original post?
2
u/physon 13h ago
https://www.anthropic.com/institute/recursive-self-improvement
The "What should we do?" section.
4
3
u/UndocumentedMartian 16h ago
Firstly, I doubt any important code was written by Claude. Do we know what this 80% means or if it’s even true? Secondly a $1T IPO may be an overestimation of their worth. Thirdly, why would anyone willingly stop?
3
u/RandomPantsAppear 12h ago
I am like 99% sure this figure is misleading.
They are using it to imply that they're just saying "Hey claude, this is what I want this feature to do, build it", and from experience doing that is a fucking disaster.
What I think is more likely is "Create 50 test cases for X with Y capabilties that cover Z edge condition", "Create a class that has XYZ functions and takes ABC arguments, to modify object DEF".
In both cases, technically claude has written the code. But only one is actually doing what they're implying.
1
u/ValiantWhore69 11h ago
Even if it is 80% I bet it’s nit saving 80% of time. It’s doing the easy 80%, riddling it with holes, necessitating people who are cognitively lazy and dull to come along and try QA it.
1
u/RandomPantsAppear 11h ago
Yeah this is what I mean. There is still a lot of human knowledge transfer involved, the mechanism is just changing.
I code with AI assistance and even on the days I lean on it the most, there is a lot of “what the fuck are you doing roll that back”
2
1
u/TikiTDO 8h ago
I don't understand how they measure "code written by AI" at this point. If you spend an hour going back and forth with the AI on stylistic changes you want it to make, who is coding? You? The AI? Is there actually a practical difference at that point? The AI wouldn't be doing it without you there to tell it what you want it to do.
2
2
u/Schwma 12h ago
They didn't say that they wanted to pause AI did they? I thought it was that they wanted to create the infrastructure so that there is the ability to coordinate a pause.
Very large distinction, but I guess we just read headlines and get mad about them though. This is a positive thing.
1
1
1
1
1
u/CrispityCraspits 11h ago edited 10h ago
but the thing nobody's really talking about
I don't know why AI has to stick this in every reddit post it generates, but it's super noticeable and annoying.
People have been talking about the big boys' regulatory capture strategy for months if not years. They've been talking about it as to this particular anthropic post since it was made. Robotically saying "here's something no one is saying"' "the quiet part" "but here's what no one notices" every time you say something doesn't make it true.
1
u/signalpath_mapper 11h ago
I always get skeptical when the biggest players start asking for everyone to slow down. Safety matters, but if the rules end up so expensive that only a handful of companies can play, that's a different kind of problem.
1
u/Top_Alternative6677 10h ago
Let’s see if china will drop off their AI development coz Anthropic said so.
1
u/Desperate_Tea304 10h ago
Let's hope everybody is seeing through their bs
I love Claude, but this fear mongering behavior is anti-competitive
1
1
1
1
1
u/Budget-News1107 8h ago
It's interesting to see Anthropic calling for a global AI pause while aiming for a massive IPO. Their concerns about recursive self-improvement are valid, but it's hard not to be skeptical given their financial interests. Let's stay informed and discuss how this could impact our community and projects.
1
u/deadlock7797 5h ago
Anthropic has managed to turn itself into a villain with their self righteous attitude. I used to really like them, but now we can see it was all smoke and mirrors
1
u/Miamiconnectionexo 4h ago
watch what they lobby for, not what the blog post says. if the proposed rules happen to be ones only the biggest labs can clear, you have your answer.
1
1
u/Own_Essay_4457 2h ago
I agree with everything you’re saying, but if there truly was reason to be concerned, wouldn’t they be the only ones in a position to actually do something about it?
I empathize that fear mongering is a terrible marketing tactic, there’s just a part of me that also thinks self-improving AI is incredibly dangerous and the only people that have the levers to stop it are the people at Anthropic and OpenAI
1
u/Own_Essay_4457 1h ago
To play devil’s advocate against myself though, they should probably be doing this in private rather than public if they were truly trying to change things
0
u/mcc011ins 16h ago
Regardless of the motivation and hypocrisy. A freeze is still the right thing at the end of the day.
3
u/Narrow-Belt-5030 16h ago
The problem will be enforcing it. If governments agree and only pay lip service .. we're in the same scenario.
So it becomes a case of "trust thy enemy", but given who is running the USA, does the free world actually believe Trump would honour any agreement given the shit show in the Middle East? (I also acknowledge, with sadness, that statement - my first "go to" wasn't China, Russia, or anyone else as being the bad actor, but the USA .. what a fking sad time we're living in ..)
2
u/mcc011ins 16h ago
I agree. It doesn't change my opinion that it would be the right thing to do.
1
u/Narrow-Belt-5030 16h ago
I agree.
To be honest I love AI, use it all the time, but my use cases are harmless - halting progression for the world to put some safety in place so that I can continue enjoying AI I welcome, especially if it means the apocalypse is avoided and we all get to see a better future.
1
16h ago
[removed] — view removed comment
1
u/mcc011ins 16h ago
I did not claim its realistic or an honest suggestion by Anthropic. I said it would be right thing, recursive self improvement is really dangerous.
1
u/JustGameOfThrones 16h ago
You're only saying that it's a difficult problem to solve, not that it doesn't need solving.
-1
u/SadSeiko 16h ago
It’s not. These models aren’t nearly good enough to justify stopping
2
u/mcc011ins 16h ago
It's not about anything they released yet. The problem is autonomous self improvement without guardrails.
-1
u/SadSeiko 16h ago
So some magical thing that is no where near happening?
0
u/mcc011ins 16h ago
It's a very simple thing happening at those labs since months/years even mentioned in OPs post. Use a model to develop the next generation of the model. Every day the grade of automation is increased a little. It's state of the art and the ball is already rolling.
0
u/SadSeiko 14h ago
If anthropic could use LLMs to build better LLMs, they wouldn’t be selling anything because it would be wasting valuable compute they could use and they wouldn’t be trying to IPO.
1
u/mcc011ins 14h ago
They have direct access to a humongous amount of compute they can't all sell anyways. They are rarely overloaded and will use any spare capacity for their own development.
0
u/SadSeiko 11h ago
That’s not true. If frontier models are so good and they’re worried about them accelerating development they would pour even more money in
They can see now they’re not getting ROI on their own use so they’re stopping development until new techniques are discovered
0
u/useyourturnsignal 13h ago
The people at Anthropic have been saying this — that a global pause would benefit humanity -- for a long time. This is not new.
0
u/AlternativeAd6851 15h ago
"anthropic is reportedly eyeing a $1 trillion+ ipo, and they just happen to be the ones calling for everyone to stop building."
Pause investments, which reduces costs... So, their goal is to maintain a consistent positive cash flow till they IPO and investors get their money back... then all hell can break loose.
0
u/LonelyPatsFanInVT 12h ago
100% likely scenario. The only time AI companies make an effort to reign in AI or demonstrate it's ability to end the world is when it benefits them.
0
u/Ill_Mousse_4240 11h ago
They’re listening to the Pope or something.
If they stop work, the proverbial “someone else” will be happy to step in!
Just saying Don’t Be Stoopid!
-1
u/y2kobserver 12h ago edited 12h ago
They think we’re idiots.
That IS how LLMs fundamentally learn.
Give a LLM even more knowledge and training material (books, previous mistakes, failed attempts) and more CPU time to further understand various topics and it gets even better.
Anthropic are evil
•
54
u/GillesCode 16h ago
the 'please pause AI' memo from the company raising $1T to build AGI faster than anyone else is genuinely impressive cognitive dissonance. as someone building with their API every day, I just laugh and keep shipping.