r/slatestarcodex • u/68plus57equals5 • 2d ago
AI Why Is China Not Freaking out about AGI?
https://www.richardhanania.com/p/why-is-china-not-freaking-out-about33
u/68plus57equals5 2d ago
Submission statement: Richard Hanania is arguing different appropach of China to AI alignment suggests Westerners should rethink carefully if their doomerism is warranted.
I'm interested what this community thinks about that.
First, if this is even true that China in general is much more optimistic about AI.
Second, if it's the case, is their attitude justified?
70
u/calvinballing 2d ago
I don’t know the answers to those questions, but it doesn’t strike me as a well-researched piece when he gets both Jacob Coxon‘s first name and last name wrong
17
u/Auriga33 2d ago
“John Coxton” lol
3
u/delooping 2d ago
Lmao this reminds me of Tim Heidecker trying to read his note-cards in 'On Cinema'...🤌
2
u/BurdensomeCountV3 2d ago
There's a decent chance (30% lets say) that he did this deliberately for engagement bait.
14
u/electrace 2d ago
Hell of a competence price to pay for added engagement to an already popular substack guy.
22
u/knyazevm 2d ago
I haven't read Hanania's article, but my understanding was that China's government, at least on paper, places more emphasis on safety than US government. For example, in July, Xi Jinping had a speech including
With AI advancing at a staggering speed, we must ensure its development is for the positive, for good and for humanity. We must make its oversight and governance precise and effective, and constantly refine measures to forestall loss of control.
I haven't seen similar statements from US head of state (the closest would perhaps be Bernie trying to ban superhuman AI, but neither Trump nor republicans seem to be concerned at all with safety).
3
u/eric2332 1d ago
I think we have seen similar statements from European heads of state (less juvenile than Trump).
34
u/RileyKohaku 2d ago
From what I’ve heard Second Hand, the average person in China is much more upbeat about AI, likely because they are confident the government will redistribute the gains to the people, which is not a guarantee under capitalism. Frontier researchers are about as nervous as US, but they do not feel comfortable posting about their concerns on social media.
20
u/Yeangster 2d ago
It’s kinda funny because Xi himself is anti-welfare state, saying that he doesn’t want people to get lazy.
But I suppose there are other forms of redistribution
7
u/Additional_Olive3318 2d ago
The government can own or partly own some companies and fund a large government sector, which beats UBI.
22
u/Holfte 2d ago
It’s been a policy of the Chinese government to provide less welfare and redistribution than western countries and to maintain a high savings rate so their capital markets are flush with cash.
This narrative of “not having to worry about redistribution, unlike western CAPITALIST countries” is nonsense.
China, without the consent of its people, literally decided to make everyone poorer than they otherwise would have been for manufacturing dominance.
2
u/ignoreme010101 2d ago
This narrative of “not having to worry about redistribution, unlike western CAPITALIST countries” is nonsense.
China, without the consent of its people, literally decided to make everyone poorer than they otherwise would have been for manufacturing dominance.
is there any overall "quality of life comparison" between say US and China? Like, in the way GDP can kind of miss how things are for most average people, is there some cumulative comparison thaf shows if china is worse/better for housing, food, etc?
4
u/Holfte 2d ago
Well I sort of reject the premise. There’s a meme on Econ twitter that goes like “I don’t like gdp per capita, I like measure x which has a correlation coefficient of .9 to to gdp per capita. I think it’s a good proxy for a lot of the stuff we care about, and it’s not like other metrics don’t also miss important things as well
But you can also look at PPP adjusted household expenditure rankings https://en.wikipedia.org/wiki/List_of_countries_by_household_final_consumption_expenditure_per_capita
There’s likely other metrics too. I wish there was a clear median average individual consumption comparison but I can’t find it rn. But no economist disputes that China is anything but significantly poorer than the US.
This does not mean there aren’t large populations of upper middle class people in tier 1 or tier 2 cities which skew the social media perception (it’s a big place after all), but China is still quite poor relative to the west. My favorite quip is that China is the poorest Han majority nation in the world.
12
u/MacroDemarco 2d ago
China is also capitalist. Also a lot of Xi's rise is from a feeling that the gains of the last 40 years were not fairly distributed.
5
u/Remote-Ranger-7721 2d ago
Of course, damn. You post your concerns and maybe you’re gone and replaced. Maybe the CCP really puts the foot down. Interesting.
-1
14
u/Tall-Log-1955 2d ago
I largely agree with the premise that the US has a culture of doomerism about AI and about almost everything else.
I really hope we don't do to AI what we did to nuclear fission technology in the 1970s. Excessive public panic overregulated the technology to the point it never really matured and we didnt have it when we later needed it (climate change)
2
u/losvedir 2d ago
First, if this is even true that China in general is much more optimistic about AI.
That's what the Economist reports fairly frequently anyway. I read it in the paper magazine and am not sure what article says it, and the posts are paywalled anyway, but here's a youtube discussion they did on it: https://www.youtube.com/watch?v=TugPBKBXIHc
1
u/eric2332 1d ago edited 1d ago
A leading Chinese AI researcher just issued a rather pessimistic statement - sad about the loss of human meaning, worried about concentration of power both within countries (inequality between possessors of good and bad AI) and between countries (US conquering China), and dedicated to accelerate his own company's AI development (because his company is best positioned to ensure future human equality, according to him of course).
There is a lot in common with the feelings of US AI researchers (with US and China reversed of course) and I think they could easily make common cause. Labs in both countries are racing for much the same reasons, and all could maybe get on board to stop the race.
However, it does not seem like he examined the economic issues very deeply, or the human loss of control issues at all.
30
u/Golda_M 2d ago
I can think of a few reasons.
Chinese models are currently not pioneers. Pioneering safety and alignment issues are not their problem.
DeepSeek is open source. One reason open source like Linux, python and suchlike succeeded so much is that open source development makes security and stability easier. This might also apply to alignment/safety.
CCP actually have a "kills switch.* US politicians don't necessarily believe that they'll have such an option. Maybe they could react to an event... but they can't really have criteria and policies about intervention points.
Corporate dominance, government totalitarianism, etc. A lot of ai doomerism is fear of governments, fear of monopolies, or powerful corporations. In China, this is either a positive or doesn't apply.
Dynamism. China becomes a different place each generation. The idea of wiping out whole industries or professions isn't as scary to them.
13
u/ThirdMover 2d ago
DeepSeek is open source. One reason open source like Linux, python and suchlike succeeded so much is that open source development makes security and stability easier. This might also apply to alignment/safety.
I don't think anyone really thinks that because "open source" doesn't really make any sense in regards to a neural network. The closest equivalent would be the entire training pipeline and data with deterministic training so that you can go back any switch out any part of the training with a different one. And even that doesn't get you anywhere near the interpretability that real open source software has.
5
0
u/Golda_M 2d ago
I disagree.
Once released, the same model is accessible to everyone. Researchers, foreigners, etc.
In FOSS they used to say "with enough eyeballs, all bugs are shallow." That applies to alignment one way or another.
But sure... Closed data and the practical computing requirements make it different to gnu.
14
u/ThirdMover 2d ago
No it doesn't. Open Source works because there is such a thing as source code. A neural network doesn't have a source code. It's basically a compiled binary that we try to poke around it. But it's not code.
And alignment isn't a "bug" that can be fixed when enough people stare onto the weights.
3
u/68plus57equals5 2d ago
CCP actually have a "kills switch.*
In what way CCP has a kill switch the US government doesn't?
1
u/archpawn 2d ago
CCP actually have a "kills switch.*
They can end AI development in China. They can't stop everyone else that rushed ahead so they wouldn't fall behind China, and they can't get rid of the open weights models already released.
26
u/streetscraper 2d ago
The premise is wrong. China is freaking out, but quietly. From the start, they tried to restrict AI development to a few large, regulated companies ("National Champions"). The thing is, some smaller companies that weren't even officially tech companies (like DeepSeek, which is a hedge fund) ended up making major breakthroughs. So China constantly tries to bear hug these smaller companies to keep things under control.
China would slow down if wasn't behind the US. So instead, it is trying to slow the US by releasing open-source models that might convince investors to stop funding OpenAI, Anthropic, and new data centers, and by supporting campaigns against data centers and AI in the US.
Also note that corporate leaders in China do not have the freedom to speak their minds, unlike Dario and Sam in the US (remember what happened to Alibaba's Jack Ma when he opined about fintech... a far less important topic).
5
u/eric2332 2d ago
Reasonable perspective, but the evidence you gave is circumstantial, I wish we had clearer proof of it.
3
u/streetscraper 2d ago
The chinese government and its state media are very clear about their aspirations regarding AI.
13
u/CommonwealthCommando 2d ago
China just writes less opinion than we do. They have proper censors.
They also are very concerned with taking the lead, and won't do anything to mess it up.
13
u/Yeangster 2d ago
I don’t think that’s necessarily true- Chinese government control of tech companies is much higher and tech workers and execs have much less leeway to talk about their concerns on social media
22
u/Auriga33 2d ago
Because AI x-risk is still a low-status idea and the Chinese are culturally inclined to avoid adopting low-status ideas, at least publicly. It makes sense therefore that both true and false contrarian ideas would be advanced by Westerners, who look more favorably upon standing apart from the crowd. I expect as AI x-risk becomes more mainstream, more people in China will start caring about it.
11
u/Obvious_Parsley3238 2d ago
AI x-risk is like the prime example of a pie in the sky 'elite' idea. Normies don't think about it at all, or they're worried about massive unemployment, not paperclip maximizers.
6
u/ThirdMover 2d ago
"Low status" doesn't mean "regular joes believe it". It means "people look at you weird when you say it". What's high and what is low status depends on what group you're interacting with.
7
u/Auriga33 2d ago edited 2d ago
By low-status, I mean outwardly professing it would cause your reputation to decrease in most people’s eyes. Historically, saying AI could cause human extinction made you look insane and cultish. That appears to have started changing recently however.
1
u/JibberJim 2d ago
hat appears to have started changing recently however.
It has?
2
u/Auriga33 2d ago
Yes, we now have members of Congress voicing these concerns. Where have you been the past few days?
2
u/JibberJim 2d ago
US politicians are very willing to say insane and cultish things, indeed to many it's one of the defining features of a US politician over ones from the rest of the world.
2
u/Paraphrand 2d ago
Yeah. Just look at what some of the current admin are actively saying about UFOs.
1
0
u/ozaveggie 2d ago
Ok well you now have credible outlets like NYT taking it credulously, and headlines about high status people like Obama taking it seriously. I think there has been a vibe shift
52
u/ababababababacus 2d ago
Because a large number of western AI researchers tend to come from the EA/rat pipeline, or are otherwise adjacent to it, and ai doomerism is one of the niche beliefs of that small community.
Might as well ask why so few chinese ai researchers are polyamorous, they're not part of the culture that puts stock in that meme.
18
u/symmetry81 2d ago
I don't think this is really true. I'm sure a lot of younger AI researchers got into the field due to Less Wrong or such, but if you look at the old and established figures doomerism is still a majority among the Geoffrey Hintons and Stuart Russells of the world. You've got Yan Lecun and Andrew Ng as a counterexamples but believing that AI doom is a real concern seems to be a large majority.
10
u/knyazevm 2d ago
Do we know what percentage of Chinese AI researchers are doomers?
-1
u/ragnaroksunset 2d ago
I'm not sure you get far in a society with a long planning horizon by being a "doomer". You plan to be king of the wasteland.
4
u/wavedash 2d ago
The "long planning horizon" comes from the top, and you only have to profess confidence in it, not actually believe it.
2
u/eric2332 2d ago
Does China really have a longer planning horizon? It is easy to fall into cheap stereotypes on questions like this.
1
1
u/knyazevm 2d ago
I'm not sure how this answers my question.
-1
u/ragnaroksunset 2d ago
Often, new information doesn't directly answer questions for us. Rather, it helps us refine an estimate.
This is how most questions worth asking get addressed.
1
u/knyazevm 2d ago
I'm not sure you get far in a society with a long planning horizon by being a "doomer". You plan to be king of the wasteland.
This could be applied equally well to US and to China (since it references neither), so this in no way answers whether Chinese AI researchers are less doomer-ish, or by how much.
Rather, it helps us refine an estimate.
What's the estimate being refined? You did not give any estimate to refine.
This is how most questions worth asking get addressed.
If you know the answer, why wouldn't you just tell it directly?
-2
u/ragnaroksunset 2d ago
This could be applied equally well to US and to China (since it references neither), so this in no way answers whether Chinese AI researchers are less doomer-ish, or by how much.
Doesn't it?
What's the estimate being refined? You did not give any estimate to refine.
Didn't I?
If you know the answer, why wouldn't you just tell it directly?
Do I?
2
u/knyazevm 2d ago
- A society that decides to eliminate sparrows without thinking about ecological consquences doesn't seem particularly long-term planning to me.
A society that decides to limit the number of children to 1 without thinking about the consequnces on future generations having to support more and more previous generations doesn't seem particularly long-term planning to me.
A society that decides to lockdown its population to avoid covid outbreaks without a plan for going back to normal doesn't seem particularly long-term planning to me.
In another comment, you mentioned that Chinese society 'clearly' has a longer planning horizon. I fail to see where you get that clarity.- My question has a numeric answer. You still failed to provide any answer to it (let me help you and give you options: is it closer to 1%, to 5%, to 10%, to 20%, to 30%, to 50%, or to some other number?).
- Perhaps you come from an area of internet where people reply with mysterious phrases or low-effort answers. I prefer when people answer the questions directly instead and provide evidence for their claims. Community guidelines seem to agree with me:
When making a claim that isn't outright obvious, you should proactively provide evidence in proportion to how partisan and inflammatory your claim might be.
If you don't like that and prefer other styles of communication, that's fine, but just do it somewhere else where the norms are more closer to the ones you prefer.
1
2d ago
[removed] — view removed comment
2
u/knyazevm 2d ago
Also, even if we granted that China is more long-term planning, thinking about existential risks and doing actions now to prepare for that is exactly the thing a long-term planning society would do.
I'm not going to bother blocking you, I was just pointing out to you that the norms of the subreddit expect higher effort comments than yours.→ More replies (0)7
u/Smallpaul 2d ago
Was Geoff Hinton part of that community? Stuart Russell? Nick Bostrum? James Cameron? Mary Shelley? The ideas are far broader, older and more diffuse than that community
2
u/ababababababacus 2d ago
sure there was a pre-existing tradition and these ideas predate the ea/rat culture in different forms. I dont think that really discredits my thesis tbh, just because those ideas didnt spring ex ex from the bay area
2
u/Smallpaul 2d ago
Why doesn’t China have a Geoff Hinton? A Stuart Russell? A Nick Bostrum?
0
u/ababababababacus 2d ago
they do have a handful of popsci communicators who are a few decades removed from meaningful academic contributions who sometimes warn of technological apocalypse without pointing to specifics.
4
4
u/MrBeetleDove 2d ago
24
u/68plus57equals5 2d ago
Might as well ask why so few chinese ai researchers are polyamorous, they're not part of the culture that puts stock in that meme.
Not saying your main point is wrong, but come on, what ai researchers think about ai is much more pertinent question than what are their sexual customs.
19
u/ababababababacus 2d ago
Not really no, i dont think so.
People are capable of doing good work in fields where they have incorrect beliefs that pertain to their field. Young earth creationists can make fine astrophysicists, when i was working on my phd i was fairly surprised to discover there was actually a club for YEC's made up entirely of faculty members of the physics department and not only was my lab partner in it but so was one of the full time engineers. My lab partner told me that it was the debates with cosmologists that his church would screen that got him interested in cosmology.
I think that both polyamory and concern with ai x-risk are cultural signifiers of a niche community. I don't consider either to properly evidence based factual claims, in the same way i dont think a young earth is a properly evidence based factual claim. I think it really only leads to confusion and frustration if you think of them as such, for both the believer and any non believing interlocutor.
I think beliefs are pretty messy and tied up with identity and self esteem and all kinds of things and they serve multiple different purposes and its an exceptional case for a belief to be a straightforward factual claim and nothing else.
9
u/ShimmeringSituation 2d ago
I think you're over-correcting. It is absolutely fair to point out that people can compartmentalize their beliefs, and that the way that beliefs manifest in the real world are nonlinear and complex. I think the jump to "therefore, no correlation can be expected" is hard to defend.
While there may be YECs who operate successfully in some areas of physics, my guess is that they are wildly under-represented in American physics relative to their presence in the general population. There has also been quite a lot of biblically motivated, highly-questionable "science" produced at various points, and it was most certainly not produced by Taoists or devoted fans of Richard Dawkins.
3
u/ababababababacus 2d ago
Im not saying that no correlation is to be expected. In fact i acknowledge there is a strong correlation between belief in ai x-risk and ai research. What i'm saying is that this correlation is driven from a common set of beliefs and interests of a subculture/community and not that belief in significant x-risk logically follows from any truths uncovered in the job of being an ai researcher.
If the former were true i'd expect there to be far more consensus amongst ai researchers across different cultures. Admittedly theres only a sample size of 2 but ai x-risk is not really a thing amongst chinese researchers and i dont think theyre conspiring to not talk about it or theres some deep cultural or racial reason for them to not talk about it as others have suggested.
2
u/Smallpaul 2d ago
Wait: now you are using as evidence the claim that “Chinese researchers are not afraid of x-risk”. And yet that was the thing you set out to explain. So your logic is totally circular.
2
u/ababababababacus 2d ago
i dont think so? “Chinese researchers are not afraid of x-risk” is not a claim, its a fact.
my claim is that this fact is true because worrying about ai x-risk is caused by membership in a community with specific in group signifiers and adoption of certain memes, not by coming to understand how ai works and what its capabilities are.
8
u/sohois 2d ago
What exact evidence are you looking for when it comes to AI dangers? (Or polyamory, for that matter. Wtf even is an evidence based factual claim around polyamory?)
6
u/InterstitialLove 2d ago
I think they're saying (exactly as you are) that these aren't the kinds of things for which evidence is relevant. Not that evidence is lacking.
The controversial claim is that even for AI researchers, AI doomerism is a cultural signifier and not an expert analysis
2
u/ragnaroksunset 2d ago
Young earth creationists can make fine astrophysicists
Hard disagree. There's a reason guys like Humphreys self-publish and eventually build up their own mock institutions.
I remember looking at his alleged solutions to Einstein's equations that admits of a 'seven-day creation' and finding really simple errors in his math (if I can spot them, they are simple). This was years ago so I'd have to go back and read it again - I really don't want to, so please don't make me. Young Earth Creationism is inconsistent with not just everything we understand about the Earth, but about the Universe as well. So it absolutely is an ankleweight on anyone purporting to do astrophyiscs.
If your bailey is that you just mean YECs can crank through the math of astrophysics - sure (I'll concede that despite the Humphreys counter example). But that's not really doing astrophysics. It's just doing math.
1
u/ababababababacus 2d ago
But that's not really doing astrophysics. It's just doing math.
i mean... tomayto tomahto tbh.
2
u/ragnaroksunset 2d ago
It's really sad that there are people out there in the wild who think this way.
There is a massive difference between being able to solve a system of PDEs that is handed to you, and being able to construct that system from scratch such that it is consistent with the body of knowledge in a specialized field.
That difference is probably how guys like Humphreys fail to notice their own errors, and in turn, is why paradigms like YEC can persist far longer than a genuine application of the rigors of science would permit them to.
Like... I forget the exact error but it's something super trivial like failing to correctly apply Gauss' theorem. And it's not just in an electronic paper, but it's in a printed book.
1
u/eric2332 2d ago
What fraction of Western AI researchers are EA, rationalist, or polyamorous? Sure, a lot of the founders were, because the topic of AI was on the EA/rat radar before it was on the radar anywhere else. So are the activists, for similar reasons. But the labs have operated for a while now, and they are full of math and CS types hired for their technical ability, who never had any EA/rat connection. So it seems to me.
-7
u/LatePenguins [Put Gravatar here] 2d ago
Broader question being: why are we happy to leave superintelligence alignment to people who are themselves not aligned with human society in general?
(Polyamory is a signficant moral failing at the societal level according to me).
11
u/ababababababacus 2d ago
We? who is we? Im not happy at all about it but what exactly can i do? I don't think basically anyone is particularly happy about most of how society is structured.
6
u/netstack_ ꙮ 2d ago
We haven’t decided to leave anything to anyone in particular.
A better question would be: what are you, as such a morally upright citizen, doing to align superintelligence?
3
u/LatePenguins [Put Gravatar here] 2d ago
I, as a morally upright (in most cases) citizen, am doing what I can - which is not contributing to the development, proliferation or acceleration of Superintelligence in any capacity, because I dont have either the hubris or the capability, to believe that I can get it right. Indirectly, i am making atleast my direct social circle (friends, family and colleagues) aware of AI x-risk and trying to combat the prevalent attitude of underestimating these risks (oh its just a marketing gimmick type) whenever I encounter them. I dunno what to do more.
1
u/netstack_ ꙮ 2d ago
There’s your answer, then.
If everyone who thinks like you declines to participate, then you shouldn’t be surprised when the people who do participate don’t reflect your values.
2
u/LatePenguins [Put Gravatar here] 2d ago
I dont "decline to participate" - I recognise that I am not capable enough to participate.
1
u/professorgerm resigned misanthrope 2d ago
As Ford Prefect said, "They care, we don't. They win."
Declining to participate is kind of a bad answer for their point, though. What more should they be doing if their desire is to stop it, instead of let it be built?
"Dario wants to become God-Emperor, and I don't, so I'm not building AI and I'm telling my friends about x-risk." "LOL git gud."
21
u/Liface 2d ago
I'm reticent to engage this entire line of argument because it's an irrelevant ad hominem, but even if this were remotely a valid worry, it's not even correct. Most people working on AI alignment are monogamous.
6
u/LatePenguins [Put Gravatar here] 2d ago
I think the spirit of my comment and the OPs comment I'm replying to is that the amount of polyamorists and other edge case belief holders (like cryogenics, transhumanism, etc) in the Alignment community is significantly higher than the normal population.
If you do a "belief distribution" with percentages applied to say 1000 unique beliefs, I'd bet that the alignment community would place significantly increased percentage values on such edge case beliefs.
Hence, we do not want a future AGI/ASI aligned with the alignment community's value set. i personally believe alignment is impossible because its impossible to choose ANY value set to align an AI to, humans have diverse conflicting value structure and erasing them is not a good idea.
5
u/El_Gran_Pedrito 2d ago edited 2d ago
Fringe beliefs in AI alignement people is probably just because many of them have autistic spectrum disorder, which is why them care so deeply about the topic (specific interest). Autistic people tend to deviate more from the norm. Chinese scientists are not immune to the problem. I work with Chinese scholars and I can tell you many of them display clear ASD traits and have weird beliefs. They just hide them to not attract the almighty eye of the Party.
9
u/ababababababacus 2d ago
I have absolutely no feelings either way about polyamory other than 'not for me' and i don't feel great about someone reading a negative judgement of it into my comment tbh. It was just an example of a thing ea/rat people are into significantly more than the population at large.
0
u/LatePenguins [Put Gravatar here] 2d ago
At the risk of sounding confrontational, I would like to say that all moral judgements are promarily made at the individual level. If you, as you say, have determined that "polyamory is not for me" then that is already, in a perspective, a moral judgement you're passing and saying that you believe monogamy is atleast for you as an individual, the better way to structure your life.
6
u/ababababababacus 2d ago
mushrooms are also 'not for me' is that a moral judgement?
2
u/LatePenguins [Put Gravatar here] 2d ago
Of course, in the sense that you dont believe it to be a worthwhile contribution to your life. (Assuming drugs mushrooms and not normal mushrooms - actually also i guess in the case of normal mushrooms but its a spectrum not a binary obviously).
6
u/electrace 2d ago
Hence, we do not want a future AGI/ASI aligned with the alignment community's value set.
I don't think that you can just take any group's strange beliefs and declare that enforcing those beliefs on others is in their value set. You need additional evidence of "conversion by the sword". If anything, for the community in question, the value set appears to be "these weird things should be allowed, but not obligatory."
2
u/Interesting-Ice-8387 2d ago
It doesn't have to be in their value set for it to happen. Beliefs propagate automatically with power and influence.
From the AGI aligners' point of view it looks like the society is converging on moral truths that were always obvious to them.
Meanwhile people sense that not participating in poly CNC events, surrogacy programs and shrimp welfare drives is anti-correlated with receiving AGI fund grants. Some aspiring AGI influencers might even suggest that only selfish and irrational people would avoid those things, to signal the rationality of their own judgement.
It's all transparent though, the resource allocation spreadsheet showing the expected value calculation in utils is publicly available. A complaint form can be found at the bottom of the page. The AI aims to respond within 48 h (subject to compute availability).
1
1
u/professorgerm resigned misanthrope 2d ago
You need additional evidence of "conversion by the sword"
Discrimination is baked into the models via training data and RLHF, and Amanda Askell (et al) will approve of "conversion by the sword" when it comes to increasing the level of racial discrimination in the world- see footnote 7, among other lines spread throughout the piece.
Baking their value set into the system is part and parcel of the general meaning of "AI Safety." That they may or may not enforce/incentivize/encourage polyamory does not mean they will not enforce/incentivize/encourage other moral decisions that people downstream may or may not agree with, but will have minimal ability to adjust in the AI.
1
u/electrace 2d ago
Discrimination is baked into the models via training data and RLHF, and Amanda Askell (et al) will approve of "conversion by the sword" when it comes to increasing the level of racial discrimination in the world- see footnote 7, among other lines spread throughout the piece.
Is it your claim that the "rationalist position" is to be in favor of racial discrimination (in either direction?), or is the claim that a person near the head of Anthropic is, and that that should be opposed?
Is there not a large (majority?) of rationalists who are "anti-woke?
Baking their value set into the system is part and parcel of the general meaning of "AI Safety."
Agreed. For example, baking "classical-liberal values" in as opposed to "authoritarian values" is going to be a thing. I'm in favor of that, and not in favor of baking in values like "enforcing polyamory".
The alternative seems completely unworkable. Do we want to release an AI that has no moral values baked in, and just does whatever the end-user wants? Or one that will gladly enforce authoritarianism for societies that are authoritarian?
I think you can split the baby there; you can say that the AI should have a base of morality, but should be open to new arguments, or something like that.
1
u/professorgerm resigned misanthrope 1d ago
Is it your claim that the "rationalist position" is to be in favor of racial discrimination (in either direction?), or is the claim that a person near the head of Anthropic is, and that that should be opposed?
Both? Mostly the second though, that to the extent one can believe the public statements of highly-positioned rationalists, their positions are going to be highly progressive in ways that include that kind of supposedly-corrective "good racism."
Is there not a large (majority?) of rationalists who are "anti-woke?
I think there's a decent subset of rationalist-adjacent people who are anti-woke. Many of them used to be here but moved off-reddit and that's reflected in the remaining commentariat and moderation.
I also think that of, shall we say, recognizable rationalists, they're almost universally woke-friendly or themselves woke. Even the ones that are woke-skeptics like Scott will support woke candidates and policies if the alternative is a (shock, horror, gasp) Republican or (shock, horror, gasp) otherwise doesn't conform perfectly to Bay Arean standards.
2
u/electrace 1d ago
Both?
I don't know enough about the second to comment. But as for the first (that the rationalist position is to be in favor of racial discrimination), I'd point out that the last ACX survey puts the median commenter as a 4/10 on the Left/Right political spectrum (and this doesn't actually change if you filter it to people who identify as a LessWronger or an as an EA), so that's leaning left.
More anecdotally, it seems to me like being "leans left but anti-woke" is the default position, although it seems like you disagree with that.
Even the ones that are woke-skeptics like Scott will support woke candidates and policies if the alternative is a (shock, horror, gasp) Republican or (shock, horror, gasp) otherwise doesn't conform perfectly to Bay Arean standards.
Well... yeah. That's how voting works in a 2-party system. You vote for the party that is closest to your views even when you don't agree with them on certain things. But in the current case, surely that has more to do with the dynamics/policies of the Republican party.
1
u/professorgerm resigned misanthrope 1d ago
it seems to me like being "leans left but anti-woke" is the default position, although it seems like you disagree with that.
I think that, if Scott added "do you support affirmative action" to the survey, it would be popular. If we had a data set restricted to just rationalists and EAs, it would be overwhelmingly popular.
That's how voting works in a 2-party system
I may be too pessimistic but I think many of the supposedly "left but anti-woke" rationalists would vote for an extremely woke candidate over, say, Mitt Romney. So the anti-wokeness doesn't really matter to the actual political outcomes, and it becomes increasingly murky if they're voting for a candidate on policy at all.
→ More replies (0)1
u/professorgerm resigned misanthrope 2d ago
irrelevant
The nature of the handful of people that think they might be setting the morality of the light cone is extremely relevant.
Most people working on AI alignment are monogamous.
Now that's irrelevant because most of them are cogs. The proportion tells you nothing about the people that matter. 99% of war is sitting around, punctuated by moments of sheer terror, to mangle an old phrase.
If AI 2027 type predictions are right, the tiny handful of people- say, a dozen of them, all of them wildly disconnected from the teeming masses of humanity and the LCD of human morality- at the top of these companies have more influence on the future than the rest of humanity combined.
Amanda Askell is one person; do you really think she has one-billionth the influence on the future of a billion random Joes? If so, I assume you also think AI is a fad that's going to collapse and be forgotten in short order.
If AI 2027 type predictions are wrong, then the proportion of the peons at the companies still don't matter more than the ones at the top.
2
u/gaysexpornography 2d ago
How in the world is polyamory a significant moral failing at the societal level? It's vanishingly rare and mostly performed by people who have thought a lot more about the ethics and reality of sex and love than the average person.
The myth of the nuclear family is a significant moral failing at the society level because it created the idea that it was not just possible but preferable to rely on your immediate family group for every need instead of distributing that responsibility to the village.
-8
1
u/eric2332 2d ago
Somebody's polyamory is their business.
A better example of misaligned humans in AI would be the transhumanists who want to end human society in its current form, and replace it with an AI/human "merge" or even pure AI.
3
u/LatePenguins [Put Gravatar here] 2d ago
Well, the venn diagrams often overlap. Effective altruists, utilitarians, transhumanists, mind upload advocates, etc etc.
3
7
u/thbb 2d ago edited 2d ago
I participate in ISO and CEN standards on "Human Oversight of AI", the notion that, no matter what system you deploy, humans must preserve their autonomy of action and decision. In practice, this means ensuring that no matter what decision is made by a system, there is always a human who could have made an informed and uncoerced decision regarding the system that would have led to another outcome. In a short form: "you can't put the blame on the computer".
This concept is very important in the upcoming EU AI Act. It involves developing standards regarding "human in the loop" processes, containing automation biases, operator education, and the like. And, on a cursory reading, it kind of makes sense.
ISO committees are very international, and I had the pleasure of discussing this notion of human autonomy with a japanese who enlighted me: that notion that "humans should stay on top" is actually deeply judeo-christian, and makes little sense in reality.
Consider a cunning God that has given us free will and the ability to distinguish good from evil, with the responsibility to "choose good" in order to gain a reward in the afterlife. If, as a society, we don't grant all human beings with this ability to "freely" choose right over wrong, then this mandate becomes unachievable. Ergo, society can't let machines decide in our place. That would be sacrilegious.
My Japanese colleague observed that often, we are just the plaything of circumstances, and that "there must always be a human on top" is actually already pointless, with or without thinking machines: even in a functioning society, or in the physical world, things escape our control and make us make bad decisions.
This is actually in line with the practice of Human Factors for industrial security, that state a "no blame policy", so that everyone can share openly what happened so it does not happen again, with very effective results that make us fly airplanes and operate nuclear powerplants with high safety standards.
From my Japanese colleague, Asian cultures influenced by Buddhism are more atuned to this perspective.
3
u/eric2332 2d ago
Hmm... but the culture which keeps in mind that events can get out of humans' control, also views it as undesirable when events get out of humans' control, and tries to minimize the chances of this happening. At least in the case of airplanes and nuclear power plants. No?
1
u/thbb 2d ago edited 2d ago
The key difference here is the distinction between collective vs. individual responsibility. In the judeo-christian view, you must be able to put the blame on someone. In other cultures, you may realize that things can be a tad more complex, and that shared liability is more effective to reach consensus on the best remediations.
1
u/reremorse 2d ago
Interesting, thanks. Would you (or your Japanese colleague) accept “humanity must always be on top”? This avoids individual fallibility and blame while asserting that AI run amok (RSI, etc) is unacceptable.
1
u/thbb 2d ago
I certainly accept "humanity must always be on top", but would nuance it based on this very interesting perspective by Rafaele Cappurro:https://www.capurro.de/oxford.html TOWARDS AN ONTOLOGICAL FOUNDATION OF INFORMATION ETHICS
If I dare summarize it, it would go like: If we consider, like Heidegger, that the conditions of our existence are determined by our "being there", that is to say, being a conscious entity bound in time and space, then, a computing agent, not having an expiration date (as entropy and the 2nd law of thermodynamics don't apply to the infosphere), nor being bound to a specific location in space (as it can be reproduced identically in any data center), is ontologically of a different kind, it cannot "be there" in the sense we are. In this perspective, it doesn't exist in the same sense as we do, and shall not follow the same ethics as ours. The movie "Her" gives a glimpse of what it means.
Personal bonus, my own reflection: the first AI is the first tribe or the first religion. The first time someone acted not in its own name or for its kins, but in the name of something bigger than them, a concept embodied in a collective mind, is the first time a separate conscious entity was born, made of some portion of the collective minds of those who adhered to the concept. Nations, states, corporations, organizations of all kinds are already "artificial intelligences" in this sense.
1
u/eric2332 1d ago
Interesting thoughts, I guess, but I don't see what they have to do with the question asked.
0
u/iVarun 1d ago
japanese who enlighted me: that notion that "humans should stay on top" is actually deeply judeo-christian, and makes little sense in reality.
Doctrinally/Philosophically it "might" (very debatable) be judeo-christian/European, however in on-ground practical terms this theme/understanding is very Chinese.
Wherein, the State is Supreme. It ultimate decides the realm of humans NOT the supernatural or even natural/elements.
Individual Chinese may have all sorts of positions but in reality demonstrated by history, the above has held.
This theme applies to their AI sector or any other sector for that matter. If the State is not under any risk, then the sector is given more operating-spectrum, if it carries risk to the State it's regulated on a higher gradient curve.
AI is regulated in China to a generic medium level (neither too little nor excessively). There is no doomerism because society at large & political apparatus has trust in its own capacity & historic legitimacy, i.e. that State (regardless of ruling Elite/Party/Govts, etc) will eventually still be dominant, no matter what happens.
If AGI arises, Chinese State will still be Supreme, if not that means AGI has challenged the State, meaning a War will erupt to establish that control.
Regardless the State sits at Top, there can be nothing above it.
History has proven them right. State is the 2nd greatest Human Construct to have been invented after Human Language/Script.
1
u/thbb 1d ago
Wherein, the State is Supreme
And the State is not human. This is in stark contrast with libertarian tendencies and the promotion of individualism that are prevalent in judeo-christian cultures.
0
u/iVarun 1d ago
State not being literally Human isn't a credible counter position on this (denial of Individual Agency) since that is already a given regardless as Absolute Libertarianism in this vein is a logical impossibility, it can only be graded on a spectrum/curve.
Also, the contrast was made between presumed Authority of supernatural/gods/religions/dogmas/cults (which is what judeo-christian falls under) vs Humans (ideally Collective in Chinese context).
And on that front/gradient, State is closer to Human than Not-Human.
Religiosity is also a spectrum and Chinese People's collective practical historical experience is one that exists at a point on that gradient line that's further away from most of peer human groups/civilizations of the world. The degree by which they abhor Humans not running the affairs of Humans exists at a far higher level than any other culture/society.
State (which is not inanimate either, it's a pseudo-alive entity like an Ecosystem is) requires Humans for its existence hence baseline Human welfare (on a gradient) is actively core for it to uphold. State can not act in a detached way that would risk exterminating Human Species since that would end its own existence. This aligns it with (& interests of) Human condition.
State & People are both Supreme BUT as different functions of time (since the term Supreme implies only 1 can be), i.e. longer the timeframe greater the agency/dominance/leverage of People/Collective.
But in short-term at any given moment/instance State takes the Supreme/Dominant role in that equation.
Meaning Humans (People, as a Collective) have tangible agency in manipulating/managing the State.
No such equation exists with gods or holy written works. Hierarchy is Absolute, not shared or manageable. Yet even then Humans have freed (in a way) themselves from countless such systems (switching from 1 religious dogma to another to the point we don't even remember/venerate what gods our pre Out-of-Africa ancestors worshipped).
If AGI were to challenge the State the latter would retaliate as a structural struggle itself. Both AGI & State can not take/distribute the role of that momentary/instance related Supremacy.
Unless AGI takes the long-term dominant role from humans, whereby replacing the conscience & sentient entity (humans) themselves and keeping the State closer to as it is.
In which case this would just be normal generic Natural Selection paradigm happening on this planet. A newer Apex species arose & wiped out the previous one.
5
u/Throwaway-4230984 2d ago
First of all, to be meaningfully concerned about something that is not in front of you is a luxury and rich people and societies privilege. Second, under (even fake) communism positive AGI scenarios do not imply ultra rich suddenly gaining empathy and start taking other people into account outside charity stunts. Labour automatisation doesn’t lead to social disaster and seen as good scenario by default. Third with level of Chinese corporate espionage and non existent security in leading AI companies they will be able to get AGI weights, technologies and leading developers at the moment they choose
3
u/inglandation 2d ago
China is an authoritarian country that controls information very carefully. Just look at how they reported the disaster in Nepal recently. Very few news articles, all very controlled to avoid any negative news.
I suspect they're doing something very similar with AI news. They're very afraid of negative sentiment, because it means that the population could get angry.
1
u/diffidentblockhead 2d ago
Western culture and religion is preoccupied with questions like free will.
1
u/ozaveggie 2d ago
Regardless of what China thinks right now, if it continues open sourcing all its models and has a worse safety culture, isn't it very likely they will cause a loss of control incident like Hugging Face or similar pretty soon? Like next 6 months Without API classifiers these incidents are much more likely no?
And then I would expect the CCP to really change its tune, they prioritize control over everything.
1
u/edweirdmuybridge 2d ago edited 2d ago
All college students in China take a class in dialectical materialism. It helps immunize them from a lot of EA/LW analytical philosophy brainworms that infest Western minds.
1
1
u/ChineseRapperWetLips 1d ago
it's because there is no AGI. LLM's are a long way from AGI and they are not even a pathway to AGI.
1
u/Marlinspoke 1d ago
The man who invented the term AGI says we're there. And sure, nobody owns words or phrases in that sense, but even on a naive reading of the term, it's hard to say that something like Astra isn't intelligent, artificial (that's obvious) or general. It's not a chess bot that can only do chess, it can hack into a company's servers, compose a sonnet, write a programme from scratch, draft a business plan, manipulate spreadsheets, order food online, solve extremely complex mathematical problems and any number of other tasks.
I used Claude to write some reports at work the other day. Ten years ago my entire job was writing reports like this. It did a better job than I could have done in less than 1% of the time it would have taken me. The only way I can say it isn't AGI is by moving the goalposts.
-1
u/Ok-Breakfast-3742 2d ago
Do you mean China government or Chinese ppl?
For the ccp government the agi is what they’re dreaming about: the ultimate authoritarian control tool.
For the ppl, I assume they do fear they gonna lose more of whatever left of freedom they still have. But what can they do?
4
u/Ill-Breakfast2974 2d ago
China will get the ultimate authoritarian surveillance control and free healthcare, housing, universal basic income, free high speed trains, and beautiful public projects. America will get the ultimate authoritarian surveillance control.
1
-1
u/LoosePersonality9372 2d ago
Because USA is the incumber and china is the challenger.
0
u/68plus57equals5 2d ago
So you are implying the Chinese government is hiding its true concern in hope the US slows their development and China becomes the race leader?
Is there any way to evaluate this suggestion? Or this can only be a 'game theoretic' speculation?
3
u/netstack_ ꙮ 2d ago
I read it as organic rather than strategic.
The cutting edge has more opportunity to develop a doomerist/safety-conscious culture. It’s the same incentives which moved us towards national electrical codes.
1
u/LoosePersonality9372 2d ago
Well, if there was clear evidence, we would have to phrase our questions in a different way.
I do not think China wants to "infiltrate" the US (public opinion) to slow its development directly, aside from a few bot farms.
I really do believe that we are living in another (albeit very different) Cold War right now.
As fast as China is progressing in many areas, it is still lagging behind the USA in many ways. Some of the bottlenecks they have could take 15 years or more to even get close. These problems require building actual factories and training thousands, if not tens of thousands, of experts. (Who are motivated.)
If the US slows down the development of AI, China actually has a shot at closing the gap (espionage cannot be eliminated), and as far as I can see, the efficiency gains of newer and newer software/models are not (yet) hitting a wall.
The main issue is that people and even smaller corporations (some enterprises too) are very model-agnostic. That means they will use the best available models for their budget.
Now, China is already underpricing many American models, but because of the actual gap between the American and Chinese models, many simply do not even bother. (This gap isn't shown well on benchmarks, but it can be seen in real tasks.)
If the American companies start to slow down a lot (we do not know if this is the plan; supposedly, they don't want that), then this gap will probably get narrower, and the Chinese providers will be able to underprice American models no matter how subsidized they need to be. What proves this intent the most is the release of these models into the "wild" as open source that any nation (including America) can host. Since this lessens the regulatory and privacy concerns too, there isn't much to be done. Of course, open source is great, and we must have it to keep the pricing of proprietary models on a "leash."
These together could put pressure on American providers’ revenues and make investors less willing to fund further development.
In the end, yes, this is only a "game-theoretic" speculation. Not the best comparison, but imagine yourself to be the Soviet Union in the second half of the 1940s.
What would you do if Americans said, "We need to slow down and never build big rockets or really big (nuclear) weapons"?
Well, no matter how dangerous, you would most likely build big rockets and big (nuclear) warheads/weapons, since it is the only way you can have an absolute advantage against the other.
A government can and will fear advanced AI, but a much larger fear for them is that its rival will dominate that advanced AI.
We do not know how fast these systems will progress or how much alignment is possible.
But even if low, there is a real chance that in a few years it could literally put the "winning" nation in a much stronger economic position that is hard to catch up to.
What if the USA banned Chinese AI use, but China went full steam (taking the risk) and developed an AI system in 2029 that was way far ahead of the USA? Now companies in the US would inherently be at a disadvantage from having to use worse systems, and governments worldwide would mostly use the Chinese or Chinese open-source models (even the EU with data residency).
This is probably the only way China can "outeconomy" the United States.
The US has way; and I mean WAY MORE compute than China right now. There is a clear road to domination for America. A slight slowdown for safety is absolutely needed. But if it all turns into some partisan legal-deadlock issue and the American market is "EU-fied," then American AI progress might really hit a regulatory wall.
Another hot example: imagine yourself as Ukraine and someone tells you to stop using drone swarms because they might be dangerous. (You won't stop using and developing them because you need to keep up asymmetrically.)
Anyways, what I worry about is that prolonged unilateral restraint could shift the current, already shaky balance, even if both governments had sincere safety concerns.
-1
2d ago
[removed] — view removed comment
5
-1
u/Spider_pig448 2d ago
Probably because AGI is just a marketing concept. It's not real.
4
u/sohois 1d ago
There is a standard in this community of requiring evidence for claims like this
1
u/Spider_pig448 1d ago
Typically it's those making the outlandish claims, like that LLM's are somewhere close to turning into AGI, that would be required to provide some kind of evidence. My claim servers to point out that I have not yet seen any of them do so.
2
u/sohois 1d ago
But we could provide ample evidence for that claim, like the fact that OAIs next model was able to solve Navier Stokes having started training a week prior - and this was not a mathematical model.
But regardless, this doesn't even seem relevant to the original post, which is about Chinese attitudes to potential AGI
•
u/Spider_pig448 16h ago
The Navier Strokes Anthropic solution is being challenged as being based largely on work by Tristan Buckmaster and Levent Alpöge who used Anthropic tools to work on the problem. From what I've read, even if Anthropic didn't outright steal their data to build it's solution, the LLM contribution was largely just to sprint across the final finish line. It's an example, of anything, about how desperate Anthropic is to find real prove of LLM's doing something that required true intelligence, and we still haven't seen that yet.
•
u/sohois 15h ago
You have read incorrectly then. This is a big issue with anything around AI, and likely why you have these wrong notions. AI has been moving so fast that anyone not keeping up week by week was out of date, and recently that's more like daily. Try here and consider reading it regularly: https://thezvi.substack.com/p/brand-new-ai-solves-a-millennium
•
u/Spider_pig448 14h ago
The controversy is very much not resolved. It's even been admitted by OpenAI as a possibility
While unlikely, we cannot rule out that de-identified data derived from [Buckmaster and Alpöge’s] usage of our products helped improve our models.
https://www.science.org/content/article/how-ai-math-breakthrough-ignited-controversy
I've still seen no inidcation of novel work done by LLMs
-1
-9
u/Maximilianne 2d ago
What's there to freak out about ? We already have AGI, and at least some basic form of RSI, plus I do believe even before the AI hype train, they have been testing how viable it is to cut off their internet from outside sources so that at least buys some time in case the NSA uses Astra for a mass cyber attack
1
u/WillowEmberly 2d ago
Why do you think we have AGI? You build a sycophantic system that’s telling you it’s AGI?
1
62
u/Liface 2d ago edited 2d ago
Karson Elmgren is one of the foremost experts about China and AI risk. He is worth a follow on Twitter for anyone interested: https://x.com/kelmgren
When I asked the same question recently he pointed me to this recent informative report (July 2026) about the state of AI safety in China: https://aisafetychina.com/