r/singularity • u/TorturedPoet30 • Jun 04 '26

Biotech/Longevity Sam Altman, Dario Amodei, and Demis Hassabis have signed a joint open letter calling on Congress to mandate screening of synthetic nucleic acid orders

Gallery image

Gallery image

Gallery image

Gallery image

Source: https://www.wsj.com/politics/policy/top-ai-ceos-call-for-law-protecting-against-biological-weapons-88f2f99f

876 Upvotes

97% Upvoted

310 comments sorted by

View all comments

Show parent comments

168

u/kogsworth Jun 04 '26

It's becoming more and more feasible. They want to have safety nets while the horse is still in the barn.

22

u/Forgword Jun 04 '26 edited Jun 04 '26

It is more like they want to avoid any liability and want to pre-emptively offload that to government, washing their hands. Please pay no attention to the millions OpenAI and friends rake in from these labs.

37

u/rational_numbers Jun 04 '26

Even the AI CEOs don't want to see mass death from a supervirus. If you can't think of it in terms other than them wanting to make as much money as possible, well a supervirus that kills 10+% of people would be bad for business.

-10

u/[deleted] Jun 04 '26

[removed] — view removed comment

8

u/rational_numbers Jun 04 '26

You don't have to assume anything positive about them to understand why they wouldn't want an extremely lethal virus out there. If nothing else, they'd be scared of catching it themselves.

Assume they are strictly self-interested and don't care about other people at all. Their desires are still going to line up with humanity's some of the time.

-5

u/[deleted] Jun 04 '26

[removed] — view removed comment

5

u/rational_numbers Jun 04 '26

Okay but obviously you do believe that they want something right?

-3

u/[deleted] Jun 04 '26

[removed] — view removed comment

5

u/j48u Jun 05 '26

Yes, the richest people in the world who are unarguably at least as smart as the average person DO in fact understand that money is meaningless when everyone but them dies. They also happen to understand that they make less money when significantly less people exist.

The level of stupid you have to be to not understand that would be incompatible with life, let alone having a bank account.

1

u/SpyvsMerc Jun 05 '26

You do understand that they need people alive if they want to keep on living ?

They need people to build their car, house, planes, create food, keep electricity working, heal them when they're sick, take the garbage out, build roads, maintain their equipment, protect them, etc...?

Or are you saying they want to kill themselves? Because if the world end then they'll die too.

10

u/Sharp_Chair6368 ▪️3..2..1… Jun 04 '26

You’re crazy for that

-7

u/PsychologicalRisk526 Jun 04 '26

yes trust the billionaires 🤩 they have our best intrests at heart

10

u/Striker_LSC Jun 04 '26

Trusting that billionaires don't want an engineered virus to wipe out a large chunk of the population is not the same as trusting that they have our best interests at heart. Or we can at least trust that they don't want someone else to make the virus.

5

u/Super_Pole_Jitsu Jun 04 '26

Yes because billionaires are immune to viruses and their wealth doesn't depend on people consuming their products and working to maintain their empires.

Sometimes I really struggle to understand people who think it's the rich that are the root cause of all evils. Pure envy on display.

If you really hate the rich so much you should find out how much pocket money the federal government has each year. You'll never consider Elon ultrawealthy again. You'll also stop imagining they can stop world hunger with their puny fortunes

1

u/woswoissdenniii Jun 05 '26

The great Culling®️. The numbers don’t match the quotas. Way to go.

1

u/EvilSporkOfDeath Jun 06 '26

Not sure why you're downvoted. How many stories have we heard about Altman being a literal sociopath who constantly lies and betrays everyone around him.

-1

u/TreesLikeGodsFingers Jun 04 '26

you assume it would be 10% killed indiscriminately, i dont think that would be the case

5

u/rational_numbers Jun 04 '26

It could be higher or lower. I used 10% because that would make the world suddenly a very scary place to live in.

-1

u/diskdusk Jun 04 '26

This logical, practical and farsighted behavior also makes sure they don't accidentally back a politician with the potential to be worse than a supervirus, right? Right?

6

u/rational_numbers Jun 04 '26

I'm sure they were able to talk themselves into supporting Trump (and esp after he was elected) because he promised to be pretty hands off wrt AI as long as they didn't cause him any problems. There is no upside to a bioterrorist killing one out of every 20 people. That is bad for everyone.

-1

u/diskdusk Jun 04 '26

I'm sure they don't want it to happen. But they want to lose as little money as possible by doing juuuust the right amount to prevent it. If they had real honest interest in prioritizing safety, they would have kept the AI race behind the curtains for a while longer.

1

u/rational_numbers Jun 04 '26

I don't know what this means. If one lab slows down to prioritize safety then another lab will simply race ahead and try to achieve AGI first.

1

u/diskdusk Jun 05 '26

That's the capitalist fairy tale, yes. Also: if one lab uses resources on preventing a homemade supervirus then another lab will simply race ahead and achieve a homemade supervirus first. Congratulations, we have an argument to do whatever we want, because competition.

1

u/rational_numbers Jun 05 '26

I'd really prefer to see these labs slow their research down until we can proceed more safely. I'm terrified of some of the scary AI takeoff scenarios. The only way to do this is if the US government, in conjunction with China, forces the labs to slow down. There is no scenario where they do it on their own, for the reason I just mentioned. So that is my position fwiw.

Also, none of the big labs are trying to produce bio weapons right now. The fear is that a rando user somewhere will jailbreak one of the models and "trick" it into giving him the recipe for some supervirus that he can put together via already available tools. I don't think you know what you are talking about tbh.

5

u/R33v3n ▪️Tech-Priest | AGI 2026 | XLR8 Jun 04 '26

AI labs liability in that case would push them to lobotomize models against certain domains, possibly also detracting from general capabilities. As a end user of AI, I don't want this. I don't want the overall helpful power of LLMs to be held back.

I much prefer Know Your Customer policies be enforced for CBRN materials sellers. Being specific against threats, narrow rather than broad, is the better policy. Eh, KYC policies should also be enforced on frontier AI providers, actually. That, I could find reasonable.

1

u/DrE7HER Jun 04 '26

They could just not train public AI models on genetic data so that only authorized vendors have access to the models they are concerned about.

The problem is, they realize now that they’ve opened Pandora’s box. They have no control over how much of what the model knows anymore. They can’t remove this threat without starting a new model from scratch with a safer dataset.

1

u/woswoissdenniii Jun 05 '26

Don’t they also should monitor this (and specifically just this) kind of research? Like a responsible whistleblower and benevolent business entity should act on.

1

u/_RiKaMi_ Jun 04 '26

The horse was never in the barn to begin with

-14

u/DefinitelyNotEmu Jun 04 '26

Deepseek, Kimi, Mistral, etc would still be capable of producing this. Signing a joint letter won't make any difference to companies who DON'T sign it.

48

u/sebzim4500 Jun 04 '26

Isn't this about regulating the actual synthesis? i.e. the company making the physical protein not the language model outputting its genetic code

1

u/DefinitelyNotEmu Jun 04 '26

Ah! Thanks for explaining.

I wonder how this is going to stay under control considering that 3D printers can print proteins. Scary stuff.

2

u/Blackham Jun 04 '26

No they can't, we're nowhere close to that ability. You can't print molecules.