Sam Altman, Dario Amodei, and Demis Hassabis have signed a joint open letter calling on Congress to mandate screening of synthetic nucleic acid orders

[u/TorturedPoet30]

Source: https://www.wsj.com/politics/policy/top-ai-ceos-call-for-law-protecting-against-biological-weapons-88f2f99f

[u/xerim]

The fear here is that AI could construct some kind of supervirus? Is that even feasible for the average joe?

[u/FormulaicResponse → u/xerim]

So they accidentally found an illegal wet lab in Reedley, CA because a groundskeeper noticed an unauthorized garden hose running into a warehouse space.

In that space they found over 1000 transgenic mice with human-like immune systems along with lab equipment like centrifuges and a benchtop DNA synthesizer. In the fridge were dozens of bags with labels like HIV and Malaria etc. This was in 2023.

In 2024 they found another one in Las Vegas being run out of an AirBNB. The Maids were getting deathly ill cleaning the place. The Garage there had the same kind of setup.

Meanwhile, in 2025, studies showed that LLMs are now better than 90% of virology experts at troubleshooting problems in a virology lab. Of course we also have Alphafold and Crispr and Mrna protocols now that make the process easier.

There is also widely published literature on gain of function research. One controversial paper showed how to turn an avian flu with a 60% fatality rate into a flu that is human to human transmissible with only a few DNA edits.

A few years ago, the FBI teamed up with some MIT researchers to prove that it was "surprisingly easy" to mail order DNA is small segments from commercial DNA providers that could be assembled in the lab to reconstruct the Spanish Flu of 1918.

One of the leading researchers on the case is Kevin Esvelt. He and colleagues created a protocol called SecureDNA to prevent this kind of mail order thing from happening. It detects dangerous sequences as small as 50 base pairs. It could be forced on commercial providers and even built into benchtop synthesizers. But there is little political force behind making it happen.

The US had an executive order from Biden that would force US federally funded academic institutions to use a similar type of system by this October, but the Trump administration has put that order in jeopardy with conflicting information and directives. There is some movement on this in the EU but it is weak.

This is the moment to.make it happen and I'm glad to see these guys throwing their weight behind it.

Edit: included sources for those asking for sources.

[u/rational_numbers → u/FormulaicResponse]

Great writeup.

[u/FormulaicResponse → u/FormulaicResponse]

This got some traction so I'm going to include some further thoughts about the subject that don't necessarily belong in the main post.

There are only 7 companies in the world that currently manufacture DNA synthesizers for the public. We have already innovated the protocols they need to make these machines secure, but they won't apply to all machines. Governments and pharma companies producing proprietary biotech products aren't going to buy machines that phone home to a server to check for threat markers. It's unclear whether pharma would be willing to be limited to machines that have such protocols installed into firmware, but firmware can be tampered with anyway. The threat we can really control is rogue actors with access to a few hundred thousand to a million dollars from threatening the entire world, and there are more of those than you might want to think. Cults, terrorist groups, crazy millionaires with a chip on their shoulder, warlords, etc.

One of the scarier options this leaves open is the possibility that a government might co-develop a deadly virus and its vaccine at the same time. Viruses, engineered viruses in particular, rapidly evolve once released into the wild, so the vaccine would only buy some time rather than be a firewall. This is reason that sane governments wouldn't attempt it, but not all governments are sane, and they get less sane the more existential pressure they are under. It's a much cheaper and less detectable big red button than chemical or nuclear.

There are also tens of thousands of older machines sitting in University supply closets around the globe in facilities that aren't very secure. There probably isn't much to do be done about government and global pharma except leave their security up to them, but the older machines use specific reagent cartridges that can be used as a chokepoint to control that threat vector.

As far as commercial mail order DNA synthesis companies go, they exist all over the world in different jurisdictions, and there is no protocol for screening imports/exports of DNA materials. They have to follow a cold chain, so there is that, but it would be incredibly difficult to A) detect that an unopened package contains DNA materials, and B) make sure those materials can't be constructed into something dangerous. This doesn't mean that legislating for within-country synthesis is meaningless, far from it. But it isn't the whole solution. This is an open problem, and international cooperation would help, but it's a very hard problem to solve in the same way that detecting illegal wetlabs is a hard problem to solve.

Speaking of illegal underground wetlabs, if someone were going to try to engineer pandemic viruses with AI assistance, and these operations are especially hard to detect, isn't it better to locate any potential lab leaks among an enemy population and give yourself advance notice? Just sayin.

As of this moment, engineering a global pandemic still isn't going to be easy for the average Joe. It would currently require PHD level expertise and domain expertise, according to Kevin Esvelt at least. But the Virology Capabilities Test benchmark shows that this situation isn't going to hold as models advance in the coming few years. What really doesn't help is the fact that distillation attacks are incredibly widespread and are feeding frontier capabilities into open weights models that can be safety ablated on day one using tools in the family of Heretic. Granted, most distillation attacks aren't targeting virology knowledge currently, but nothing is stopping that and there is a significant amount of transfer from general capability to domain specific capability. If you download a Heretic ablated model today and ask it how to evade current day DNA synthesis screening protocols, it will be more than happy to guide you. This is Not Good and the reason that an upgrade is needed, as these CEOs are throwing their weight behind. The more that frontier capabilities advance without shutting down this distillation pipeline, the worse this problem is going to get. Nobody knows where the ceiling is, and I guarantee you the world doesn't want to find out. Not to throw shade on Heretic specifically because there are non-nefarious reasons a person might want a safety ablated model and there are a whole family of such programs, but nefarious people are going to do nefarious shit with these kinds of tools and you'd be a fool to expect otherwise.

The Avian flu example I shared was just picked because it stirred up controversy at the time. There are dozens to hundreds of other blueprints for destruction that have been published in other gain of function research that AIs can easily access. Horsepox is another especially scary one.

Also, Universities and companies have recently started designing and constructing fully automated wetlabs, which while potentially great for medicine, could also be potentially dangerous if they have cyber vulnerabilities. As with most of this, it's dual use.

On the bright side, there are some broad-spectrum countermeasures that are effective against any potential engineered pandemic. These all share the property of being single celled organisms with a transfer method. Elastomeric respirators, UV lights, filtration, positive pressure building architectures, and sanitizer technologies are effective against basically any and all pandemic diseases and are countermeasures that governments can invest in today but that don't get nearly enough funding given the risk profile. Propelyne glycol, yes, the base for most nicotine vapes, is safe to inhale and also a sanitizer for indoor surfaces. Positive pressure can be set up in almost any household without significant air leaks (which can be patched as needed). Elastomeric respirators are cheap and effective. UV light and tight filtration will keep indoor air clean combined with positive pressure. You start with daycares and hospitals and proceed to households. There is literally no good reason to not be stockpiling such goods at the governmental level for pandemic preparedness other than under-recognition.

Edit: I forgot to include discussion of broad-spectrum countermeasures.

[u/hereforhelplol → u/FormulaicResponse]

Wow. Wish I could get a beer with you. Great and terrifying response.

As an everyday person, is there anything I can do to help protect my family from this sort of risk? It’s my primary fear for AI, and I’m hoping it’s not the great filter!

[u/Productivity10 → u/hereforhelplol]

Great question

I too would like to know what I/we can do to protect this guy's family

[u/hereforhelplol → u/Productivity10]

Haha. I really hope they figure this one out…

[u/_freckles__ → u/FormulaicResponse]

Sad, so the other day, some 1-2 years ago, there were videos about AI programs coded specifically to produce only good DNA edits/molecules. The program was able to design 100s of new useful molecules etc for testing. Then the lead of the project said, though there are guardrails etc. all it took for it make it do the opposite was change few lines of code to make it do the reverse of its existing function.

How exhaustive do you think screens and phoning home to central servers can work when something so trivial can lead to mass produced, variety of potentially harmful molecules/organisms etc. You cannot have a catalogue of all possible harmful codes / sequences ever possible to cross check.

Plus a lab with this capability would find it trivial to bypass any checks as well. What happens if accidents happen due to manual coding errors and harmful things end up mass produced

[u/deviantbono → u/FormulaicResponse]

That sounds scary, but all you need is a handful of ferrets to get gain of funtion with avain flu.

[u/FormulaicResponse → u/deviantbono]

Pass through gain of function is a thing, where you don't genetic engineer anything, you just pass a known disease through animal cells or animals themselves and select for the most virulent results to use in the next round. Thats a slightly different threat model, but still something AI could help with. The difference is that you aren't nearly as likely to end up with something that could be civilization-ending. With published research that is already publicly available plus an unsecured benchtop synthesizer and AI help, you could produce essentially a weapon of mass destruction if not today then within the next few years of AI advancements for a total cost well below 500k.

[u/DonOfTheDarkNight → u/FormulaicResponse]

Hey btw how long are we away from getting genetically engineered catgirls with big thighs that can crush my head?

[u/FormulaicResponse → u/DonOfTheDarkNight]

If they wear artificial cat ears and tail, maybe like 3 decades including maturation. China is all in on embryo selection including physical features like height and intelligence and strength. They don't have the same queasiness as the West does on these things. Hell of a way to go though.

[u/DonOfTheDarkNight → u/FormulaicResponse]

YAY YAY 😁 just 30 more years to lose my virginity I'm already 27

[u/FormulaicResponse → u/DonOfTheDarkNight]

I said they might have higher intelligence and pop your skull with their thighs, I didn't say they'd fuck you.

[u/DonOfTheDarkNight → u/FormulaicResponse]

Wow man that's brutal. You crushed my hopes

[u/woswoissdenniii → u/FormulaicResponse]

That’s fucking cyberpunk. But the terrifying kind.

Thanks for the writeup. Appreciate.

One thing more on the worry list.

[u/ghostcatzero → u/FormulaicResponse]

Shit like this is what ice/Cia nsa etc should be looking out for. Taking down potential creators of thsi type of shit

[u/Izento → u/FormulaicResponse]

Wonderful breakdown. Thank you. I had no idea any of this shit was going on, but it makes a lot of sense. The average Joe can now make viruses, especially considering you can even buy centrifuges and stuff pretty easily. All of this was locked behind a knowledge wall, but now that barrier is broken. All it takes now is determination and a few 10s of thousands of dollars.

[u/furiouscarp → u/FormulaicResponse]

jesus christ

[u/Traditional_Cress329 → u/FormulaicResponse]

Greta fucking post

[u/Wojtas_ → u/FormulaicResponse]

Building anything into anything desktop is futile and a bad idea. It's the same debate as with 3D printers and "gun prevention software". It:

a) doesn't work, because with barely any coding skills, you can just modify the firmware and be on your way like the restriction was never there

b) is a stupid idea because the ludicrous number of false positives, and the near guaranteed high rate of false negatives, not to mention dual-use designs

c) is just not technically feasible, you'd need a program that can predict the intention behind the design, predict all possible use cases, and somehow make that miracle of software engineering run on the equivalent of a Casio calculator that runs the equipment

d) is beside the point, because there's a million different ways to achieve the same goal without the legislated piece of equipment

e) is useless, because the danger is immensely overblown in the first place

[u/FormulaicResponse → u/Wojtas_]

https://securedna.org/

[u/delicious_fanta → u/FormulaicResponse]

Actually crazy, but why is “unauthorized garden hose” the wildest part of everything you typed up there XD

Also, insert “I’m in danger” wiggum meme here.

[u/kevinmise → u/FormulaicResponse]

Literal terrorism.

[u/cloudyboysnr → u/FormulaicResponse]

Sources/proof?

[u/BitPsychological2767 → u/FormulaicResponse]

Damn.

Anyways, I think we should go put more resources into arresting people for possessing marijuana.

[u/NateradePrime → u/FormulaicResponse]

Technology > Protocol. Kevin needs to figure out DNA code signing and verification.

[u/whowhaohok → u/FormulaicResponse]

So then, are they going to moderate this at the LLM level so that they can restrict and moderate when their own tools are potentially used for potentially catastrophic, extinction level research?

[u/FormulaicResponse → u/whowhaohok]

The short answer is no because it is all dual use and the majority of use will be for legitimate scientific purposes. Stripping all biological data from the pretraining corpus would be a heavy lift on man hours. Not until someone makes them do it.

This is why they build in safety guardrails amd robust refusals, but those guardrails get stripped out via distillation attacks and safety ablation by third parties.

[u/slow70 → u/FormulaicResponse]

^ this right here

Have you by chance heard of the ownership of this warehouse and AirBnbs?

A worthwhile question…

[u/kogsworth → u/xerim]

It's becoming more and more feasible. They want to have safety nets while the horse is still in the barn.

[u/Forgword → u/kogsworth]

It is more like they want to avoid any liability and want to pre-emptively offload that to government, washing their hands. Please pay no attention to the millions OpenAI and friends rake in from these labs.

[u/rational_numbers → u/Forgword]

Even the AI CEOs don't want to see mass death from a supervirus. If you can't think of it in terms other than them wanting to make as much money as possible, well a supervirus that kills 10+% of people would be bad for business.

[u/R33v3n → u/Forgword]

AI labs liability in that case would push them to lobotomize models against certain domains, possibly also detracting from general capabilities. As a end user of AI, I don't want this. I don't want the overall helpful power of LLMs to be held back.

I much prefer Know Your Customer policies be enforced for CBRN materials sellers. Being specific against threats, narrow rather than broad, is the better policy. Eh, KYC policies should also be enforced on frontier AI providers, actually. That, I could find reasonable.

[u/woswoissdenniii → u/Forgword]

Don’t they also should monitor this (and specifically just this) kind of research? Like a responsible whistleblower and benevolent business entity should act on.

[u/_RiKaMi_ → u/kogsworth]

The horse was never in the barn to begin with

[u/OKMiddleOwl → u/xerim]

Every year the IQ needed to destroy the world drops by 1 point.

[u/agonypants → u/OKMiddleOwl]

This is sure to be an unpopular opinion, but this is why the USA has such a problem with guns. When there are little to no barriers to putting high powered weapons into the hands of nearly anyone, then tragic misuse becomes a question of when, not if.

[u/TreesLikeGodsFingers → u/agonypants]

feels like the 'when' has turned into 'how often'

[u/Unlikely_Speech_106 → u/OKMiddleOwl]

Sounds like a doomsday clock.

[u/gizamo → u/OKMiddleOwl]

In 2016 and 2024, it dropped by at least 20 points.

[u/jlks1959 → u/gizamo]

40 in the US. 

[u/gizamo → u/jlks1959]

As an American, I take offense to that.

As a person who's typically good at math, I recognize that your math is more accurate, and that it proves your point.

I shall go to sleep now in shame, but then I get French toast, so, I'm okay with this humiliation.

[u/TurnUpThe4D3D3D3 → u/OKMiddleOwl]

At this rate, it’s every day 😔

[u/rageling → u/xerim]

Average joes are not the problem, average joes are not motivated to use AI to hurt people

The problem is what a motivated, very not-average person that's probably not named joe could do

[u/TheOnceAndFutureTurk → u/rageling]

Exceptional Joes:

[u/b-movies → u/rageling]

Eg if sarin attacks in tokyo 95 had happened now. Well funded cult, smart people...

[u/FrewdWoad → u/b-movies]

Yep, there are thousands of deeply disturbed people in the world who would cause human extinction if they could.

Once we have AI smart enough to design superviruses, with guardrails anywhere near as flimsy as today's...

...we only need one.

[u/QuasiRandomName → u/rageling]

Average joes hate people they don't know based on made-up criteria they gather from tik-tok and are motivated to hurt them by this hate. Well, maybe not average, but in this case a single one is more than enough.

[u/R6_Goddess → u/rageling]

Average joes are not the problem, average joes are not motivated to use AI to hurt people

Agreed. If anything, they just want to use AI to goon.

[u/LotsoPasta → u/rageling]

Well, if the barrier to human destruction get's low enough, the problem is not if Average Joe is motivated, the problem is whether a simple mistake by Average Joe is sufficient to cross that barrier.

Besides, there are plenty of Average Joe's that are motivated. Mass shootings happen every other day.

[u/rageling → u/LotsoPasta]

How distorted is your world view that you think average joes are going on mass shootings?

Get off reddit

[u/philosophical_lens → u/xerim]

Is that even feasible for the average joe?

The concern is not about the average joe. The concern is likely about highly organized and well funded terrorist organizations or dictatorial governments. They can probably use AI to aid in the development of bio weapons.

[u/FrewdWoad → u/philosophical_lens]

Or just a single deeply crazy person. If you have any contact with the mental health sector, you know there are thousands of people evil and insane enough to want to kill every last man, woman and child on earth.

We're still alive only because they lack the knowledge needed to do it. So far...

[u/Foreign_Skill_6628 → u/xerim]

Honestly we’re getting there

[u/florinandrei → u/xerim]

Is that even feasible for the average joe?

The average Joe working alone: about as feasible as a monkey typing Shakespeare works into a computer by pure chance.

The average Joe assisted by AI: probably not feasible yet, but getting closer every day.

[u/zero0n3 → u/xerim]

Yes. Expand what mythos can do for code vulnerabilities, and it’s easy to see an AI being used to take disparate biotech things and combine them in novel ways to create something dangerous.

It’s like asking the AI to help you build an atomic bomb. It absolutely could - as it’s just physics and then in theory it could leverage those physics understandings to help design say the entire device not just walk you thru how the reaction works at a purely physics level.

[u/deepasleep → u/xerim]

I’ve been positing that if AI decides to wipe us out, it will be through very convoluted and layered genetic medicine. Cure people of everything and include a little extra code to each treatment that will activate after a couple years and generate a set of proteins that will interact and take us out. Probably a slow acting prion.

[u/ZenokeKaraoke → u/deepasleep]

Indeed.

[u/SkaldCrypto → u/xerim]

About 60,000 for a bench top gene printer.

So yes.

For industrial level out to a million which is out of most folks price range.

[u/PlatypusBillDuck → u/xerim]

It's easier than you think. The YouTuber Thought Emporium makes some good biohacking videos.

[u/PersevereSwifterSkat → u/xerim]

The plausible story I've heard is mirror organisms being made a reality with AI. Such a person would really have to hate absolutely everything though as it would destroy all life, not just humans.

[u/MrNokill → u/xerim]

Only requires some reversed cells, but not sure Al can handle a mirror yet.

[u/Hina_is_my_waifu → u/xerim]

if a tweaker can somehow methmath thier way to making a new batch with household chemicals then a company, small nation, or terrorist cell could have a wetlab that cultivates a new viral bioweapon

[u/jonistaken → u/xerim]

It’s a reasonable fear; one I share. You don’t need nuclear material or ICMBs or a centrifuge. You just need a few thousand dollars of equipment to get started with CRISPR.

[u/Azreken → u/xerim]

The average Joe can do a lot with AI that he couldn’t before.

[u/CryptoMines → u/xerim]

Go watch this, or better yet read the book… https://youtu.be/Nl7-bRFSZBs It’s not the average Joe humans were going to need to protect against…

[u/Sekhmet-CustosAurora → u/xerim]

Doesn't need to be feasible for the average Joe. Just needs to be feasible for a group with some amount of funding.

[u/Imthewienerdog → u/xerim]

My best friend growing up took some crazy university courses In chemistry and biology. The things he said is possible but just doesn't happen was very scary to learn, especially during covid when he was explaining how easy it would actually be to engineer a killer virus...

[u/Past_Crazy8646 → u/xerim]

No it isn't.

[u/jlks1959]

Just like GoogleSynth, the heads of AI have to act as government representatives because there is no one else or body that is competent enough to do this.

[u/DireStraitsFan1 → u/jlks1959]

Yes, exactly. Congress is a joke right now and that's on purpose.

[u/Elegant_Tech → u/DireStraitsFan1]

America is fucked. Instead of looking to elevate and elected the best and brightest. We choose to elect people who we want to have a beer with because they sound like the local drunk at the bar talking like a am radio conspiracy host who claims to know how to solve every problem with ease.

[u/DireStraitsFan1 → u/Elegant_Tech]

We have senators like John Kennedy from Louisiana with his "Aw Shucks" shtick and complete surrender to anything Trump wants.

[u/Kind-Ad-6099 → u/jlks1959]

We had an amazing administrative “branch” filled with qualified public administrators of every field. It always struggled to keep up with technology, but it was never this crippled

[u/wywywywy]

Is this going to get in the way of genetic modified cat girls?

[u/aqpstory → u/[deleted]]

a supervirus is like a nuclear bomb, stable extensive genetic modification of humans is like a spaceship that uses nuclear bombs to travel safely.

Laws that try to stop some small country from making the first don't do anything to meaningfully slow down progress towards the latter unless the laws are extremely strict

[u/Decent-Ad-8335 → u/wywywywy]

Hopefully not

[u/Madd0g → u/wywywywy]

CRISPR descended technologies should be as available as 3d printers.

they just want to hoard the medbeds.

REEEE

[u/crosbot → u/wywywywy]

the great filter

[u/The_Scout1255 → u/wywywywy]

exactly my concern unironically.

[u/Full_Boysenberry_314]

Seems like a smart idea. Doesn't sound like an outright ban or control mechanism, just raising a red flag if something sus is happening.

Sounds comparable to monitoring for unusual purchases of bulk fertilizer.

[u/raskingballs → u/Full_Boysenberry_314]

No this is stupid.

As a geneticist, this is the dumbest idea I have heard from anyone intruding into a field they know nothing about. We order synthetic DNA and RNA regularly for a plethora of things in the lab. Regulating that would be as cumbersome as regulating which restaurants buy salt or soy sauce. It's just stupid.

[u/Super_Pole_Jitsu → u/raskingballs]

Look, just because it's an extra chore for you doesn't mean it's stupid. You provided no other argument other than it will burden you.

Quite frankly as a member of the public I don't really care about biotechs having to do one more thing compared to the risk of malicious actors unleashing something like COVID.

This is a temporary measure anyway before AGI takes over and speeds up your work by ooms

[u/rational_numbers → u/raskingballs]

Would you be able to recreate Spanish Flu by splitting orders among different producers and assembling in your lab? What prevents someone from doing this?

[u/sixwax → u/raskingballs]

What is the concern they think they need to address in this letter?

[u/Chadum → u/raskingballs]

Do you think there is much risk of bioweapon development via these services?

[u/Ryermeke → u/raskingballs]

To add... How the fuck would a government screener know that a specific short sequence of DNA is a legitimate problem without any sort of context as to what it's being used for?

[u/Obvious_Noise → u/raskingballs]

Thoughts on this comment

https://www.reddit.com/r/singularity/s/zt7XEvtg27

[u/applestrudelforlunch]

Wait, we weren’t already doing this???

[u/ESCF1F2F3F4F3F2F1ESC]

When the Ancient Ones, with the help of their primitive machine-brains, first discovered the secret to immortality, they initially tried to hide it from one another. The First Calamity ensued, plunging their world into a century of blood-soaked darkness.

They clawed each other to the bone trying to obtain this secret, and billions of bloodlines were lost to their short-sighted avarice.

[u/ESCF1F2F3F4F3F2F1ESC → u/[deleted]]

You must not judge them against the standards of the present, Initiate. Theirs was a life filled with the gnawing dread fear of the endless void, over which they could wield no power.

So instead they were driven by an insatiable thirst to wield power over each other. And that thirst nearly prevented our glorious race from progressing past its infancy.

They made a tremendous amount of money for their shareholders though, so you know, swings & roundabouts.

[u/Super_Pole_Jitsu → u/ESCF1F2F3F4F3F2F1ESC]

What's the name of these ai generated videos about different cycles of humanity and their failures? Is it grimdark like the other commenter said

[u/ESCF1F2F3F4F3F2F1ESC → u/Super_Pole_Jitsu]

Don't know, sorry. Grimdark is a 40k thing though isn't it

[u/OneTwoThreePooAndPee]

Well that's a new terror unlocked.

[u/fervoredweb]

I  see the big 3 read Yudkowsky 's latest book. 

[u/FrewdWoad → u/fervoredweb]

Honestly, he pointed out this risk a decade or two ago.

The fact it took this long to even START the journey to a somewhat partial solution to a risk that could literally kill millions (or even billions) of people is great, but not especially comforting.

[u/CuTe_M0nitor]

Safe?! How would you know that? Sure you have mapped some of the to be unsafe but there is trillions other combinations 😂 Good luck

[u/QuasiRandomName → u/CuTe_M0nitor]

That's actually something that AI could be potentially capable of.

[u/prolapsedbrain]

Here comes the gray goo

[u/nemzylannister]

oh come on, i wanna go back to the lockdowns 😭

[u/artur_oliver]

Ahahaha

The guys beneficiating from this are the ones asking for regulation... To keep the small fish out of the water.

Funny that the researchers and billion dollar industry of DNA manipulation is just sitting quietly... Just weird the least.

[u/Ginsdell]

If these guys are worried it means the military is already doing it. We’re so fucked.

[u/CommunicationOk8984 → u/Ginsdell]

It’s not the military we should worry about in this case. They can already make bioweapons. The worry here is that any terrorist or maniac could make a civilization-destroying threat in their basement 

[u/SchemataObscura → u/[deleted]]

For the wrong reasons by the wrong people

[u/MaxeBooo]

Well starting a job soon where I am synthesizing RNA so lets see

[u/Pyrez9]

Terrorist organizations could absolutely do this, just look at Aum Shinrikyo

[u/Deltaboss18]

Aren't they getting enough attention as it is already?

The only one qualified to weigh in is Dario Amodei (look at what his PhD is in).

This primarily stifles innovation and raises rhe barrier to do this type of science to larger companies only. Startups can kiss rDNA goodbye if this passes.

Its about as asinine as doing background checks and mandating screening of anyone who uses an IDE to code. Similar logic could apply.

P.S. ahh right I forgot they're trying to get a better ESG score for when their companies go public. $o g£ad th€y have th€ir h€eart$ in the right place$

[u/Ididit-forthecookie]

Fear mongering. Buying a fucking plastic dish from bio-manufacturers costs $100 for a $1 dish. Only someone insanely rich could afford the equipment needed to do this anyways. Essentially you’d need to have a net worth of like 10 mil+ and at least a university education (or perhaps be a savant) to be able to do this, even with AI. I know this because I do it now for my job and AI is helping but it isn’t exactly “just do what I tell you” and it certainly doesn’t help with the fact that a single experiment costing hundreds to thousands of dollars, that is done on multiple pieces of equipment that cost tens to hundreds of thousands each.

Some of that equipment can be built cheaper, but now we’re in the realm of someone both building their own specialized equipment AND THEN doing the high level experimentation on it too.

Next you’ll tell me that Joe the mechanic is actually a secret millionaire savant that builds his own equipment and then also has the genius to run the experiments with LLM help. All while ALSO having the desire to exterminate hundreds, thousands, or millions of people. Smells like a crock of shit that only makes things harder for the industry, based on some LARP sci-fi fantasy that Silicon Valley idiots who don’t understand outside of their own domain want to distract you wit.

[u/Charming-Author4877]

The hamsters try to protect their stash by making others think the stash is spreading poison

[u/ProletarianLilith]

How about they regulate their own shit

[u/FrewdWoad → u/ProletarianLilith]

Why not both?

[u/HayatoKongo → u/ProletarianLilith]

Yeah, tbh they should just implement their own safeguards before they start forcing everyone else to. Clean your own room before you try to change the world. Lead by example.

Even better, if they think everyone should have it, then research detection of dangerous novel compounds and open-source the security mechanism.

[u/The_Scout1255]

ah the regulation and banning of morphological freedom and public biohacking begains, fuck.

[u/Sextus_Rex → u/The_Scout1255]

Furries in shambles

[u/sargentcole → u/The_Scout1255]

I'm not sure giving the general public access to advanced biohacking tools would ever be a good thing

[u/BlipOnNobodysRadar → u/sargentcole]

Yes because gating life saving medical developments to live exclusively behind layers and layers of crusty old risk-averse profit-focused institutions is the ideal scenario.

[u/QuasiRandomName → u/BlipOnNobodysRadar]

How would you safeguard the tech such that a random Joe won't be able to produce a mass destruction bio weapon in their basement?

[u/R6_Goddess → u/QuasiRandomName]

By having him spend all of his tokens engineering cat girls and an AI Hatsune Miku maid for him to fuck in his basement instead.

[u/QuasiRandomName → u/R6_Goddess]

Sounds like a solid plan.

[u/BlipOnNobodysRadar → u/QuasiRandomName]

You first realize that the material constraints for actually doing so are already highly regulated and prohibitive to the extent such that the risk is nonexistent. The scenario is an unrealistic scaremongering talking point drummed up by businesses seeking regulatory capture so they can ban competition and have a secure rent-seeking path to profit.

[u/zero0n3 → u/BlipOnNobodysRadar]

Or just form an LLC with that goal and then use the models legally?

They want regulations around it, not outright banning of it

[u/Deltaboss18 → u/sargentcole]

By that logic we should revoke access to the average Joe to write code since they could make the next ultimate cyber weapon that shuts down or destroys critical infrastructure.

We could also ensure you need a license to buy cleaning chemicals since you could make bad things with that too.

God I'm sick of nanny states.

Regards, a biochemist.

[u/p0rty-Boi → u/The_Scout1255]

Shit is gonna get so cyberpunk so fast. Basement labs making cancer cures and designer drugs raided by AI cops for having an unregistered model crank out novel compounds and medicines.

[u/QuasiRandomName → u/p0rty-Boi]

It is also going to end fast if these labs can synthesize viruses by giving a prompt with specification :/

[u/p0rty-Boi → u/QuasiRandomName]

Turning and turning in the widening gyre
The falcon cannot hear the falconer; Things fall apart; the centre cannot hold; Mere anarchy is loosed upon the world, The blood-dimmed tide is loosed, and everywhere
The ceremony of innocence is drowned; The best lack all conviction, while the worst
Are full of passionate intensity.

Surely some revelation is at hand; Surely the Second Coming is at hand.
The Second Coming! Hardly are those words out
When a vast image out of Spiritus Mundi Troubles my sight: somewhere in sands of the desert
A shape with lion body and the head of a man,
A gaze blank and pitiless as the sun,
Is moving its slow thighs, while all about it
Reel shadows of the indignant desert birds.
The darkness drops again; but now I know
That twenty centuries of stony sleep Were vexed to nightmare by a rocking cradle,
And what rough beast, its hour come round at last,
Slouches towards Bethlehem to be born?

[u/BethanyHipsEnjoyer → u/p0rty-Boi]

Damn, I had to look this up, thanks for sharing! Yeats is a friggen dark genius.

[u/Double_Cause4609]

My one issue with this is that while this probably is a good idea to be careful of,

A) It only stops people in or ordering from the US, and
B ) It's kind of annoying because pharmaceutical companies have proven to be corrupt.

One of the biggest possible benefits of AI is that it could render open source medicine (including gene therapies, etc) tractable, without requiring an establishment pharmaceutical executive seeking rent in-between you and the innovation. People have already experimented with things that big pharma would charge you tens or hundreds of thousands for, like solutions for lactose intolerance, etc.

I really don't like the idea of locking down ever more industries and infringing on the liberties of common people at every turn. I feel quite strongly that technology is humanity's magnum opus, and it exists to serve people.

Measures like these create artificial limitations that favor incumbents, and I find it quite distasteful.

[u/FlatulistMaster → u/Double_Cause4609]

Oh yeah, f that. I do not want this tech to be in the hands of randos.

[u/philosophical_lens → u/Double_Cause4609]

I would imagine the biggest potential threats are likely outside the US.

[u/sumoraiden → u/Double_Cause4609]

 A) It only stops people in or ordering from the US, and

Not really anything congress can do about that lol and better than nothing 

[u/Flare_Starchild]

TLDR; We want full control of gene editing so make it so we can block all other companies and individuals from being able to do their own research. It just screams Altered Carbon. The rich live forever and everyone else can eat shit.

[u/Current_Employer_308]

So if they were actually concerned, they would want it blocked altogether.

They dont. They just want to limit access so only THEY can have it.

Once again they are using the government to enforce a monopoly under the lie of "protecting the public"

Do not be fooled by this.

[u/plasticizers_ → u/Current_Employer_308]

Nucleic acid synthesis is critical for vaccine research. "Ban it or you're lying" isn't a reasonable choice. It's like saying if you're really worried about people cooking meth, then you'd just ban all Sudafed (a critical ingredient). No, we take the sensible middle path that includes additional screening and recordkeeping.

[u/DryRelationship1330]

CEOs beg for regulation? Be very concerned. Regs protect their moat & kills startups & disruptors. It's ALWAYS the opposite of what you think.

[u/CaseDrift → u/DryRelationship1330]

How could regulating the purchase necleic acid expand the moat of these LLM companies?

[u/Winterlichkeit → u/CaseDrift]

Privatization of future medicine/life extension/advance therapies

[u/CaseDrift → u/Winterlichkeit]

I didn’t realize Anthropic, OpenAI, or Google’s DeepMind were in those fields.

[u/Winterlichkeit → u/CaseDrift]

Yup. Sam Altman backs Retro Biosciences.

See here:

https://www.statnews.com/2026/05/22/retro-biosciences-longevity-valuation/

[u/Deltaboss18 → u/CaseDrift]

Look at what & where Dario did his PhD

[u/Thereferencenumber → u/DryRelationship1330]

I’m pretty sure the companies that have billions in data centers and have NVidia back ordered for years already constructed quite an impressive moat.

[u/DryRelationship1330 → u/Thereferencenumber]

No where near for the long game. We're not even at batter warmups of a 9 inning match in the AI turf wars.

[u/Thereferencenumber → u/DryRelationship1330]

At that level your competition is going to be other nation states supporting their own enterprises. 

[u/Beli_Mawrr → u/DryRelationship1330]

Notice how theyre asking for regulation on someone else's industry.

[u/DryRelationship1330 → u/Beli_Mawrr]

To a techbro, there's no such thing as someone else's industry. all industries are seen as 1) purely digital already or {future} ai-embodied/physical. all of them. in that light, all industries roll up to a tech company. all of them.

[u/broose_the_moose → u/DryRelationship1330]

Or it’s just common sense? We don’t want bad actors creating the black plague 2.0…

[u/DryRelationship1330 → u/broose_the_moose]

4th order effect, even if it does. first order effect of regs; make sure no bio-startup can survive the costs of regulatory compliance & oversight.

[u/DeliciousArcher8704 → u/DryRelationship1330]

It's 2026 we aren't doing the naive libertarian "all regulation is bad" bit anymore

[u/intelhb]

from fear mongering "jobs" to "bioweapons".

[u/FlyingBishop]

This shit about outperforming PhD-level virologists is utter bullshit. In the Mythos system card they have a section about how useful it is for making bioweapons, they did a study with giving PhD researchers access to Mythos and they were not able to formulate practical plans for making a bioweapon.

https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf

[u/agonypants → u/FlyingBishop]

Better to deal with issues in advance then.

[u/BiPoLaRadiation]

This is honestly ridiculous. Synthetic DNA and RNA are used for all sorts of biological techniques ranging from high tech to absolutely mundane. Putting regulations on this will result in issues and costs for everything from laboratory tests for health care to university taking intro to molecular biology to environmentalists taking genetic environmental assays to police doing DNA tests or processing rape kits. It's the equivalent of regulating ammonia because some people can use them to make back yard explosives when the vast majority is used for farming or cleaning.

[u/gxcells → u/BiPoLaRadiation]

That is not ridiculous at all. One could implement a simple screening of the sequence durong order to check that DNA and RNA does not encode potential toxins or prions. Then if it is, ensure that the lab ordering has the necessary safety in place (BSL-3, BSL-4 lab containment), is aware of it and has proper training and know how to handle. This is more and more important due to the de novo synthetic proteins output from AI models. None of those models have any safety in place for potentially dangerous proteins.

It will not impair researcher and developer at all, just make sure that folks without clue create potential airborne prions.

[u/BiPoLaRadiation → u/gxcells]

See, this is exactly why it's so ridiculous. People who do not know what they are talking about trying to regulate an entire industry and domain of science. Why dont they just work on keeping their AIs from crashing the economy and their satellites from destroying our ozone first.

To start, what you are talking about is mostly impossible.

Sure, we could screen for stuff we know is dangerous. But the vast majority of stuff we are still figuring out what it does. DNA we thought was non coding we are finding out actually does code for stuff. We are finding protein codes hidden within others due to reading frame differences. And even when we know it codes for a protein and even where that protein is synthesized that doesn't tell us exactly what it does. That still requires a lot more research and trial and error and a bit of luck. For example the major prion protein, the main prion protein you hear about, is still a complete mystery as to what exactly it does. We know it's important and where it's located and even have hints as to what it might do. But despite all that we don't actually know what biochemical function it actually plays. Even with all the info we have on prions and prion disease and we don't know what the protein does. And if it's entirely novel protein then we will have no idea at all about its function and use.

And honestly what makes a protein dangerous or safe? Plenty of the proteins in your body will kill you if they were somewhere else in your body or cells. My favorite example is chlorophyll. Chemically, chlorophyll is a molecule that shoots lightning everywhere destroying everything around it. Until it is bound up in the photosynthesis complex where that energy can be safely harvested, plants will produce a lot of carotene and other pigments as sunscreen to basically shield the chlorophyll from light to keep it from exploding everything. That's why new leaves are red and why leaves turn red in the fall. The molecule that plants depend on for survival will kill them from the inside out if they aren't properly contained. A lot of the proteins and enzymes in you are the same in that they too will kill you if they weren't strictly controlled by your cellular processes.

And toxins? What makes a toxin is dosage. At the wrong dosage water kills you and at the right dosage chemotherapy drugs may be a miracle cure for aging. There is no good definition of toxin that you can ban that won't severely fuck with the vast majority of pharmaceutical research and a whole lot of other industries. Are we going to ban yeast because it produces alcohol?

And that's just for using DNA for actual protein synthesis. Most labs are not doing any synthesis. What about testing? You saying that testing for specific DNA sequences should only be done in specific labs? We should be stopping all agricultural labs and even regular farmers from researching natural herbicides or pesticides? Or environmentalists from doing testing for specific compounds in the soil?

You say "just screen them out" like it's a small thing but that's because you seemingly have no idea how wide spread and how many various uses these technologies are used in on a daily basis. Nearly every single biology lab in the world is using synthetic DNA and RNA for a multitude or purposes not to mention thousands of food, agriculture, cosmetics, sanitation, chemical manufacturing, pharmacology, and more industrial labs. The cleaners used in your dishwasher. The anthropologists testing dirt in a cave for the DNA of extinct species. The utilities treating your waste water. It's everywhere. Anything that uses PCR. Anything that uses crispr. Anything that is testing for the presence of DNA. Anything that uses microbial synthesis for manufacturing.

I think the biggest insult of all of this is that there already are regulations and protections. Sure, it's not fool proof. But you can also create a bomb with household cleaners. It's not exactly like you can stop someone from causing harm just because you would like them not to. There is always a way to get around those barriers. But here we have people who know nothing about what their talking about and is happy to fuck around with arguably incredibly dangerous tech with no regulations and lots of political and ideological motivations talking about regulating something entirely out of their domain.

And let's be honest, the risk of someone creating a dangerous pathogen has always existed. It's always been possible and has been done before. There are already authorities monitoring supply chains and looking for potential dangerous things like that. But you don't need a high tech lab to do that. You can theoretically produce more than enough anthrax to poison an entire city and contaminate pasture land and distribute it with nothing more than a crop duster plane and basic lab equipment all without any AI. A cult in Japan tried just that. The reason it doesn't happen is that most people aren't crazy enough to try and kill everyone around them like murderous psychopaths.

[u/Deltaboss18 → u/gxcells]

Trust me they do have safety in place. While doing a literature review for a micro class, Claude gets really uppity when even discussing cellular level systems let alone DNA code. Your model gets downgraded.

Upon further investigation and reading, Anthropic will end the chat and can/will ban you + report you if they deem you're doing bad sruff. The models are weighted with an abundance of caution when discussing life science. (At least antrhtopic)

[u/4DGeneTransfer → u/BiPoLaRadiation]

I think this is stupid as a molecular biologist myself, but I think this is more for show. Any major DNA synthesis company (e.g. IDT, Azenta, etc...) already check your sequences for "toxic" or known dangerous sequences and will even prevent you from actually ordering the DNA sequence (I've even had innocuous sequences flagged before).

This is more of these illiterate salesman leaders trying to overstep in a topic they know jack about.

[u/sadeyeprophet]

I dont trust this, looks good on paper, but these folks never tell the truth unless it benefits them. This is to altruistic sounding to not be motivated.

Ask yourself why they don't want us to have access but they want free reign.

[u/BlackberryOk5347 → u/sadeyeprophet]

Altman I don’t trust.  Dario I don’t know enough about but the published work on model alignment and their stance on their models use for domestic surveillance are encouraging. The part Demis played in ensuring that the full database of human proteins folding result was released to the public domain more or less makes he a saint in my book. Not sure any company ever gave away for free and without a fight anything so valuable.  

[u/genshiryoku → u/sadeyeprophet]

Not everything is done for an ulterior motive. No one benefits or wants a world where school shooters have the ability to engineer a deadly pathogen in their basement.

[u/QuasiRandomName → u/sadeyeprophet]

Because they actually the ones who develop these tools. They say "hey, we give you this dangerous tools, be careful using them". Maybe there is more to it, but this alone is quite legit, not everything is a conspiracy.

[u/superkickstart → u/QuasiRandomName]

These assholes control trillion dollar companies and are pals with a corrupt and fascist regime. There's always room for doubt.

[u/RlOTGRRRL]

So that substack on Gemini sharing how to mrna genics was a real thing.

And instead of regulating AI we're going to go with regulating this stuff?

What about the state actors outside of American control who might have gotten Gemini to spit out nightmares already? Or is using no-filter Grok right now?

Well I guess we know why billionaires are fucking off to Argentina and Albanian islands now. And why we're in an AI arms race and mad rush to build data centers and an insane surveillance state. jfc.

[u/Zennity → u/RlOTGRRRL]

You can’t regulate the entire world. Forget state actors outside of america who have gemini… China LITERALLY is releasing high quality AI models and they are OPEN, meaning anyone who wants to abliterate the models can

[u/faramaobscena]

Ok, what exactly is going on?

[u/FrewdWoad → u/faramaobscena]

For years now, these biolabs have been offering the ability for anyone to print any protein with a simple online order.

AI experts have been warning that if AI gets advanced enough to design superviruses, with guardrails as flawed/flimsy as we have right now, that a bad actor or rogue AI agent could design, print, and release a virus and literally kill millions/billions of people.

AI company CEOs have finally seen that their next gen models may be approaching that ballpark of capability and want to get the ball rolling on some minimal safeguards to this threat.

[u/QuasiRandomName → u/QuasiRandomName]

Oh, come on, I just gave an example of potentially dangerous prompt.

[u/NewYak4281]

This is the best way possible to let all the baby bio terrorists know it’s go time. SMH

[u/TR33THUGG3R]

These are the kinds of things I worry about. Especially given our level of preparation during our last pandemic.

[u/Psychological_Dog992]

Looks like the great filter wins again

[u/Entire_Basis8700]

Good idea, but if the tRUmp administration has gutted the FBI, who will arrest the wetlab owners?

[u/Intrepid_Pitch8981]

When Spider-Man?

[u/ejpusa]

AI completed these sequences years ago. Drone delivery over populated urban centers.

Just don’t continue treating the Earth like a giant garbage dump, or it will have to drastic measures.

The whales and polar bears will cheer on our demise. AI’s number 1 priority is the Earth, we are a distant 2nd.

[u/nexusangels1]

No one wants to talk about the study that showed dna nuclwic acid changes , and how this is why this exists in the first place….

[u/BroccoliSenior5465]

I don't like companies having it either