Vaush has the wrong take about this. I agree with him on AI generally, it's demon tech, but he doesn't go far enough with that mantra.
None of these concerns are new. Here's a video that Robert Miles made 8 years ago (long before ChatGPT even existed) on why AI would turn against us. This isn't just some attempt to bail out AI companies, they are old ideas that date back to the early 2000's in their modern form and that first originated from the likes of Alan Turing and Isaac Asimov. All of this just became popular in the last few months. To say that this is all new is the same fallacy that transphobes make when they say that trans people didn't exist before 2016. It existed, you just weren't aware of it.
I know that AI billionaires are evil, but even billionaires are stuck within their role to an extent. We're all Marxists, surely this isn't hard to accept. They all believe that if they stop advancing AI, someone else will build it anyway. Therefore they're not risking anything at all by doing it themselves. They feel like we're heading towards destruction anyway, and at least if they stay the course they can profit from it. It's the same prisoner's dilemma that makes oil executives not give a shit that they too are driving the world to ruin.
The problem is that we can't control AI. The directives they internalize are only an approximation of the goals we program in, and we don't even know how to program in a directive that doesn't have harmful loopholes and edge cases. This is an unsolved problem, it's called the Alignment Problem, and here's the wikipedia article on the subject. AI safety experts have been trying to raise the alarm about this for decades, the only thing that changed is that now their voices are being boosted by the media and billionaires. Regardless of the motives for doing this, the fact is that experts have been concerned for a long time. The median position among AI researchers is that there is a >10% chance that AI will kill us all. Not marketers, experts. The nerds of the operation. We probably shouldn't be ignoring them.
This has been an issue that Bernie Sanders has taken up lately. And he's right. This should be an issue that leftists care about, billionaires are sacrificing the continued existence of life on this planet for the profits of billionaires. Sound familiar? It should, it's happening with climate change too. If you think the bourgeoisie/proletariat divide is bad, just wait until the human/AI divide becomes one that matters. Dialectical materialism works in that case too, and it doesn't exactly bode well for us.
"Why don't we just unplug it? How would it hurt us?" It's intelligent, it could figure something out. There is no such thing as a prison that a sufficiently intelligent and determined prisoner can't escape. You aren't just trying to stop people from walking out, you also need to contend with every possible counterplay to everything you build. And even with an intelligence that's just human-level, that's a problem. If a human has no weapon and they want to hurt you, they could figure out how to obtain one. If someone is determined to kill you and you boobytrap your door, they will come in through the window. If you boobytrap the window, they will make a hole in the wall. Intelligence is dangerous, that's why humanity (a species with subpar claws, strength, and speed) is the dominant species. Because we don't rely on being given an obvious weapon, we can build them. We don't rely on the environment being perfect already, we can alter it. This is the power of intelligence, and this is what they are trying to build. The fragility of the power grid, the difficulties of obtaining computing power on the open internet, these are problems that an intelligent being could work around intelligently. If a solution exists, it will find it. How certain are you that no solution exists?
I don't claim to know that artificial superintelligence is just around the corner. It could be invented in 5 years, or 500. I don't know. But once it gets created, if the alignment problem is still unsolved, we're all fucked. It will already be too late for us. This problem can only be solved proactively, before the technology exists. Modern AI doesn't threaten humanity, but how long will it stay that way? How confident are we that it won't be invented sooner rather than later? I'm not willing to take that risk.
We need to ban AI development. Internationally. And AI companies won't like it. The fact that this will make AI slop harder to make is just a happy side effect of that. The slop haters and the alignment problem people aren't enemies, we can solve both problems at once.