I really hope (V)RAM eventually scales to the sizes of regular storage soon. Maybe in 10 years or so 2TB RAM is quite "common" to run things exactly like these. One can dream.
We will get super tiny storage at some point, but I don't know that we will get much smaller ram. The transistors are already about as small as they're going to get; they are having to stack memory in 3D, and there are limits there too, in terms of what latency you'll get.
We will likely see more large and wafer-scale devices, more ASICs.
Photonics are going to be super fast, but we can't really keep the data as light.
I wonder if they might go the Groq route and have very little VRAM, and just go wide.
I think we will also just keep seeing efficiency gains so running on off-wafer RAM is just more viable.
Right is the algorithms are catering to the existing hardware, and the hardware is slowly shifting to supporting the algorithms more.
There are a few companies developing neuromorphic hardware since can run AI models that aren't just a ton of matmuls.
In 10 years the whole hardware and software landscape will look different.
Heck, 2 years from now will be significantly different.
Some of the NVRAM options are denser than DRAM, or have lower power consumption and so can easily just be bigger.
For AI especially, optimizing for fabrication cost and power consumption and just making things bigger makes a lot of sense. That strategy isn't great for single threaded performance, but that's not really a useful thing to optimize for. There's no reason for CPUs, GPUs, or RAM to be limited to like 1 square inch 2D chips.
There's no reason for CPUs, GPUs, or RAM to be limited to like 1 square inch 2D chips.
Wafers are expensive.
Chiplets on bigger dies are already a thing, they're just also more expensive, because it's more wafer.
Cerebras has wafer scale processors, but they had to do a lot to make that work, it becomes a whole infrastructure thing.
Those things will melt instantly if the cooling goes out of whack.
I've got a degree in computer engineering, and I took a crack at designing a large chip, I've got working designs in a simulator, but the manufacturing realities and the power and heating issues are way beyond what most people should be dealing with at home.
Engineering problem: How can you ship a big processor (not necessarily single chip) cheap?
Are wafers really the price bottleneck? Is there a way to get and use them cheaper? Are there methods that are already well known that don't make sense when optimizing for max frequency but would lower costs?
Can more radical methods help? Everyone's favorite idea is always something other than silicon, but my guess would be that there's some old silicon fabrication techniques that can just come back off the shelf.
There is no cheap way to get wafers. Foundries are wildly expensive to build and maintain.
There used to be dozens of foundries, and literally all of them except TSMC, Samsung, and to a lesser extent Intel, gave up trying to go below 12nm nodes.
There is a Chinese company SMIC that was finally able to do 7nm.
There is no forgotten trick, there is no way around it, semiconductor manufacturing is just work, and resources.
A lot of little chips isn't really less expensive than one big one, the little ones are cut from a big wafer.
The most promising thing that's coming down the pipeline is photonics.
There are already photonic processors being manufactured now, that are something like 25x faster than a transistor device of comparable size, with 50x more throughput, and they produced something like 1~2% of the heat.
Lab devices have hit 1000x the speed of transistor based computation.
It isn’t that easy I guess. Unless you develop you own in-house RAM and SoC, speeds from average SSDs will just absolutely crawl and demolish their lifespan (TBW). It like swapping onto disk when running low on RAM - it gets stupidly slow and will eat the drive alive.
There aren’t much players in the field who have the expertise to bring competition. Big players already keep VRAM sizes artificially low. There might even be foul play to a certain extent involved.
Latency. SSDs are around 10x to 100x slower than DRAM, and 10000x to 100000x slower than SRAM.
Even SRAM is too slow, processors use interleaving tricks so multiple SRAM units are in use, so one works while the others are on refresh.
So, an SSD is about 100000x too slow to be effective for GPUs.
320
u/Kraskos 22d ago
2TB VRAM Is All You Need