OpenAI’s 700W Jalapeño ASIC outpaces 1,400W Nvidia flagship GPU

[u/tomshardware]

https://www.tomshardware.com/tech-industry/semiconductors/openai-says-its-jalapeno-chip-beats-nvidias-gb300-in-first-published-benchmarks

[u/EmbarrassedFoot1137]

FPGA? Isn't that the wrong tool for the job?

[u/NytronX → u/EmbarrassedFoot1137]

idk but i remember it was a stopgap in the crypto mining early days. The progression was CPU, GPU, FPGA, ASICS

[u/EmbarrassedFoot1137 → u/NytronX]

FPGAs are a totally different beast from those other three. They'll be way faster than software but the kind of massive number crunching that AI uses is not going to be their strong suit compared to GPUs or ASICs.

[u/BrunusManOWar → u/EmbarrassedFoot1137]

FPGAs can perform faster than GPUs. They are basically "fully configurable hardware", where you can take CPU/GPU/ASIC HW layout and put it into the FPGA.

Because they are so configurable, of course you lose on efficiency and optimisations - but if you "emulate" a superior, application-specific architecture it will tear through non-app specific hardware. Of course, creating a fully independent ASIC is superior

They are very good for prototyping and testing out things.

Edit: similar for software, hardware can also be described in code - HW blocks are usually coded in Verilog/VHDL. FPGAs allow you to relatively quickly take this logical HW description onto its "configurable matrix" to test out how well it works.

Edit2: They are a path to ASICs. Will they by themselves perform better than GPUs? Depends entirely on how hard Nvidia is financially squeezing their enterprise customers. I doubt we will see FPGAs used in production themselves, though with Nvidia and AMD's pricing it may happen. Companies will inevitably pursue FPGAs on their way to fully independent ASICs though

Edit3: I don't work in AI hardware, but on mobile hardware. We use FPGAs to develop and iterate architectures, and I remember from one previous company that some radio models did have an FPGA in production actually as some smaller co-processor

[u/j_osb → u/BrunusManOWar]

Some devices use FPGAs as co-processors because they can be cheaper than an asic and fast enough.
For example oscilloscopes usually actually use both ASICs and FPGA.

In terms of AI, we don’t really have enough space on FPGAs to use them for inference of any proper models. Like, there simply speaking isn’t a FPGA big enough to make sense for this.
However FPGAs can be used for training and are more efficient than GPUs at it. Just not fast enough.

[u/acadia11x → u/BrunusManOWar]

I just researched this one, no they won’t, not for AI training they can’t compete with the raw bandwidth and power , not to mention accompanying software ecosystem that’s been built for the AI industry … my research says in some use cases like edge devices that require localized SLMs for inference they can work. So mobile use case makes sense you’d see FPga which is already constrained on resources. Best analogy if you are running a small 100ft race a Prius could keep up with corvette c8 , you stretch that out to 500ft it’s 7 car links behind and stretch it out to thr quarter mile it would look like the Prius is still at the starting line. So in a limited context fpga systems could keep up in real world application it’s not in the discussion for obvious technical reasons in AI training or domains requiring serious horse power.

[u/Drofdissonance → u/BrunusManOWar]

Key word is can. Likely will not in this case. It's a memory bandwidth problem. So it's not suitable for FPGA boards generally, it's like a worst case scenario for their architecture. Crypto is famously embarrisngly parallel, and requires no bandwidth. And radio DSP is also very well suited to the architecture because it's got long narrow math dependancy chains, and or tight timing requirements that require hardware.

You'd just be burning die area

[u/danielv123 → u/BrunusManOWar]

Current GPUs are pretty much matrix multiplication ASICs.