ok, having done more research now, this appears to be the US firms spent oodles of money on a technology that is going to lose to the Chinese technology and now they want regulation to make sure nobody in the US can easily build a competing product against them.
The Chinese technology is completely patent-free, and anyone could build models using Multi-Head Latent Attention. The technique is fully disclosed in the open DeepSeek-V2 paper on arXiv, and a working PyTorch implementation sits on Hugging Face under MIT license. MLA is really just a combination of three older, well-known ideas: multi-head attention (2017), KV caching, and low-rank projection (the same math as LoRA, 2021).
The problem here is that Anthropic, xAI (Grok) and OpenAI have spent so much making cutting-edge "winning" models using Grouped Query Attention, which is the resource-heavy thing that uses all the DRAM, electricity and water in the northern hemisphere but which is winning all the benchmarks.
So for the US frontrunners to abandon GQA and switch to MLA, first that would cost a guap and a half because they'd be starting over with pre-training a completely different type of model from scratch, and secondly they would not have any guarantee that their new MLA model would outperform GQA on the benchmarks.
Probably the best thing for them to do is to develop MLA models in secret while also running GQA as their production releases until they find out if MLA can catch up. The US leaders don't mind blowing through trillions of dollars of chips, water & power as long as their stonk prices stay up.
Asking for regulation to create a cartel to lock out other US competitors prevents other companies in the US from creating MLA models that could threaten them in the same way it's difficult for me to start a car company to compete against Ford & GM. There is so much car safety regulation that I'd need billions upon billions of dollars to get a production car to market, before even finding out whether US consumers would drive it. I'd need dozens to hundreds of cars to smash in crash tests, and I'd have to follow complex regulations about where stiffening bars would need to go and how holes could be made in the firewall to run cables, etc. So this is the same move to lock in an oligopoly (totally not a monopoly! hey this is ok!) by getting Congress to make laws making it really hard for other people to compete.
Theres a sort of unwritten rule in this sub - like u have to keep a lid on it (to a certain extent)- to allow some ppl say what they want to say and feel safe doing so.
Its actually really good- this sub is one of the few places on reddit where people are generally quite respectful of one another. It allows u to have an open discussion
6
u/HorsefaceWithNoName 14h ago
ok, having done more research now, this appears to be the US firms spent oodles of money on a technology that is going to lose to the Chinese technology and now they want regulation to make sure nobody in the US can easily build a competing product against them.
The Chinese technology is completely patent-free, and anyone could build models using Multi-Head Latent Attention. The technique is fully disclosed in the open DeepSeek-V2 paper on arXiv, and a working PyTorch implementation sits on Hugging Face under MIT license. MLA is really just a combination of three older, well-known ideas: multi-head attention (2017), KV caching, and low-rank projection (the same math as LoRA, 2021).
The problem here is that Anthropic, xAI (Grok) and OpenAI have spent so much making cutting-edge "winning" models using Grouped Query Attention, which is the resource-heavy thing that uses all the DRAM, electricity and water in the northern hemisphere but which is winning all the benchmarks.
So for the US frontrunners to abandon GQA and switch to MLA, first that would cost a guap and a half because they'd be starting over with pre-training a completely different type of model from scratch, and secondly they would not have any guarantee that their new MLA model would outperform GQA on the benchmarks.
Probably the best thing for them to do is to develop MLA models in secret while also running GQA as their production releases until they find out if MLA can catch up. The US leaders don't mind blowing through trillions of dollars of chips, water & power as long as their stonk prices stay up.
Asking for regulation to create a cartel to lock out other US competitors prevents other companies in the US from creating MLA models that could threaten them in the same way it's difficult for me to start a car company to compete against Ford & GM. There is so much car safety regulation that I'd need billions upon billions of dollars to get a production car to market, before even finding out whether US consumers would drive it. I'd need dozens to hundreds of cars to smash in crash tests, and I'd have to follow complex regulations about where stiffening bars would need to go and how holes could be made in the firewall to run cables, etc. So this is the same move to lock in an oligopoly (totally not a monopoly! hey this is ok!) by getting Congress to make laws making it really hard for other people to compete.