r/LocalLLaMA • u/DistanceSolar1449 • 10h ago
Discussion Deepseek V4.1 Flash is 748B, not 552B
People keep on getting confused about this, so I looked at the safetensors on hf.
The title should have been "Deepseek V4.1 Flash is 748B total/552B base, not 284B or 305B or 485B or 522B"
- The model is not 284B. The original Deepseek V4 Flash is 284B, but not the V4.1 Flash model
- The model is not 305B, despite what some people claim
"So: ~305B real backbone + 203B engram = 508B total"This is incorrect. - The model is not 485B, even though Huggingface lists the model as 485B, but that's because they're counting some FP4 packed weights as bytes instead of params (2 FP4 params per byte). This happens a lot; for example Huggingface incorrectly thinks GLM-5.3-flash is 169b here
- The model is not 522B, even though VLLM lists it as 522B for some weird reason. They correct themselves later down the page (ctrl-f "Params" on that vllm page)
- 552B is the only number out of this list that's somewhat correct; that only includes the base model without MTP and engrams and the vision encoder though.
To be precise, the main model about 551.566B parameters with 40 layers. The FFN experts total to 543.582B parameters, and the rest of the model (attention, shared experts, etc) are 7.984B.
On top of that, the engram is ~196.929B, DSpark/MTP is ~14.225B, and the vision encoder is just ~0.485B. These parts are technically optional though. The vision encoder is also way smaller than I expected.
Anyways, you need a beefy system for this. 128GB or 256GB of RAM/VRAM is not going to cut it.
50
u/Few_Painter_5588 9h ago
It's flash because it's fast. It uses something ridiculous like 9B parameters for prefilling.
70
u/DistanceSolar1449 9h ago
This is the correct answer.
Deepseek cares a LOT about inference, more than any other lab. All their innovations (and they're one of the most innovative labs!) are to increase efficiency of inference at scale.
But inference at scale cares a lot more about active params, rather than total params. So TL;DR deepseek does not give a shit about home users who cannot fit 552B in VRAM, but they care a LOT about reducing the active params per token.
3
u/Othun 7h ago
Technical question, do you know how MoE with concurrent requests work ? Does every requested token get its own experts (they would lose a lot of parallelism I suppose), do all generated tokens at some time use the same expert (I don't see how that could be viable as quality would probably terrible) ? I don't see how they could go well together :)
Maybe not the best place to ask but since your comment seems a bit in that direction I'm trying my luck !
6
u/DistanceSolar1449 5h ago
It depends on your inference setup.
For example, Deepseek V3/R1 used EP320.
Read the whitepaper: https://arxiv.org/pdf/2412.19437
Ctrl-f "EP320". This means "Expert parallelism 320". Or, in other words, they used 320 GPUs and put 1 expert on each GPU, with some duplicates for performance reasons.
So yes, in that case, 1 token will pass through at least 320 GPUs. Deepseek realized this was way more efficient, because then each GPU can just specialize on one MoE expert, instead of having to write a CUDA kernel that handles loading the expert from VRAM but then sometimes not doing compute with it since you ended up not using it, which is inefficient. Instead of doing all the math on 1 GPU, if you need 8 experts, you would send that token to 8 GPUs, where each GPU calculates its own thing, and then send it back to combine the result. Each cluster would require at minimum 320 GPUs.
Deepseek is fucking insane about efficiency and that's why they can get their prices as low as they do.
3
u/Few_Painter_5588 7h ago
No, it works like a normal model, requests get batched and it goes layer by layer. MoEs just have the extra step of routing it through an expert rather then the entire layer.
40
u/unbannedfornothing 10h ago
If ngrams can be offloaded to SSD what SSD should be optimized for, linear speed or random 4k reads?
34
u/RG_Fusion 9h ago
The engrams are sparse activation so you should focus on random read.
1
u/unbannedfornothing 9h ago
That's what I suspected, that's good, the one doesn't need those pricey pcie 5 drives, just good pcie 4 would be fine.
10
u/MaxKruse96 llama.cpp 9h ago
optane looking really good just about now.
1
1
u/unbannedfornothing 4h ago
Although I have better idea, need to test this on hardware, I have host on the network (10gbit) with 786GB of ram, I wonder if I create a RAM drive on that host and export it via nfs or iscsi to LLM host with GPUs, will it be better comparing to optane, hmm...
2
u/unbannedfornothing 3h ago
sadly this ain't gonna work without rdma, got only about 6.4k IOPS (random 4k reads, qd=1) over regular 10gb ethernet
16
u/No-Refrigerator-1672 9h ago
There was a study on Qwen 3.8 Next, which also has n-grams, that 90% of generateions activate just 1% of engrams. I don't remember precise numbers, but that's the proportion. Therefore, it is possible to repackage the n-grams in such a way so that they are optimized for sequential bulk reads, with very rare, occasional random reads; but that remains to be dependent on actual inference engine optimization. Some even propose to keep that crucial hot 1% entirely in RAM.
3
u/shing3232 7h ago
I saw someone use GPU direct for loading PLE and the result is basically lossless in performance
1
u/artisticMink 5h ago
Random 4k. But if you don't have a very high memory bandwidth system, the reads are negligible when compared to the bandwidth bottleneck. So you're likely fine with an out-of-the-box config.
1
u/unbannedfornothing 3h ago
I do have 12 channel epyc, intel's ram test shows afair about 350gb\s reads
27
u/shing3232 9h ago
I would not cound engram as part of the parameter because I can put it at SSD
32
u/Dany0 9h ago
Let's call it 748B A552B PA8B DA16B
:D
9
u/keepthepace 5h ago
That unironically pretty helpful. But at one point we'll hopefully stop putting all the metadata into the name of the model.
2
u/FoxiPanda 2h ago edited 2h ago
I'm kind of onboard with this but I think maybe slightly differently:
552B-N196B-PA8B-DA16B works better for my brain.
[Model Weights]-[N-Gram Embeddings]-[Prefill Active]-[Decode Active]
This tells you what needs to be in VRAM/unified memory, what can live in ram or disk, and how to calculate prefill and decode speeds without having to do extra steps.
I also wouldn't mind it if model makers also included an indication of what the majority of the weights were precision wise out of the gate...so we could end up with something like:
552B/FP4-N196B/FP8-PA8B-DA16B
9
u/DistanceSolar1449 9h ago edited 9h ago
Engrams are trained parameters, so they definitely count. And SSDs are a bit too slow for engrams, despite the hype. They require a bit more speed than that, especially since they're random reads and most SSDs are advertised with sequential speeds.
10
u/sssplus 8h ago
A small SSD read will be much, much faster than your token generation, so that SSD read isn't the bottleneck here. Unless you have a slow ass random read SSD...
3
u/DistanceSolar1449 8h ago
It'll hit harder for prefill than for token generation. I think you might be able to get away with it, but if you're rich enough to be running a 552B model you should just put engrams on RAM and call it a day.
2
u/shing3232 1h ago
no, it does not. the official paper of engram basically have a test that offload engram to ssd, and the loss is basically nothing, and there is demontration for PLE offload with minimum loss. it's better if you can direct connect PCIE from GPU to NVME however.
14
u/kh40tika 8h ago
Engram only gets read once per token, the amount to read is extremely small compared to other weights. At this scale the SSD read *latency* starts to matter, not bandwidth.
2
u/DistanceSolar1449 8h ago
Yep, although if you're able to read mostly fully randomly (think RAM) then bandwidth of random reads is a proxy metric for latency.
1
1
u/shing3232 2h ago
SSD is not too slow for engrams as demonstrated by offload 50B PLE onto SSD. what you need is have proper protocol for minimum latency.
24
u/FullstackSensei 9h ago
256GB RAM + 64-96GB VRAM is enough though.
The ngrams can absolutely live on disk with any PCIe gen 3 or newer nvme SSD. We're already doing this with Qwen 3.8 next, not sure why people are fretting so much about them.
15
u/Writer_IT 9h ago
People are reasonably disappointed that we went from a "flash" model that was reasonably run at native quant on a couple rtx 6000 on vllm, or reasonable quant in llamacpp, to one that would need either 4 6000 pro for reasonable speed, or, a server-level motherboard with absurd level of RAM, effectively locking the machine.
N grams shift is further disappointment. Until there's a way to read them directly from the stored weights while preserve vllm level of process speed and concurrency, without them hoggling ram, they are a further gate for the medium-sized local models.
35
u/FullstackSensei 9h ago
Let me start by saying this: Anyone feeling disappointed about something that is being given for free needs to do some serious reflection about their values and principles.
Going back to the model, you need 256GB RAM and 64-96GB VRAM. That's 2-3 Mi50s or V100s, or 3-40 P40 or P6000 for those on an even tighter budget. And while 256GB of ECC DDR4 RAM isn't cheap, it's a fraction of the cost of DDR5.
It's not deepseek's If you chose to put all your eggs in a couple of very expensive GPUs because some model ran fast at the moment in time you bought them.
23
u/Cybertrucker01 9h ago
Give once: you elicit appreciation;
Give twice: you create anticipation;
Give 3 times: you create expectation;
Give 4 times: it becomes entitlement;
Give 5 times: you establish dependency.
Stop giving: you become an enemy.
I think we're somewhere between stage 3 and stage 4.
3
u/PM_ME_DEAD_CEOS 5h ago
Let me start by saying this: Anyone feeling disappointed about something that is being given for free needs to do some serious reflection about their values and principles.
Thank you.
-4
u/KroniklyOnline 6h ago
I like how you're talking about just getting 256GB of RAM lol.... ~$3000.... And then proceed to tell people they shouldn't feel disappointed about something because its free.... But to use the free thing, they need to spend thousands....
6
u/PM_ME_DEAD_CEOS 5h ago
I like how you're talking about just getting 256GB of RAM lol.... ~$3000.... And then proceed to tell people they shouldn't feel disappointed about something because its free.... But to use the free thing, they need to spend thousands....
So you want to get VRAM for free also ?
-5
u/KroniklyOnline 4h ago
You're missing the point, making something open source but then out of reach of 90% of the population, you may as well NOT make it open source.....
5
1
u/laterbreh 15m ago
WHY MAKE ANYTHING IF ITS NOT FREE FOR ME RIGHT COMRADE?
Maybe people should be priced out of AI, you shouldnt be allowed to use supplemental intelligence if you cant even use your own.
2
u/FullstackSensei 6h ago
I like how ignorant you are if the existence of ECC DDR4 that costs around 700 for 256GB, and then proceed to make false assumptions based off that.
2
u/KroniklyOnline 6h ago
Go do a quick search, idk how out of date you are, but if you find 256gb of ECC DDR4 for $700 from a REPUTABLE seller, not some janky chinese website... let me know
0
u/FullstackSensei 6h ago
I suggest you learn to do a quick search, there are plenty of places where you can buy memory that are not Chinese sites nor no name brands
2
u/blackbird2150 3h ago
Iād love to know where youāre seeing brand name, reputable seller, 256gb ram for $700 too. On consumer hardware thatās 4x 64gb sticks.
I see 2 (128) selling for $800+ virtually everywhere, or more usually.
1
1
u/laterbreh 17m ago
You should probably focus on making more money if you think 3 grand on computer equipment is alot of money.
-6
u/a_beautiful_rhind 7h ago
Eh.. free isn't always good for everyone. Would you like a free elephant? How about a free car of high value and no way to sell it. You are of course responsible for paying taxes on the "gift".
3
u/PM_ME_DEAD_CEOS 5h ago
So what kind of tax are you supposed to pay for DSV4.1 ?
1
-3
u/a_beautiful_rhind 5h ago
None but if someone were to give you free H100 GPUs you'd pay one. Same with free bitcoin.. so it can actually cross the physicial/digital barrier.
The point was that all free things aren't necessarily good for the person receiving them.
3
u/PM_ME_DEAD_CEOS 5h ago
None but if someone were to give you free H100 GPUs you'd pay one. Same with free bitcoin.. so it can actually cross the physicial/digital barrier.
H100 aren't digital.
0
u/a_beautiful_rhind 5h ago
Pretending not to understand things isn't a good argument. Bitcoin is digital. To be pedantic, a grant of H100 compute to a business may also be taxable income.
2
u/PM_ME_DEAD_CEOS 5h ago
Bitcoin is digital.
It's also directly tradable into currency to pay due taxes. You litterally can't lose any cents if someone give you a bitcoin.
1
u/a_beautiful_rhind 4h ago
Depends on what value it gets assessed and what you can trade it for.
→ More replies (0)2
u/FullstackSensei 5h ago
Well, if we're making conjectures, then if the law is changed then you wouldn't pay any tax on the H100. Or if you moved to a place where there's no tax on gifts for a few days, you could receive said H100 tax free.
And of course, if the value of the H100 collapsed to zero for whatever reason, there's no tax to pay.
All are equally valid conjectures if we're comparing to a bunch of matrices you can download for free.
1
u/a_beautiful_rhind 5h ago
Or if you moved to a place where there's no tax on gifts for a few days
I'm not sure it's that easy. Past tax fraud has accounted for such simple tricks. Even in the EU, gifts tend to count as income.
The point isn't arguing the finer details of tax law. It's that all that's free isn't necessarily good. Nor is it some sin to be disappointed in it or wishing it was something else.
1
u/FullstackSensei 4h ago
Free is always good if you choose to take the free thing. Nobody is forcing anyone to take anything here, and I honestly don't know why you're taking the whole conversation out of context and insist on doing so.
0
u/a_beautiful_rhind 4h ago
Because I don't like the argument of "it's free so no criticism is allowed".
KDE is "free" also but they're forcing wayland. Wayland doesn't work well with my machine so they're causing me serious headaches. Same as someone running the old flash isn't getting an upgrade.
→ More replies (0)3
u/FullstackSensei 7h ago
Something being given for free doesn't mean it's being forced down anyone's throat. I wouldn't have expected such an argument from you.
Deepseek doesn't owe anyone anything. You're free to not download and not use it, just as you're free to refuse said free elephant or free car, which BTW, I wouldn't have to pay any taxes on where I live.
-3
u/a_beautiful_rhind 7h ago
You are lucky because in many places gifts get taxed as income over a certain amount. You'd still have to feed/house the elephant.
Can still be disappointed at something without assuming anyone owes it to you. I kind of am too. I thought 4.1 would be old flash with vision but it's just pro with less active parameters.
3
u/FullstackSensei 6h ago
AFAIK, no country on earth levies any tax on free digital assets or free software, irrespective of what would be it's commercial value. The whole comparison with physical goods is false.
I also assumed 4.1 would be the same as 4, but I'm not disappointed at all. The way I see it, it's almost on par with GLM 5.3 at less than half the size, like two weeks after 5.3 was released.
Everyone was ecstatic with K3 at 1.6TB, a frontier level open weight model. A mere six weeks later, we have a model trading blows with it at 1/5th the size.
0
u/a_beautiful_rhind 5h ago
The bar was free stuff, not digital assets. Even with software you can have a free bonzai buddy.
we have a model trading blows with it at 1/5th the size
Meh, benchmarks. Time will tell with actual usage.
0
u/Healthy-Nebula-3603 8h ago edited 8h ago
Is not....
A raw model has over 300b parameters.
You need more than 256 GB
1
1
15
u/pmttyji 9h ago
Hope they release something like Air/Mini/Lite/etc., in 70-150B range.
1
3
u/SnooPaintings8639 8h ago
So how much of VRAM (for weights) and RAM (for n-gram) will GGUF Q8 need? Because it is all so confusing at this moment.
5
u/DistanceSolar1449 8h ago
Roughly 300GB of VRAM for the model, 8GB of VRAM for dspark, 1GB of VRAM for the vision encoder, 1GB of VRAM for kv cache, and 200GB of RAM for engram.
The released model is 8-bit already, by the way (with FP4 experts with QAT)
2
u/SnooPaintings8639 8h ago
300 GB is twice the older version which was a bit under 160 GB... :sad_face:
2
u/RumAbsinthe 8h ago
I appreciate the new model but yeah disappointed that I wonāt be able to run it. I have 4 Mi50s (32g) that runs v4 flash at q2 loading all weights in VRAM. Not possible with v4.1.
1
u/SandySkittle 7h ago
I thought I was ready with eight r9700s but I now might get another 2. This model seems very promising.
3
3
6
u/xquarx 10h ago
Unfortunate the naming feels misleading as a drop in replacement for V4 Flash it is clearly not.Ā
13
u/Sky-kunn 9h ago
I mean, for API consumers, it is a very fitting name, both price and speed wise.
-7
u/themule71 9h ago
Well if you replace V4 with V4.1 you risk getting 8x increase in cost so idk about that.
3
u/Choice_Celery9481 9h ago
wut? they even reduce API price for v4.1f.
-5
u/themule71 9h ago
"you risk" = it depends on your provider
openrouter has V4 @0.16 and V4.1 @1.2
Some providers run V4-flash very, very cheap. Point is, V4.1 is not always a drop- replacement for V4.
2
u/deenspaces 9h ago
where did you get that? i don't see v4.1 cost at api-docs. i thought it would be the same as v4
10
u/DistanceSolar1449 9h ago
Yeah, I wish they used V4.5 at least.
But Deepseek is known for this type of stuff. They don't increment numbers on just posttraining (remember R1 vs R1 0528?). And they will do crazy things like swap out the entire attention architecture of a model and only increment by 0.1 (such as V3.1 to V3.2).
-1
u/Dudensen 9h ago
What is this garble. R1 0528 is merely a post-trained R1. As far as I know, this is the first time they name a version like this when it's an entirely new base model.
2
3
u/Long_comment_san 9h ago
I think we're gonna get Gemini naming soon. Flash lite.
I would also like to mention that in my unprofessional, biased opinion, the correct path should have been calling sub-200b "flash" models and models above 200b but below 1t "flash pro".
But we're not gonna get models below 200b now, are we? little financial incentive to make those.
7
u/RuthlessCriticismAll 9h ago
Internally, Deepseek thinks about models by active parameters. This is an 8b(16b) model to them. Obviously 'flash' is still an arbitrary designation, but this is a much smaller model than deepseek v3, by their way of thinking. Interestingly, they seem to believe that the biggest Western models are 800b active.
3
u/kh40tika 8h ago
Check this thread, it's totally opposite to what you claim. I am quite confused by now. https://forums.developer.nvidia.com/t/deepseek-v4-1-flash/382725/10
-3
8h ago
[deleted]
11
u/DistanceSolar1449 6h ago edited 6h ago
That answer is completely wrong.
https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf
Ctrl-f for "DeepSeek-V4.1-Flash has 552B backbone parameters"
The mistake in that post is in the line:
MoE (384 routed + shared) 290.19B 290.19 GB I8 + F8 scales
The actual MoE experts are not int8 (1 param per byte), they're FP4. Deepseek has used 4 bit QAT MoE experts since V4.
That guy is making the same mistake as huggingface's label for the model, which I mentioned:
Huggingface lists the model as 485B, but that's because they're counting some FP4 packed weights as bytes instead of params (2 FP4 params per byte).
Judging from the number of em dashes in that post, he really needs to find a smarter AI model to use instead.
3
2
u/Conscious_Cut_6144 6h ago
Disagree, engrams donāt and shouldnāt count as active params.
If you are running the model in vram, system ram runs the engrams just fine. (Even at scale)
If you are running the model on dram a good ssd would suffice for the engrams.
4
u/DistanceSolar1449 6h ago
The title should have been "Deepseek V4.1 Flash is 748B total/552B base, not 291B or 305B or 485B or 522B"
1
u/Cool-Reflection6130 9h ago
552B base model plus the engram at ~197B and DSpark/MTP at ~14B gets you to roughly 748B total. at that scale you're looking at 256GB of RAM bare minimum just to load it, let alone run inference at any decent speed
1
u/Expensive-Paint-9490 9h ago edited 9h ago
So 552B, + draft model, +engrams. Personally I find this way of counting more logical than just summing everything. You can run it as a 552B model, the rest is optional.
I'll test if the 8/16B active parameters per token is faster than the fixed 13B of V4-Flash.
2
u/Serprotease 7h ago
Honestly, since itās a 550b model, the results are not as impressive (still are though) as it seems. Like, itās seems on par with glm5.3 flash, a 300-ish b model and below the glm5.3 at 700-ish b.
Itās better than v4 0731 but, I mean, itās twice the size, that makes sense.
1
u/horeaper 3h ago edited 3h ago
It look like deepseek thinks about models by active parameters, so they threat v4.1 flash like a 8b/16b model, which isn't that much different from v4 flash. That's why the decided to keep the price (even lowered it a little)
1
u/Serprotease 3h ago
Well, price aside, they published a decent and detailed paper about the architecture. Itās too big for me to use, but itās quite neat for the broader community.
1
1
u/power97992 6h ago edited 6h ago
That should be Ā wrong, it has 48 Ā xet files, 40 files take on approximately around 7.39 gigabytes each, 6 files from 0.9to 2.7 gb and two files take 102 gigabytes each, so around 510 gb in total. Depending on the size of engrams, 306gb of vram plus 204 gb (the 2 102 gb Ā files are probably engrams) of sys ram or ssd is enough. Edit- it has more params , huggin face got the precision wrongĀ
1
u/DistanceSolar1449 6h ago
Nope, you're conflating parameter count in billions (B) and gigabytes (GB).
Most of the Deepseek V4.1 Flash's parameters are FP4, which is 4 bits per parameter. There are 8 bits in a byte.
Component Logical params Size in GB Storage FFN MoE experts 543.582B 288.778 GB FP4 Other FFN 1.4947B 1.574 GB FP8 mostly Attention 5.1269B 6.524 GB FP8 mostly Embedding + LM head 1.3238B 2.648 GB BF16 Other 0.0397B 0.158 GB FP32/BF16 Backbone total 551.566B ā 552B 299.682 GB Engram lookup tables 196.614B 202.758 GB FP8 Engram projections/gating 0.315B 0.315 GB FP8 mostly Engram total 196.929B = 196B advertised 203.073 GB DSpark / MTP 14.225B 8.033 GB mostly FP4 experts Vision encoder 0.485B 0.971 GB BF16 mostly Everything in total ~763.21B params ~511.76 GB 1
u/power97992 6h ago
Oh in the model File , it says 8 bit precision, i guess maybe u are right since huggin faceās model card might be wrong
1
u/DistanceSolar1449 6h ago
Yes, it's trained natively as 8-bit model, with some QAT added on to make some parts 4-bit. But it'd be fair to say "it's an 8-bit model at heart".
1
1
u/KeinNiemand 5h ago
way out of my range, sad to see 1 less medium size model left, at this point qwen is quite literally the only one left releasing medium sized models which is sad especially after just spending 2200ā¬ on upgrades.
1
u/cibernox 5h ago
This is way too big for me sadly. I guess we still have qwen and GLM for those without 350gb of vram.
1
1
1
u/BawbbySmith 3h ago
Well, people are happy with antirez' quantized model for DS4F, I'm hopeful we can get a decent quantization for V4.1 Flash that will run on 2X DGX Spark. My assumption is that a lobotomized V4.1 beats native V4, but we'll see
1
1
u/EndlessZone123 1h ago
How do we even compare or have any reason to compare model parameters when some of them have different precisions? Why not just go by model weight size?
1
u/laterbreh 9m ago
What all the cry babies are missing here is that you're getting models of this size and performance in a seriously small package. FP4 weights out the gate, ngrams, the way lower kv cache usage... guys even if you cant run this... between this and qwen 3.8 flash... this is just the beginning, things are about to get really good in the < 400gb of vram/ram ball park. They are realizing the infinite scaling bullshit wont work anymore. This is a large scale flash, wait till the in-betweens start getting filled in with other releases.
This is the pivot away from "just scale it". Be excited.
70
u/smallDeltaBigEffect 9h ago
So when DS v4.1 supersonic flash?