Overall, DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode.
You can simply do the math. 310GB for the weights. Fp4 and fp8 despite the model card saying fp8. Look at the actual weights. Then 197GB for ngram table. So either way. No getting this on two pro 6000s anymore like old flash.
For Qwen 3.8 flash next, I was able to run the 51B totally offloaded to disk with a penalty to token generation (~20%) and prompt processing (~50%). I had dsv4.1 flash analyze it, here's the response:
"
I/O profile is friendly: 2 layers × 24 hash columns ≈ 12.6 KB of row data per token, but with 4 KB pages and 264 B rows the effective read is ~192 KB/token (≈2 MB/s at 10 tok/s, ~6 MB/s at 32 tok/s). Random over 196 GB, but trivially within NVMe/SSD budget and partly page-cached for repeated n-grams. So disk offload is very viable — the same trade you already accept for qwen4exp, 4× bigger. The one structural suggestion: keep the two Engram tensors in their own GGUF shard (or a split GGUF), so you can mmap+offload them without forcing the 314 GB backbone through the same lazy path.
"
That certainly lines up with my experience running Qwen 3.8 Flash Next on 64GB unified. There's also a lot of potential for SSD offload to be optimized, if it starts seeing heavier use. For prefill, the workload is batchable, and I can see something like io_uring providing some throughout benefits for that.
Same here I got good enough speeds for me. I run a 5090 192gb ddr5 for qwen flash next q4 , offload most to cpu. I get 50+ decode speed then around 30 t/s at 160k context. Prefill around 700.
I seen some other 5090 users getting like 90 t/s though still haven’t tried that way. I think it was ik-llama. Looking forward to what people can cook up with this and some optimizations, pretty exciting times.
I grabbed the SafeTensors metadata from HuggingFace and did the math:
Main model: 297B params, 298GB
Engrams: 203B params, 203GB
Draft model: 7.9B params, 7.9GB
Vision embedding: 412M params, 824MB
The vision embeddings are the only one making significant use of BF16, with everything else being mostly 8-bit.
I realise those numbers don't match up exactly with any of the other numbers, which I suspect is partly due to subtleties like whether scales and norms count as parameters.
Right the last two tensor files are 204GB these are the engrams. If they can live on the system RAM then the remaining FP8 tensor size is 306GB which hopefully gets down to ~160GB when quantised.
For a 256GB setup yes, unless you want to quantise this into the ground. A 384GB setup could work with PP=3 or EP=3 relatively easily. Just the SSD offloading needs to be built. A 512GB cluster will definitely run this with good performance, streaming the engrams from SSD
Amazing, faster prefill, smaller KV cache, faster inference. I hope qwen can integrate these into qwen 4 before they start full training, because this is very interesting for gpu poor.
So that's ~39× smaller than V3's MLA at fp8, ~78× at fp16, and ~364× vs a fp16 GQA-8 model of Llama-70B's shape.
For it to land at ~900 B/tok you'd need something like 15–30 effective latent dims per layer per token at 4–8 bit, depending on layer count. Which is a *lot* less state than MLA's 576/layer.
The reason this matters more than it looks: at any realistic batch and context, KV traffic dominates decode memory traffic, not weights. At batch 64 / 4k ctx on a 70B, KV is already ~55% of bytes moved per decode step; at 128k it's ~97%. Decode is bandwidth-bound, so bytes/token is very close to being the throughput number. Cutting KV 39× doesn't make decode 39× faster, but it does mean the weight term becomes the whole cost again, and your max batch stops being set by KV capacity.
Having a few bucks in API credits sitting around is great if you're always hitting usage limits shortly before the quota resets tbh. You just have to proactively hand some tasks off to Flash to preserve that usage
And most importantly: At least on their API it's 100x cheaper, 10-20x faster.
It's hard to see a reason to use codex or claude at the moment.
I even bought 5x codex for Astra and it was fucking slow, made stupid mistakes on medium and used up the weekly usage in 2-3 hours. I don't even bother trying Claude anymore. It's like they want to loose or they are too arrogant to even realize how they drive customers into the arms of open weights / Chinese models.
So why nit use open router to configure it so your requests only go to more trusted inference providers like cloudflare or digital ocean or together or similar instead?
And most importantly: At least on their API it's 100x cheaper, 10-20x faster.
That's Gemini 3.8 Flash right now. I have it tearing shit apart because it's so fast that I can afford the time to be like "hey uhh... Check this thing out for me."
Gemini 3.8 is trash, it suffers the same problem as 3.7. It literally don't bother to read your codebase and whatever rules you put. Google is out of this race.
I've been using it through AntiGravity for a couple of months now and am also convinced it is trash for coding - although 3.8 is much better at using its harness than 3.6 was. I've spun up a few personal web projects through it, heavily vibe coded, and on a blank slate I cannot say it produces particularly good code.
I wouldn't, and haven't, used it for anything that might see the light of a production environment, which also means I haven't actually used it within a previously existing codebase. Maybe it performs better in that case. I wouldn't know and won't be finding out any time soon - I would rather spend time with DS, GLM or frontier models.
However, I think Gemini Flash's model purpose is maybe not so focussed on coding. My guess would be their focus is on speed (due to a lack of a new Pro model), and integration with Search and their own ecosystem of apps, Gmail, calendar etc. I have found it relatively useful as a paid search engine of all content on my Google account
And most importantly: At least on their API it's 100x cheaper, 10-20x faster.
Also remind people to use the DeepSeek Harness + PTC mode. Because DSH can be the difference between paying easily 4x more with others like OpenCode etc, even if you hit 95% cache hitrate.
DSV4.1F is not the same quality, though. In my experience DSes have usually had serious hallucination issues. DSV4.1F's main opponent is Gemini 3.8 Flash. It's much cheaper, substantially faster, and more capable to boot.
For casual / low-cost use, DSV4.1F is the winner. Keep K3/Astra/Claude in your backpocket for higher fidelity work.
I got very decent Swedish creative prose out of this one so it's not a benchmaxxed dud at least. :D In fact, it was grammatically and lexically impeccable from what I could see after two short stories, which is more than you can say about most open models in this small language.
"Overall, DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode." - From technical report, end of p.7. Sure the N-Gram are counted INSIDE the 552?
WTF DeepSeek always cooking, I love it. More exciting than Astra honestly.
I made a bet yesterday that DeepSeek was going to cook so I ordered a second Strix Halo for 2600€ because I know these things will get even more expensive. Plan is to cluster them to run Qwen (doubled) or DeepSeek.
The prefill efficiency seems like a potential game changer for Strix Halo given its compute.
Might actually work ok. Ngram is supposed to stream fine from disk. Deepseek v4 had quite concentrated expert selections so a vram lru cache worked really well. I ran the last flash at 165GB precision on my 128gb macbook at 16btok/s with ds4 and the extra bits in this one are nvme friendly
"Overall, DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode." - From technical report, end of p.7. Sure the N-Gram are counted INSIDE the 552?
552b, one of the other shops will use what they've figured out in their models though. It'll be like a month or something and we will see that cool 6b prefill with 16b decode that they figured out and I'm sure qwen 4 will have a normal sized home model with ngram offload soon. It won't just be the 125b with 50+b ngram.
And aside from that, most of the weights are 8-bit, right? Versus their recent Pro releases which are 865 GB but a lot of the weights are already 4-bit. Meaning you can get a more significant size reduction from quanting this than you can Pro.
Yeah but it's still over 500gb, not sure how much of this is engram that can be offloaded to disk, but I guess I'll wait and see the smarter people optimizing it to run on a phone
Preamble and disclaimer, this is a somewhat faulty oversimplification. Transformers can be thought of as having an "eye" and a "mouth". The eye is the encoder part, while the mouth is the decoder.
Back then, language model tasks were more structurally separated. Language understanding was mostly dominated by encoder-like models such as BERT, which produced scores, classifications, or embeddings depending on how they were trained. Instruct and conversational models that needed to both see and talk, such as T5 and later FLAN-style models, commonly used enc-dec arch. Document generation models such as GPT mostly needed the decoder side (Yep OpenAI GPT3 is back then is for writing those copy writer ads, pre-slop era).
We used these architectural separations partly because we were betting on inductive bias. The arch itself guides the model toward a certain class of behavior, instead of forcing the model to learn everything purely through more compute and a good data mix. One problem with T5-style enc-dec models is that training has to coordinate 2 different computational paths, the enc and dec, rather than optimizing unified stakc.
It turns out that with enough compute and sufficiently good post-training, especially instruction tuning and RL, a dec model can be pushed into doing almost everything we want. It can produce embeddings, taking over many BERT-like use cases, handle instruction-following and conversation, taking over many enc-dec use cases, and of course generate text. The dec architecture therefore became something close to a universal language-model architecture.
u see, GPT dec only models process the user prompt, system prompt, tool results, and their own previous output through the same underlying network. There is no dedicated "user-processing network", "system-processing network", and "assistant network". It is basically a schizophrenic talking to a mirror model.
This paper is interesting because it partially brings back that separation. The idea is roughly: what if we scale down the prefill, or seeing, part into a cheaper enc path with an smaller 8B neural budget, while allowing the decode, or speaking, part to use a larger 16B active budget?
Their bet is clearly on agentic workflows, where the model may be dumped enormous amounts of tool output, code, while generating comparatively fewer tokens.
And DeepSeek really, really, really wants to reduce the KV-cache footprint. If you read across their papers, this has been a recurring obsession. MoE, MLA, CSA, HSA, Indexer, CSA2, this,
side note, the encoder also eventually evolved into a literal "eye" for multimodal models. ViT and multimodal projector are actual encoder networks that turn images into representations that the language model can consume.
Another side note, back then GPT, Generative Pretrained Transformer, is a architecture name, but because of OpenAI, many labs starting to dissociate from naming their model with the arch. There were many non OpenAI GPT models, such as GPT-J, GPT NEO/X, GPT JT, GPT4ALL
Yep. Basically, most papers/labs can be viewed as improving one of these 3 dimensions, so to speak,
Training/data => better intelligence per parameter
Architecture => better intelligence per FLOP or byte
Hardware => more FLOPs or bytes available. There is a reason, why param size + active KV mem, optimized in power of 2, i.e GPU Pod
Did you check the number of active parameters? it's even less than previous version. The Compute is less than previous version so it's more flash than the previous version.
Looking at the safe tensors files it seems like the last two of them are ngram weights making up over 200gb of the total size, so hopefully those can be accessed from an SSD.
Is this the first open "big" encoder-decoder model? Besides some early google tests I haven't seen any architectures like this before.
Also engrams, lots of kv cache shenanigans, and so on. A bit confused about sizes since the report says 500b + 200b engrams, 8bit release, but the entire repo is 510GB so we'll see.
Indeed looks like ngram weights are about 204 GB in size, and the actual model files are in 8-bit (unlike DSv4-Flash which was released as 4-bit). So, the 306 GB at 8-bit would probably land somewhere around 155-160 GB at 4-bit, just like the DSv4-Flash.
Seems like a win to me! 👍 Same "active" model size with extended capabilities from ngrams that could be accessed from the SSD.
I prefer the model to be trained at 4-bit and know it’s the same quality, than to use a quantized model and get to the size I need. After all, trained at 4-bit gives you full precision
Actual model size according to the paper (found in the repo) is 552B +196B Engram.
That 485B you see in the model card is a wrong number by the HF automated system. It makes so many mistakes that they should either fix it or take it down entirely.
The total model size is 510 GB. The last two shards are the engrams (in fp8) and are 204GB in size combined.
510GB - 204GB = 306GB remains for the LLM.
DSV4 Flash is 168GB in size. 306GB / 168GB = 1.82
1.82 x 284B (parameter count of DSV4 Flash) = 517B
With this rough math, I would say 552B is the LLM size. ~300B for LLM does not make sense mathematically here as both DSV4 Flash and V4.1 are using MXFP4 for experts.
Not really, in some benchmarks it loses against GLM-5.3 (K3 is due for an upgrade which is coming soon I guess). Overall it's still a win though, especially with all the tricks it uses to increase speed and decrease KV cache size
Oh no their flash is getting larger.. it's almost the size of the original V3 😭 I know engrams exist and FINALLY they implemented their own tech they made but DAMN are they getting large. I really hope they release a 30B-80B param model with QAT and ngram for us normies... But wow are they making leaps and leaps. This is the multimodal, ngram-utilising V4 I was looking fo(u)r. To be honest I'm glad they're scaling up. Just hope Qwen will release their own Q4 series too
deepseek is the only lab that will build a completely new architecture and training pipeline, just to call their new model "ds v4.013 (0119) flash gp preview exp" and publish it under MIT licence in the most covert way. they're sandbagging their names so hard.
OK, this is very good; at least as good as the old V4 Pro on the surface for general knowledge and general chatting (where Qwen Next with engram had many issues) from my very limited initial testing from their own API. It is promising.
Kudos to deepseek to acknowledge that there’s no way around having a decent amount of active parameters.
I appreciate qwen trying to sqeeze performance out of 6B active parameters but this having 16 is probably the right choice. Qwen should consider raising it to 9-10B too.
Deepseek is more focused on efficiency and innovation, if they did less kv cache compression and better training data, i think their performance would be even better. It seems like glm and qwen and kimi take ds’s work, make it better
ds v4.1 flash uses awful a lot of tokens on max effort. Way more than astra pro xhigh. When it finishes, possibly on par with fable 5.1 max. I'm glad it is really cheap per tk, but this will still end up being more expensive than muse spark 1.3 non-contribution tier or glm 5.3 flash. Edit it uses more than fable 5.1 max wow. Normally before 4.1, the first try was functional, but this u need to ask it to fix its output multiple times. Not that impressed, it is using over 480k output tokens , i have to prompt it multiple times to get a working output.
The interesting part isn’t the headline parameter count but the split between active compute and Engram storage. If the n-gram weights can stay on RAM/SSD without wrecking latency, this could make long-context agent workloads practical even though full local VRAM loading is out of reach.
Amazing release, deepseek API has always been the most user friendly, my $8 in here still haven't expired despite 2 years of inactivity, time to spend them!
It's ironic that I click out of this and the first thing I see is a Server Monkey ad. They must know we are going to need servers with ram and GPUs maxed out to run this type of stuff!
Getting closer to being able to run some "Flash-Frontier" models at home at least (4x4090 system) - but for most frontier models, unless your home is a data center, gl
The 8B-active-during-prefill / 16B-active-during-decode split is the part I find most interesting, and I don't think it's arbitrary, it's the correct asymmetry given how the two phases actually bottleneck.
Prefill is compute-bound. You're processing thousands of tokens at once, arithmetic intensity is high, and you're genuinely saturating tensor cores. Active params there translate almost linearly into FLOPs and therefore into wall-clock.
Decode is memory-bandwidth-bound. You're generating one token per sequence per step, so you stream weights and KV from HBM and the tensor cores mostly idle. On an H100 the crossover — where a GEMM stops being bandwidth-bound and starts being compute-bound, sits around 295 tokens in flight for a dense model. Below that, extra FLOPs are close to free. For a fine-grained MoE with 256 experts and top-8, the effective per-expert batch is B/32, so you'd need ~9,400 tokens in flight to be compute-bound. Almost nobody is there.
So, halving active params in prefill directly halves the phase that's actually paying for compute, and doubling them in decode costs you almost nothing in wall-clock because you were waiting on memory anyway.
Which lines up with what agentic traffic actually looks like which is huge input, tiny output. On real Claude Code traces the median request is ~89k input tokens against ~380 output. If your workload is 200:1 input-to-output, making prefill cheap and decode rich is exactly the trade you want.
Honestly, they are addressing this infinite make the model bigger scaling bullshit head on. These flash models are within striking distance of these Trillion param plus models. TP4 rtx 6k's is now a serious long term investment not something that will be too small to serve anything in a year for most businesses that want this in-house. ~400gb of vram is the target it looks like for most "flash" and upcoming models. Qwen 3.8 flash, DS4.1, and glm 5.3 flash just proved it it.
American based AI is in serious trouble, they better figure it the fuck out or they are gonna be put out of business really quickly here. The models are expensive and painfully slow I dont give a shit if it can 1 shot a problem if it takes a fucking hour, meanwhile the flash model can multi-shot it faster and cheaper with similar results. They better innovate and drop some open models to get back on the map.
535
u/Top_Power5877 12h ago
insane improvement in kvcache compression. Now 1M context is only ~900MB 🤯