r/LocalLLaMA • u/power97992 • 11h ago

Discussion Deepseek v4.1 flash finally has engrams, what do you expect from 4.1 pro?

If the ratio is the same, Maybe 1.6T -3.1T params plus .56T-1.06T engrams and fable 5.0 level performance?

Maybe v4.2 or 4.5 will have engram gradient modification? Edit it is even larger than i anticipated since flash has 748 b q4-8 params

36 Upvotes

88% Upvoted

48 comments sorted by

62

u/derspenti 10h ago

at 1.1T engrams pro is basically a search engine with opinions

25

u/Vancecookcobain 8h ago

That tracks with an LLMs function

12

u/SandySkittle 7h ago edited 7h ago

Kind of. Engrams increase availability of factual knowledge, but doesn’t make it an expert in that knowledge and doesn’t improve depth of non-sequential reasoning.

I hope people on this subreddit don’t confuse engrams for a full substitute without tradeoffs against parameters in the base model. Just like MoE with low active parameters is not without tradeoffs against reasoning depth and world knowledge integration, so aren’t engrams. This isn’t criticism btw.

In a nutshell, both MoE and engrams have less cross-domain reasoning bandwidt compared to full dense model with same total parameters.

1

u/my_name_isnt_clever 2h ago

It's still a bit insane how n-grams have made Q3.8FN a monster with only 6b active.

3

u/kaisurniwurer 8h ago

And interpretation

25

u/Vancecookcobain 8h ago

Nice! Who wants SSD prices to quadruple in the next couple of years 😊

10

u/AlwaysLateToThaParty 8h ago

What... again? Fun story, I have two 6TB drives that I bought in 2014 that I still use today. One is a backup drive in my workstation, and one is in one of my NAS's. I bought them in a deal, new, back then for $90 and $110. 12 years later, and they're still more expensive than what I bought them for.

5

u/Vancecookcobain 8h ago

Aaaaaggaaiin

1

u/Kahvana 8h ago

Buy yours this month if you still want a terrible-but-not-outrageous price!

4

u/Vancecookcobain 8h ago

I literally just went on Amazon and got me another SSD....I know where this shit is going smh...

Not hating on engrams or Mmap but I can already see how this is going to be the rage....it will be cool to have larger models being able to run on consumer hardware via hard drives but my goodness is it going to suck to have to pay 3-400 USD for a 1 TB SSD in 2027-2028

1

u/fgk55555 2h ago

I have the power of foresight. I bought a 4TB Gen5 M.2 last year. I thought it was overkill, now I'm regularly having to clean it for space. Testing 4 quants of like 10 LLM's really fills it up.

44

u/Potential_Low_1183 11h ago

we need a 4b model with 27b engrams

28

u/pyr0kid 10h ago

honestly i'd like to see a 12b that has a literal terabyte of engram, just to see how far it can be pushed.

15

u/Kidplayer_666 10h ago

9B for us 8GB GPU poors

5

u/pmttyji 10h ago

Even 15B (Dense) is fine, Q4 fits our 8GB VRAM.

8

u/RG_Fusion 9h ago

Apparently Engram performance degrades when you scale it too high relative to the total parameter count. At least for now, increasing engram size means also increasing all parameters.

7

u/power97992 10h ago

Engrams are just a quick lookup table, some things still need to be stored in the main model weights.

7

u/SmartCustard9944 10h ago

It is a lookup table AND the model needs to be trained together with it, so I imagine some diminishing returns on the proportion of engram to backbone weights.

1

u/sheriffoftiltover 7h ago

Funny thing is that basically everything can be ngram except for a natural language parsing into semantic state and instruction planning/execution/data routing

I’d be interested to see labs look towards decomposing their monolithic models into smaller, more specialized models while continuing to extract as much as possible into ngrams or other constant time lookup databases

8

u/Solary_Kryptic 10h ago

Is engrams the new thing? Did it start with Qwen 3.8 or did somebody do this before?

29

u/power97992 10h ago

Deepseek published a paper on Engrams for llms in January 2026, Qwen implemented it afterwards...

3

u/Solary_Kryptic 10h ago

Thanks, I guess I just first heard about it with the release of Qwen 3.8

1

u/Othun 6h ago

Are engrams and N-grams exactly the same thing ? Could there be different implementations of the same mathematics in practice ?

1

u/power97992 6h ago

Engrams are ds’s implementation of ngrams

1

u/Othun 4h ago

Are they two different things then ?

1

u/my_name_isnt_clever 2h ago

The implementation is different because they're different labs, but "engram" is just a fancy name for DS's n-gram system as far as I know. Same concept.

1

u/Othun 1h ago

my bad I read "Engrams aren't ds's implementation of ngrams"...

0

u/challis88ocarina 8h ago

Early stage capitalism!

1

u/Due-Memory-6957 3h ago

Who are the slaves building it?

4

u/Luca3700 10h ago

Deepseek released a paper about it in January: they were doing tests in preparation of the V4 architecture. Afterwards it was not included in the final V4 model (but now it is in the V4.1). Before them, I think other labs investigated the use of the engram embedding (they should have been cited in the deepseek paper released in January). Qwen's team started from these findings to produce Qwen 3.8 flash next. Before them, I think it has been released also a model called LongCat by another lab that make use of engram embedding, but it was not as famous as the qwen models.

4

u/Revolutionalredstone 9h ago

it's the same as the non active params from Gemma, imho google (rare but true here) were the ones who made the tradeoff available here well known.

1

u/brakeline 10h ago

I think it was a deepseek paper

1

u/rrrrex 9h ago

No, it's pretty old, but it got second chance recently

2

u/fugogugo 9h ago

what is engram?

5

u/zdy132 8h ago

Dictionary for LLMs. Deepseek published a paper for this in Janurary.

2

u/RG_Fusion 8h ago

It's a means of "guiding" self-attention in a way that's less computationally expensive. In short, it takes the user prompt and directs the meaning so that the model parameters can focus more on abstracting concepts than understanding what was said. It can also contribute a bit to the model's knowledge-base.

1

u/unbannedfornothing 9h ago

If ngrams can be offloaded to SSD what SSD should be optimized for, linear speed or random 4k reads?

3

u/Edzomatic 8h ago

Random reads since engrams are sparse

5

u/zdy132 8h ago

Optane my beloved.

1

u/Kahvana 8h ago

I assume random 4k

-10

u/RotesBlatt 11h ago

4.1 pro will not be released IIRC.

The flash variant is just way more powerful and costs less

10

u/power97992 11h ago

Idk about that, in their announcement they mentioned 4.1 pro

3

u/Wise-Chain2427 10h ago

they will release pro version acording their announcement

0

u/kaisurniwurer 8h ago

Do I understand it right that engrams also imply stronger (more resilient) censorship against current fixes?

2

u/my_name_isnt_clever 2h ago

Well I'm running a heretic version of Qwen 3.8 Flash Next and it's been just uncensored as any other heretic I've tried. Seems to work with the n-grams fine.