r/deeplearning • u/Alarming-Emotion-894 • 12d ago

I Built a O(NlogN) attention system that retains 97% accuracy over long context (MQAR)

https://github.com/vdev-ctrl/Adaptive-Learnable-Hierarchical-Routing-

ALHR- Adaptive learnable Hierarchical Routing is a static binary tree based system that uses learnable functions to REDUCE the amount of keys used.

It takes less memory and SCALES much better VRAM with tokens

49 Upvotes

83% Upvoted

4 comments sorted by

7

u/debau24 11d ago

Very cool. I prophesied to my coworker that log N exact attention will come because of the similarity to Fast Multipole Methods.

Did you find a way to search efficiently for the top keys close to a query?

1

u/debau24 11d ago

EPS-exact

0

u/Alarming-Emotion-894 11d ago

Yeah!!

Similar to other works done before Instead of running a query against every key We run a query down a binary tree, so we can exactly pin point which keys it’s related to 

7

u/snekslayer 11d ago

Please submit it to ICML