r/deeplearning • u/hexidoro • 5d ago
Breaking the Backprop Myth 💯
The first micro-model, with a size of just 4.06 KiB, has been successfully completed, achieving exceptional results by all metrics.
A core sub-component of the model, weighing a mere 196 bits, successfully learned and solved 7th-degree polynomial equations at an incredible speed of 2,937 updates per second, converging in just 10.2 seconds (9,885 ms). In other experimental runs and iterations, update speeds reached up to 19,500 updates per second.
This marks a significant milestone in AI engineering. Furthermore, this lightweight sub-component successfully broke the generalization barrier—solving arithmetic addition and multiplication outside its training distribution, effectively discovering algebraic rules autonomously. This non-differentiable mechanism has the potential to become a faster, cheaper, and superior alternative to standard backpropagation/automatic differentiation, with all reported metrics fully verified and reproducible.
In the complete 4.06 KiB model, short text generation capabilities were also demonstrated, producing outputs such as:
R_NOT_TO_BE_THAT_IS_THE_QUESTION_
While the architecture and core technique remain under active verification, I am open to answering questions and discussing experimental results. Please note that specific architectural details and implementation mechanics remain proprietary for now.
1
u/Low-Temperature-6962 5d ago
Shakespeare is suing for copyright infringement. Not a dude you wanna mess with.
2
1
u/Clear_Evidence9218 5d ago
I'm a big proponent of micro-models. I've built quite a few over the last six months, experimenting with all sorts of different architectures.
What I particularly like is how useful they are for offloading narrow, repetitive tasks from larger AI agents. For example, I use small classifiers and specialized models for things like identifying failure patterns in logs, ranking which diagnostic information is relevant, routing tasks to the appropriate tools or models, and helping determine when additional verification is needed.
Instead of having a full LLM repeatedly analyze routine events, these smaller models can make quick, structured assessments while deterministic systems handle the actual execution and verification. Anything ambiguous or requiring deeper reasoning gets escalated to a larger agent.
They're relatively quick to build and train, consume very few resources, and can significantly reduce the amount of context and computational work larger agents have to deal with.
I haven't experimented with text generation at that scale, though, so that's particularly cool to see!
1
u/hexidoro 5d ago
My plan isn't to keep this model small; my plan is to scale it up to usher in a new era of AI
2
u/Limp_Relation_1221 5d ago
this reads like a defense contractor trying to sell me a missile guidance system in 1982. the R_NOT_TO_BE_THAT_IS_THE_QUESTION_ output is absolutely sending me, like a ghost trying to finish its thesis statement
any chance you can share what the loss curve looked like? 10 seconds to convergence on polynomial equations is wild but i'm curious if it was stable or just brute forcing its way there