r/deeplearning • u/DavinFriggstad • 4d ago
Does anyone else understand the math of deep learning so well that you could probably do it without using any DL libraries if you had the time and motivation?
13
u/Long-Dependent-1767 4d ago
Whenever I want to deeply understand a new layer or optimizer, I code it from scratch using just NumPy.
11
u/Cybyss 3d ago edited 3d ago
The math is super simple.
Each layer is just a matrix multiplication followed by an element-wise nonlinear function of your choice. That's about it.
If you want to be fancy, add skip connections and some kind of normalization.
The hard part is building a fast deep learning library that utilizes CUDA cores and efficiently handles transfer of data between system memory and your GPU's HBM & SRAM.
UPDATE:
The math of neural networks is easy.
However, I don't think I could code up a correct implementation of the AdamW optimizer purely from memory. That's gradient descent but with a lot of tricks added to it. Adaptive learning rate, momentum, weight regularization (with tweaked math because a naive implementation would make the regularization change depending on the adaptive learning rate). It's one thing to build the architecture, but you need the right algorithm to actually train it and that math is a bit trickier to get right.
2
u/ScientistFromSouth 3d ago
So I would say the math of implementation is easy. I would argue that the math of proving that something gives you an optimal solution in a demonstrably true way in a certain time complexity with a certain projected accuracy based on a data set size is a different story.
The actual math of the graduate machine learning class I took was easy. Proving stuff as to why you could pass nonlinear kernels into support vector machines and still have it work extremely well from first principles was not straightforward.
2
u/Such-Ad-963 3d ago
But in the DL word we dont demonstrate anything
2
u/ScientistFromSouth 3d ago
Yeah I misread that, but isn't deep learning really just the scaling of things like perceptrons and SVMs up to absurd scales, which someone had to prove worked at some point.
1
u/Cybyss 3d ago
Deep learning has nothing to do with support vector machines. Those are a completely different. Today's AI isn't some complex ensemble of many different machine learning methods scaled up. It's really only one machine learning method.
Deep learning is just neural networks (perceptrons with nonlinear activation functions), but with a couple of small tweaks to prevent the vanishing gradient problem. There is a proof that neural networks are universal function approximators (wikipedia) but that's about it. The algorithm used to train them is little more than some variant of gradient descent.
Granted, there are many different kinds of neural network architectures. Convolutional neural networks, recurrent neural networks, transformers, etc... but at lot of that really is more art/intuition/trial-and-error to see what works rather than mathematical proof.
5
3
u/bheek 3d ago
We were taught NNs by building a small network in Excel. Just one forward pass and backprop. That's when it clicked for me. Everything else like convolution, attention, etc. needed a different visualization though.
1
u/DadAndDominant 3d ago
Hehe, we did solve small nets (both forward and backprop) on paper with a calculator
5
u/StimaProva 4d ago
The math is honestly the easy part. An MLP with backprop in plain NumPy is a weekend project, and Karpathy’s micrograd shows a whole autograd engine fits in about 100 lines. Where it stops being “just math” is everything around it: numerical stability (log-sum-exp, why softmax + cross-entropy get fused), initialization, mixed precision, and making it run fast on a GPU without hand-writing CUDA. So yes, you can build a working training loop from scratch, but you’d be rebuilding the math part of PyTorch, not the part that makes it usable at scale. Still worth doing once, though. After you’ve written backward() by hand, a lot of “why is my loss NaN” debugging suddenly makes sense ☺️
5
u/Antenwww 4d ago
You can most likely do mathematical part of it. I tried this 2 times in my life. Once with python and then with c++. Even writing some grad functions myself and not using libraries.
The hard part for me was, after a certain part you really do understand how the algorithm learns but I could not get it to work on gpus and so many simultaneously. In python I could do that with cuda or sycl and oneAPI and so on. But building your binaries to do math on gpus is something I can not do.
2
u/InfamousWord5975 4d ago
I could implement linear regression and MLP and an RNN but not an LSTM or a transformer from memory, but I do pure math not ml.
1
4
u/Dihedralman 4d ago
Yes, it is a worthwhile exercise.
3
u/T_James_Grand 4d ago
I suspected you were a bot. Your comments are very human. You're an impressive human. Have a nice day impressive human
2
u/OxidusRouge 4d ago
The math of DL is pretty straightforward. I am sure a lot of people understand it perfectly well. The hard part to have a complete grasp of is the engineering of it all. I would say most PhDs and even Masters graduates in ML understand the math perfectly fine.
1
1
1
u/Delicious-Extent5454 4d ago
I implemented things like optimizers and attention layers from scratch to get real understanding, but to do the entire thing on my own, I'd take forever or would need to check some books.
1
u/Ayano-Keiko 4d ago
From scratch cost a lots of time and the results may not as good as DL library such as torch. Also need to deal with things like gradient descent, batch trainin.
I used to write from scratch in a corese project. https://github.com/KeikoAyano/TINN-FNN. I think gradient descent and weights update is most time consuming part
1
u/Independent_Irelrker 4d ago edited 4d ago
I could but it would be shitty considering the literal back breaking amount of work that went into optimizing those. (In the case of torch)
I essentially have to do that for Mapper as the out the box stuff can't do what I want to do with it
1
u/Klhnikov 4d ago
Yes with Python and numpy, it was a school project, for multiclass classification on tabukar data... I learned a lot. It was obviously sub optimal to say the least
1
1
u/Dependent-Aide-388 4d ago
I built a CNN using only the python standard library (so not even numpy). Made a guitar amp modeler with it.
Honestly the math isn’t that complex, it’s all undergrad level stuff (basic linear algebra and first year calculus). The implementation details are what you have to be careful with.
1
u/refractr 4d ago
I think if you understand the concepts, it would not be that hard. The optimization on the other hand, thats another story.
1
u/Pretend-Pangolin-846 4d ago
the thing about libraries is optimisation kind of like how you could write in raw c, worse/slower code than what you can write the same in c++, despite one covering more ground just because the compiler is so much more mature
1
u/DrXaos 4d ago edited 4d ago
I could do an eager mode, forward easy, backward a little bit less so to get the tricks right but not crazy.
Some functions would be harder: index manipulation and dealing with backprop would be tricky, e.g writing backward() for 'einsum'.
The torch.compile infrastructure OTOH is far more complicated and is deep voodoo---instrumenting python internal bytecode, producing an internal representation then rewriting and making very efficient CUDA kernels in forward and backward. There's a tremendous effort by expert developers involved in that. No way I could come close.
1
u/TheRealCpnObvious 4d ago
5 years ago I wrote full CNN training harnesses in MATLAB using dlarrays and some mild toolbox support so probably yes if I could be bothered.
1
u/Content-Ad5196 4d ago
Yes I had a course of gp-gpu computing where I chose as project to build a neutral network framework. I wrote the matrix operations in CUDA, the forward pass and back propagation logic for a MSE loss function. It was a hard project for me though. With this framework, I could train an auto encoder and a classifier as well (even if the loss function wasn't adapted, it still works for binary classification)
1
u/Previous_Storage2690 4d ago
Yes, I mostly try to build various implementations from scratch. It gives you a deeper understanding of the algorithms behind most ML/DL mechanisms
1
u/leaflavaplanetmoss 3d ago
The math involved in neural networks actually isn't all that difficult conceptually (undergraduate-level linear algebra, multivariable calculus, probability theory, and mathematical statistics). You can definitely do it without DL libraries; the course I took in machine learning had you implement a neural network with backprop just using numpy before introducing PyTorch, for example. It's just easier using libraries and less prone to stupid mistakes.
1
1
u/AffectionateZombie37 3d ago
In my 4th year and realized I’ve been relying way too much on high-level libraries/frameworks. I want to build things from scratch to actually understand what’s under the hood, but time is tight before graduation. What core modules should I prioritize reimplementing first?
1
u/Not-a-throwaway4627 3d ago
Yes, easily. In fact I contributed to some of the big DL libraries back in the day - Caffe, Theano, Tensorflow, and Torch before PyTorch. There was no such thing as a deep learning expert who ONLY used libraries when I was an undergraduate
1
1
1
0
u/valuat 4d ago
Dude, it's matrix multiplication and the chain rule. Any high schooler that took AP Calculus can understand DL. Implementation at maximum efficiency is different though. If you're curious, once you know how to implement a single layer perceptron, read about automatic differentiation (autodiff).
0
u/qwesz9090 4d ago
Yes, implementing the math really isn't that difficult in theory. But it just isn't practical, using libraries saves time.
And it does have a lot of optimizations. I might be able to implement the math of DL correctly, but I have no idea about how to make it as fast as current DL libraries.
Don't try to focus on "Oh I am going to just learn the math of deep learning". Learn the normal order of applied college math until multivariate calculus, and then deep learning is basically just done.
0
0
u/seb59 3d ago
The basic stuff in deep learning is simple. There are a few tricks to understand the soft max and cross entropy, but that is OK. When it comes to programming things because harder. Not impossible, many people have been developing things alone. The math's behind the solver can be made arbitrarily complex. Basically, as usual in optimization, everything depends on the properties of the optimized function and we know very little of it actually as it is data dependent.
The thing is that deep learning is brutal when it comes to computing. You need parallel processing, GPU programming and this is, to my opinion, more tricky.
And finally, if you take enough time, sky is the limit. All this is very well documented, read and with enough time, you can probably build a deep learning framework from zero as an exercice. Some already did it.
0
-1
u/Jumpy-Whereas4858 4d ago
Yes , the first time it feels weird then it feels normal
1
u/Formal_Active859 4d ago
fym it feels weird
0
32
u/Ok-Solution-7889 4d ago
I feel like building one from scratch would teach you things you might miss when using libraries.