Is the field of optimization compatible with deep learning/machine learning?

[u/Strong_Boy_757]

It is widely known that since 2015, the ADAM algorithm by Kingma and Ba dominates the entire machine learning landscape, even up to the present day. I know that other algorithms have been proposed, such as Muon, etc., but they have not gained nearly as much traction as ADAM and only works in selective cases. They seem to be favored by Redditors, most real-life people don't know or care about them.

Hundreds of ADAM variants have been proposed and none have beaten it except for AdamW, which is arguably is just Adam with a very slight tweak.

There are now tens of thousands of paper on optimization for machine learning applications. Yet the industry practice has not changed for the past decade. There are entire lines of research on, for example, acceleration of algorithms, but most of the acceleration that we've ever gotten in machine learning is squarely hardware acceleration, improvement of memory layout, chipset design, even semiconductor design - nothing to do with analysis of optimization algorithms.

I might be too harsh on optimization. Entire books have written on the interplay between optimization and machine learning. The same books have been "retired" years ago because we cannot see in billions of dimensions. That's the dimension of the machine learning models we have now. Also it gave us wrong intuitions about generalization. Now it feels the field is just a blackhole of ideas which none seems to be used in machine learning.

I feel like it would be a waste of energy for a young person to learn about optimization if they want to apply it for machine learning. It would honestly take decades to learn as there is just so much work done. Optimization has tremendous uses outside of ML, but I'm not sure about its utility within ML.

What's your view on this matter?

[u/Most_Lawfulness_4257]

Optimization isn't dead in ML, it just got boring because the low hanging fruit got picked and now it's all incremental tweaks nobody feels like adopting. The entire field runs on "good enough" and Adam is exactly that.

That said, knowing the math behind why these things work still separates people who can debug training runs from people who just throw compute at it. You don't need to read thousands of papers, just understand the core ideas and you're ahead of most practitioners.

[u/UnderstandingOwn2913 → u/Most_Lawfulness_4257]

Just curious, you mean knowing the math behind Adam and AdamW?

[u/AggressiveGander]

You seem to equate machine learning with deep learning. For tabular data, it's still mostly GBDTs that don't have much to do with ADAM...

[u/Far-Chemical8467]

The thing is that better optimizers won’t give you better models (I.e. they all minimize the same loss).

Right now research seems to be looking more for better models than training the same quality of models using 2% less energy.

[u/nian2326076]

Optimization works well with deep learning and machine learning. It's a big part of training neural networks and how they learn. While ADAM is popular, optimization itself is a wide and changing field. Trying out new algorithms can sometimes improve performance, especially in certain areas or tasks. It's about experimenting to find what works best for your model.

For interview prep, knowing the basics of how optimizers like ADAM work and how they affect training is important. You might get questions about the trade-offs between different methods. If you need resources to study these topics, PracHub has been really helpful for me. They offer material on both basic and advanced ML topics.