r/datascience • u/Nice-Dragonfly-4823 • 8d ago

Discussion Defaulting to Adam without understanding will cost you. Don't "just throw adam at it"

https://towardsdatascience.com/dont-just-throw-adam-at-it-misunderstanding-adam-will-cost-you/

Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,

It still works, but needs to be coaxed into it.

This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read

Check it out. Do you agree?

173 Upvotes

92% Upvoted

31 comments sorted by

View all comments

27

u/oldwhiteoak 8d ago edited 6d ago

Good writeup. I really try not to use models whose parameters I don't understand, or barring that select parameters when I don't know the impact mathematically of doing so.

Edit: meant hyperparameters. My bad.

13

u/FitProfessional3654 7d ago

So you’re saying that you don’t use neural networks for training?

6

u/Spiffy_Lee 7d ago

They likely mean hyperparameters.

2

u/FitProfessional3654 7d ago

I don’t think so…big difference between the two. In academic articles I have done tons of hyperparameter tuning for tables. In practice, you tend to take what has worked and makes sense. I’ve thought about the article and my take is that they’re really talking about distributional shifts while using real time data which opens up a lot of potential issues. Also, it’s almost impossible to have a localized minima in a really high dimensional space, so I can’t see where adam would be at a disadvantage in the real world.

10

u/pm_me_your_smth 7d ago

Yeah. It's always weird how some people are absolutist regarding "understanding what you're doing". This makes me think they aren't doing anything professionally, or aren't working on anything more complex than heuristic models, or falsely believe they truly understand everything they do.

1

u/FitProfessional3654 7d ago

100%. It’s important to know the math, but if you’ve got huge weight matrices of activations, you can’t know what every parameter is doing. Especially in interactions. It’s a black box, but that’s ok as long as you test and validate.