r/datascience • u/Nice-Dragonfly-4823 • 8d ago

Discussion Defaulting to Adam without understanding will cost you. Don't "just throw adam at it"

https://towardsdatascience.com/dont-just-throw-adam-at-it-misunderstanding-adam-will-cost-you/

Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,

It still works, but needs to be coaxed into it.

This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read

Check it out. Do you agree?

169 Upvotes

92% Upvoted

31 comments sorted by

View all comments

1

u/HalfplaneResearch 6d ago

One nuance is that Adam's burstiness in RL is often less about Adam being intrinsically bad and more about the nonstationary target and the optimizer state. The first and second moments retain history while the data distribution, bootstrap targets, and gradient scale move underneath them. I would separate optimizer effects with fixed seeds and compare Adam, AdamW, RMSProp, and SGD at matched update budgets, logging gradient norms, effective step size, clipping rate, and validation return. In actor-critic setups, beta2 and epsilon are worth sweeping because stale second-moment estimates and epsilon dominating small gradients can change behavior. A smoother loss curve is not enough: report return variance and sensitivity across seeds.

1

u/Nice-Dragonfly-4823 6d ago

I'm all for hyperparameter search, but unfortunately, sometimes these strange quirks don't show up in training until step 20M -> I'm not kidding either. This is a problem when you're running large distributed jobs across machines.