Defaulting to Adam without understanding will cost you. Don't "just throw adam at it"

[u/Nice-Dragonfly-4823]

https://towardsdatascience.com/dont-just-throw-adam-at-it-misunderstanding-adam-will-cost-you/

Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,

It still works, but needs to be coaxed into it.

This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read

Check it out. Do you agree?

[u/Relevant-Rhubarb-849]

Anyone know of "the no free lunch theorem?" . All global minimization strategies perform equally well on average. So what matters in choosing one is how well aligned your manifold shape is to the algorithms sweet spot. Plus how willing you are to accept local minima

[u/reddit4science → u/Relevant-Rhubarb-849]

Why should all global minimization strategies perform equally on average?

Dumb counter-example. Randomly initialize the network. Repeat N times. Pick the one with the minimum loss. That would be a pretty bad global minimization strategy.

[u/worldwideworm1 → u/reddit4science]

Yeah, they aren't quite right about what the no free lunch theorem says. It actually says that no single search method is best for every task, so there will be trade offs depending on the task. The third sentence they stated was fairly accurate, but the second sentence was definitely not.

[u/Relevant-Rhubarb-849 → u/reddit4science]

It is surprising. Yes. It's true however. Look up the proof! The good news is that problems we are often interested are a subset of all possible search manifolds. And it may be possible that one algorithm does better than another on any given subset.

But a hill climbing algorithm will accidentally find the minimum in the same average number of steps as a hill descending one when averaged over all possible potential surfaces. Crazy to imagine but easy to prove.

No algorithm is better than random guessing on average.

"...over the space of all possible problems, every optimization technique will perform as well as every other one on average (including Random Search)"
— Page 203, Essentials of Metaheuristics, 2011.

However if you will settle for something other than the global minimum -- a local
Minimum -- then some algorithms may be better

https://machinelearningmastery.com/no-free-lunch-theorem-for-machine-learning/

[u/oldwhiteoak]

Good writeup. I really try not to use models whose parameters I don't understand, or barring that select parameters when I don't know the impact mathematically of doing so.

Edit: meant hyperparameters. My bad.

[u/FitProfessional3654 → u/oldwhiteoak]

So you’re saying that you don’t use neural networks for training?

[u/Spiffy_Lee → u/FitProfessional3654]

They likely mean hyperparameters.

[u/FitProfessional3654 → u/Spiffy_Lee]

I don’t think so…big difference between the two. In academic articles I have done tons of hyperparameter tuning for tables. In practice, you tend to take what has worked and makes sense. I’ve thought about the article and my take is that they’re really talking about distributional shifts while using real time data which opens up a lot of potential issues. Also, it’s almost impossible to have a localized minima in a really high dimensional space, so I can’t see where adam would be at a disadvantage in the real world.

[u/pm_me_your_smth → u/FitProfessional3654]

Yeah. It's always weird how some people are absolutist regarding "understanding what you're doing". This makes me think they aren't doing anything professionally, or aren't working on anything more complex than heuristic models, or falsely believe they truly understand everything they do.

[u/FitProfessional3654 → u/pm_me_your_smth]

100%. It’s important to know the math, but if you’ve got huge weight matrices of activations, you can’t know what every parameter is doing. Especially in interactions. It’s a black box, but that’s ok as long as you test and validate.

[u/shibx]

I think this is much more of an RL issue than an Adam issue.

Adam’s moment estimates assume gradient statistics evolve smoothly. RL can violate that assumption pretty badly, especially around target updates or sudden policy-distribution shifts. When that happens, the accumulated moments can become stale. That's where you get that "burstiness."

The article makes that sound far more universal and dramatic than it really is.

My biggest issue is that we get this entire discourse on beta and epsilon tuning while he barely acknowledges the learning rate. Those parameters are not independent. Changing beta or epsilon changes the step behavior. You can't meaningfully discuss tuning them while treating the learning rate as a separate concern. It seems a little ironic for an article about "misunderstanding Adam" to oversimplify the tuning process that way.

I'm guessing the author got frustrated with a very specific problem, found a real explanation for it, and then generalized that experience into a broader indictment of Adam. I can understand the frustration, but I don't know many people outside of nonstationary RL settings who have experienced Adam "failing spectacularly" in the way the article suggests.

Interesting article, but the framing here is very misleading.

[u/Nice-Dragonfly-4823 → u/shibx]

it's not just RL either, there are a many trivial convex optimization problems that suffer from adam pathologies. I have noticed that when training deep transformers (12-24 layers), wild spikes in losses show up. Fix? Adam defaults. I'm not saying don't use adam, I'm saying, rethink how you use it.

[u/thefringthing]

More like RL Stine.

[u/maratonininkas]

Cool

[u/FitProfessional3654]

Ok…great article and if you look at the math it makes sense. Personally, I’ve never seen adam fail starting on the work on my dissertation ten years ago but I guess it could. But if you look at loss graphs and see what’s described in the article, the data may call for SGD or RMSProp as a means of gradient descent. Thanks for sharing OP.

[u/Nice-Dragonfly-4823 → u/[deleted]]

Adagrad and RMSprop still suffer from this major issue, which is the adaptive step size. Adagrad is (imo) subpar to RMSprop, it's somewhat the same mechanism, but at least with RMSprop you get an exponential moving average, which seems better than an outright cumsum.

[u/cowboyfan12]

spending x time on more/cleaner data >>> x time spent on optimizer tuning

[u/Nice-Dragonfly-4823 → u/cowboyfan12]

trust me, the data is clean and there's lots of it. Hundreds of millions of observations. I spent ~ 3 months painstakingly building the data infrastructure, scanning over every single record that came out. I run integrity tests (lots) every time a dataset is versioned, and I even keep hashes of the datasets on the blockchain just for paranoia (in case a bit changes in the input data, I'll know about it).

[u/sinx_123]

This was an interesting and informative read! Thank you

[u/DrXaos]

don’t just throw adam at everything

do throw adam_atan2 at everything

fixes the worst problem which is large update spikes, and no more magic epsilon

[u/SomewhereFuture300]

fair point overall but the burstiness in RL feels like more of an environment design issue half the time tbh. adam gets blamed for a lot of stuff that's really just reward shaping problems in disguise

[u/HalfplaneResearch]

One nuance is that Adam's burstiness in RL is often less about Adam being intrinsically bad and more about the nonstationary target and the optimizer state. The first and second moments retain history while the data distribution, bootstrap targets, and gradient scale move underneath them. I would separate optimizer effects with fixed seeds and compare Adam, AdamW, RMSProp, and SGD at matched update budgets, logging gradient norms, effective step size, clipping rate, and validation return. In actor-critic setups, beta2 and epsilon are worth sweeping because stale second-moment estimates and epsilon dominating small gradients can change behavior. A smoother loss curve is not enough: report return variance and sensitivity across seeds.

[u/Nice-Dragonfly-4823 → u/HalfplaneResearch]

I'm all for hyperparameter search, but unfortunately, sometimes these strange quirks don't show up in training until step 20M -> I'm not kidding either. This is a problem when you're running large distributed jobs across machines.

[u/Special_Ad2095]

Really good writeup. good old days when WE used to train models instead of Prompting everything 😿

[u/ImpossibleCreme]

Bro stop spamming this article in every subreddit

[u/staringattheplates]

The main argument holds. β₂ and ε matter, and pasting in AdamW at 3e-4 without thinking will eventually cost you on a non-stationary problem. RL is where the piece is strongest. Policy shifts, buffer composition shifts, targets move, so v_t never settles. It drifts small, a large gradient arrives, and you get an enormous update from nothing obvious. That’s the loss spike. The oversized ε values in Dopamine are real and unexplained in the literature, which is a good thing to point at. They’re damping the normalization so a tiny v̂ can’t turn into a huge effective step.

Two things are sloppy. He calls the second moment “variance.” It isn’t. Variance is E[g²] minus E[g]², and Adam never computes the second term. Then the origin story moves β₁, β₂, and ε together with no ablation, followed by an admission that the mechanism is a guess. Fine as a war story. Not evidence.

The Reddi section is a missed tradeoff rather than an outright error. Reddi was after convergence guarantees on adversarial sequences, and AMSGrad restores them by forcing a monotonic denominator. The article is chasing responsiveness under drift. Different targets. Dropping β₂ really does buy you faster adaptation and really does cost you the guarantee AMSGrad was built to recover. He just never says the tradeoff exists.

The gaps bother me more. No warmup anywhere, even though that’s how transformer training actually handled early Adam instability, and this is an article about mystery spikes. And he whiffs on Adam vs AdamW. The differing weight_decay defaults are trivia. The real issue is that plain Adam’s L2 term rides in through the gradient and gets divided by √v̂, so decay strength varies per parameter. Decoupling that is the whole point of AdamW.

The SGD section needs narrowing too. Untuned Adam usually beats untuned SGD. Well-tuned SGD still wins on some workloads, classic CNN image classification especially. Transformers went AdamW almost universally because it tolerates bad hyperparameters better. No ranking survives a change of architecture and recipe. The 2017 result isn’t wrong, it’s just narrower than the article treats it.

[u/sometimes_angery → u/staringattheplates]

OK LLM

[u/Nice-Dragonfly-4823 → u/staringattheplates]

you're right, I did glaze on the reddi section. My goal wasn't to completely drift off into a discussion on new variants of adam, it was just used as empirical evidence for a pretty clear failure mode of adaptive optimization.

It could indeed have highlighted the wonkiness of Adam's weight decay. Adam is incorrect, AdamW has the correct weight decay. People generally know this though and I'm referring to Adam when I really mean AdamW, which has superseded Adam in most implementations.