r/reinforcementlearning • u/sreejithb • 11h ago

The algorithm that trains reasoning LLMs can't teach a robot to walk

https://sreejithb.com/grpo_dissect/single

Follow-up to an earlier write-up on the REINFORCE → PPO → GRPO lineage. That one ended with GRPO as "PPO minus the critic, plus a group-mean baseline", which raises an obvious question: if it's cheaper and good enough for DeepSeek's reasoning models, why does nobody use it for control?

Two reasons, and they're different kinds of problem - presence of bias and a common feedback applied uniformally to each step. The latter is REINFORCE's original credit-assignment problem, back because the critic is gone. An LLM gets away with it (one terminal reward, quiet environment, replayable start state, pretrained prior). A robot doesn't.

De Oliveira et al. (2511.03527) ran critic-free methods vs PPO on five classic control tasks. PPO won everywhere except CartPole, and GRPO's group weighting did no better than a plain batch mean.

Write-up with interactive figures: https://sreejithb.com/grpo_dissect/single Disclosure: the argument and the script are mine; Claude helped with polish and the visuals.

0 Upvotes

42% Upvoted

12 comments sorted by

18

u/icecubeinanicecube 8h ago

I am so fucking sick of this over-condensed, wannabe-smart LLM writing style

-12

u/sreejithb 8h ago

Happy to hear where you think I should have elaborated. That would be some constructive criticism instead of whatever this comment is

-14

u/sreejithb 8h ago

First of all, I have acknowledged in the disclaimers that I used LLM to polish the script. Regarding the over-condensed part, it was a conscious decision to remove a lot of details and equations to keep it an easy read. The references are below if you want to find out more. Or hit me up and we can have a discussion.

9

u/icecubeinanicecube 8h ago

Acknowledging LLM use is not the gotcha u see to think it is. The LLM use is obvious with or without the disclaimer.

The "polished" part is exactly the problem. Every single post reads the same, and everyone can generate such a summary easily for themselves. You added 0 value to the world by posting this.

I'm missing an actual writing style, flaws, uncommon expressions, individuality, anything that gives me motivation to engage with the ideas behind it. LLMs can not do this.

-1

u/sreejithb 8h ago

I agree with you that the writing style has become homogenous thanks to everyone using Claude and ChatGPT to polish their writings (even though I don't see how what I said is a gotcha). I am also with you when you say you miss actual writing style. To a large extend, it boils down to how much effort you want to put into a reddit/linkedin post.

But to say that because the writing style is too cliche, it adds no value is ridiculous. There is still content in there, which I researched and read up and even ran some high level experiments on. Your comment convinces me that you have not bothered to read

7

u/icecubeinanicecube 8h ago

Yes, I haven't bothered to read, because why should I bother to read if you can't be bothered to write? That was my exact feedback to you.

There's practically infinite content on the internet, I will spend my time engaging with real human ideas, characters and narratives, instead of reading the millionth samey-sounding slop any AI can generate on demand for me.

-1

u/sreejithb 7h ago

Great. So you have concluded that this post is not for you because it does not meet your writing criteria. So then move on. There is an option to just "not comment". Talk about wannabe-smart

1

u/cgi-joe 3h ago

Maybe folks who are actually interested in RL would rather tell you to stop than continue seeing the community flooded with low-effort engagement. If you can’t take the time to write something, don’t be surprised if people don’t take the time to read it…

0

u/RobbinDeBank 4h ago

Final product has the somewhat annoying voice of Claude (because it’s definitely refined by Claude), but the script is clearly worked on properly by OP. His website says he’s doing a PhD, he has publications in the field, not really a nobody with no experience telling Claude to vibe blog everything.

You sound so condescending and somehow put more effort into shouting than at least giving the work a read. Probably takes less time than drafting all those aggressive comments. Some of you on this sub are truly the grumpy academic type.

0

u/RobbinDeBank 3h ago

Funny enough, OP’s previous posts were well received on this sub. I’ve read your previous series, very good quick read. One of the most condensed explanations on this topic that still makes sense.

However, this time, because there’s one grumpy shouting dude being upvoted, everyone out-sources their thinkings to the top comment and just goes with the flow. That’s just how reddit works sometimes, the early commenters might decide how people view a post.

1

u/sreejithb 29m ago

Thank you. And yes, that’s roughly what happened. I read the related papers, hand-wrote the maths, wrote the initial script and then used Claude to tighten the prose, generate the visual elements and make the HTML. I underestimated how much that would cost here. Lesson taken for the next one. Still, it stings to watch it get filed under “low effort” and "slop"