PPO is so Back!

Recently, I got the chance to plan a proper post-training run on a small open-weight model. Naturally, I'm jumping for joy. Those who know me know of my lifelong quest to find funded compute. I am keeping the model, environment, and intended evaluations private, but I can write about some of the questions I had before the first run.

Obviously the number one thing on my mind is compute provider. I want to run one stack, and part of my experiment is running large-scale baselines of the base model on various benchmarks, which I can nicely parallelize at the CPU level. Big bill incoming!

Ok now to the things that really matter. The setting is multi-turn and tool-using, with delayed outcomes and enough history that context management becomes part of the training problem. I came of age in a GRPO world. When I started doing research around a year ago, if you wanted better performance (generalization, consistency, whatever really) you pulled out ole reliable GRPO + rubrics. Then GLM released their 5.2 blog post. Had I not read that, and the reactions from the research community at scale, I'd probably be sitting here writing about a complex composite reward function. But nope, seems PPO is back. And I have compute now so I'm really not mad about it fr. I've felt for a while that GRPO is not good for long-context, multi-turn environment tasks. They use critic-based PPO to learn from compacted traces with different lengths. If that helps with the trajectories I'm working with, I'm buying it. I should probably reread the PPO paper tbh. You know how people used to say that GRPO was PPO without the critic. I'm like, "oh, PPO is just GRPO with a value estimator."

I'm very excited to train on multiple GPUs. I'm counting on models being better at debugging optimizer and tensor sharding than they were in early 2025. Debugging DDP in Cursor with Sonnet 3.7 was my personal hell. I've also been thinking about whether the critic needs to be the same size as the actor. As I'm writing this I'm diving into Natural Language Actor-Critic and Asymmetric PPO: mini-critics boost LLM reasoning. The latter makes me want to try a smaller critic with a larger actor (although no, I will not be using a set of mini critics trained on disjoint prompt shards, no matter how efficient that may be).

The comparison I most want to run is asymmetric PPO against GRPO. I think there is a misconception that GRPO will get us the results we want faster. I feel more like it may give us noisier results and slower convergence here. Also entropy collapse, we don't want that.

I hope to get some interesting results and update this experiment blog as I go. I have a sense that my intuition will sharpen a ton and that many of the decisions will begin to make absolute sense. I was revisiting GAE to understand how it improves credit assignment on multi-turn tasks, especially those with compaction or multiple actions per turn, and the discounted backup just made so much sense this time around. I remember working through the ARENA workbooks a couple months ago when my world was all GRPO and yeah, it made no sense.

Looking forward to sharing more of my learnings!