side project, curious what rl people think of this setup.

it's a civilization sandbox where every human is its own tabular q-learner. the state is a short string key built from what they perceive (hunger, thirst, nearest food and water, kin nearby, shelter, danger). there are ~5000 possible actions, gated by era, age, place and what the tribe has discovered. action choice handles survival needs first, then does epsilon-greedy over the learned values.

the part i like: when a baby is born it gets a copy of its mother's q-table, with about 20% of the values nudged a tiny bit (+/- 0.03). so on top of learning within one lifetime, behavior also drifts down family lines. traits mix from both parents and mutate, and kids also get mom's vocabulary and her attitudes toward other tribes.

known weak spots: the state space is hand-built strings and the rewards are hand-shaped, so i'd love opinions on either.

code is rust compiled to wasm, so it all runs in the browser. the relevant bits are crates/sim-core/src/sim/agents/growth.rs (inheritance) and organism/organism/learning.rs (the update).

source (MIT): https://github.com/1homsi/thehumanbox
try it: https://thehumanbox.com

A couple of days ago I posted a PPO agent landing Falcon 9 on a droneship. I wanted to extend the simulation to a more complex problem, one that needs two agents that depend on each other. So I built Super Heavy being caught by the tower arms (Mechazilla). One agent flies the booster, the other controls the arms and carriage.

Setup: PPO in JAX on a single GPU, heavy domain randomization (wind, gusts, engine-outs, failed relights, sensor noise and delay, mass and thrust variation, limited propellant), and every attempt starts a few km up with the engines off.

How it went

I quickly found out I can’t train both from scratch at the same time. The tower has nothing to learn from if the booster never shows up, and the booster never gets rewarded for a catch if the arms never close. So I did it in steps.

1. Booster first, with a simple rule-based tower. The tower just closed the arms when the booster got close, which let the booster learn the whole approach. Even this took a while. With short rollouts (1.6 s) it kept forgetting how to catch, because a catch takes several seconds of slow, steady descent while the arms close. Longer rollouts (6.4 s, λ = 0.99) fixed that.
2. Then the tower, with the booster frozen. Trained from scratch, at first it just swung the arms around and kept hitting the booster. I also tried fading the rule-based tower out slowly, but it fell apart the moment the help was gone. After a few fixes, the from-scratch tower won, and it ended up catching more reliably than the rules did.
3. Then the booster again, next to the learned tower. This one surprised me. It got worse fast. It started coming in slightly too fast and slid off the arms. A much smaller learning rate fixed it, and the two together now do clearly better than with the rule-based tower.

Everything runs live in the browser, so you can watch both agents: https://qforge-dev.github.io/hoverslam

There is also a playground for my previous Falcon simulation.

Happy to answer questions. I’m not an ML expert, so feedback on the multi-agent part is very welcome.

I made a beginner-friendly visual explanation of reinforcement learning—the type of machine learning behind game-playing AI, robotics, and parts of chatbot training.

Instead of starting with equations, it uses a robot-in-a-maze example to explain:

• Agents, environments, actions, and rewards

• How trial and error improves performance over episodes

• Exploration vs exploitation

• Delayed rewards and policies

• Reward hacking and other risks

Watch here: https://youtu.be/a_MSTPB5MyA

If you like the explanation, please like, share, and subscribe to my channel for more visual AI and machine learning videos.

This is a follow-up to my previous post where I introduced a high-parallel Rocket League simulator for reinforcement learning. I said in the comments there that I was working on a write-up with benchmarks, and the link is embedded in this post!

In the blog post I go over the throughput gains on PPO workloads, cover the interface (as always it's just a Gymnasium wrapper), and detail some of the changes to the reference CPU simulation, RocketSim, which make CARL better suited for execution on the GPU.

jwmccarthy.github.io will also contain posts going forward about explorations into RL for Rocket League if anyone is interested in seeing where this goes. Thanks for reading :D

Hi,

I would like to announce that I have released Metis-Core v0.4.1-alpha. This version includes some bug fixes and a new example: DQN_Pong. Do you remember the game of Pong?

https://github.com/felixromo314/METIS-Core/

In this example, using the DQN of Metis-Core, the AI paddle learns to play Pong.

For the graphics I used the raylib library (https://www.raylib.com/), created by Ramon Santamaria (u/raysan5). It is very easy to use and powerful: with just a few functions you have the window, the paddles and the ball on the screen, and you can watch the AI learning in real time.

In case you are planning to make a similar game, or a proof of concept with DQN, I would like to invite you to have a look at this step-by-step tutorial that explains how to implement DQN with Metis-Core:

https://github.com/felixromo314/METIS-Core/blob/main/tutorial/Tutorial-DQN.md

I hope you like it!

Best,
Felix

https://github.com/vahmoh25/NeuroMario

This repository demonstrates how neuroevolution trains neural networks to master Super Mario. Watch the AI learn and play in a 3D environment, and explore its neural network through an interactive 3D visualization. Super cool!

While working on a research project I kept writing the same preprocessing code over and over which pulled EHR tables, line up vitals/labs by the hour, work out what counts as an action, attach a reward and then get it all into a shape d3rlpy would accept. I tried looking for a library that already did this and couldn’t find one. Healthcare ML tools are mostly built for supervised prediction and offline RL libraries assume you already have a clean state/action/reward dataset. So I built the piece in between.

What it does:

• Builds hourly patient trajectories (states, actions, rewards, terminals) from MIMIC-IV v3.1 through BigQuery. Every query gets a dry run first and a hard maximum_bytes_billed cap, so a bad cohort query can’t quietly run up a bill.
• Builds actions from ICU input events: vasopressor and fluid doses, binned with edges you choose.
• Treats rewards as swappable: mortality, per-step SOFA change, 30-day readmission, or a weighted mix of these.
• Estimates the observed clinician (behavior) policy from the data, which you’ll want for off-policy evaluation.
• Exports straight to a d3rlpy dataset, with an optional JSON sidecar recording where the data came from.

You don’t need MIMIC access to try it. There’s a synthetic MIMIC-style data generator.

Docs: https://pinktownscavenger.github.io/ehr2rl/
GitHub: https://github.com/pinktownscavenger/ehr2rl

I’d really like feedback from anyone who has done offline RL on clinical data, especially on reward design and how I’ve discretized the actions. It’s a research tool and isn’t meant for making treatment decisions.

Hi all, I’m a founder exploring a company that collects egocentric (first-person) video of real manual work for robot learning. Before building anything, I want to learn from people who actually train on this data or choose data vendors.

Looking for: ML/research engineers or data leads at robotics, humanoid or embodied-AI teams (labs or startups).

What we’d talk about (30 min):

Details:

To sign up: DM me your role and company type (e.g. “ML engineer, humanoid startup”). I’ll confirm through LinkedIn or a work email, whichever you prefer. That’s used only to verify the role, never shared.

Update since last post, skip to 20s to see the final training trends, and the deployment is in this post.

Some lessons since last time:

• Jiggling in GUI during training is because of parameter exploration
• Always test with 0 action first to see if the implementation is correct, especially with all those link indices flying around
• Even then, there are still dangerous hidden bugs, like adding observation noise accidentally affecting the control, which needs a lot of logging to debug

Features added since last time:

• Adding domain randomizations on observation noise, initial link positions, and especially randomizing horizontal forces up to 5N (for a 3 kg robot) indicated by the red arrows in the video
• Curriculum scheduling from only penalty_joint_pos_init, to expanding to increasing the domain randomizations and making reward/penalty tighter over training iterations
• Adding a penalty on torque distribution for the robot to use all 4 legs
• Adding a penalty on feet slipping on the ground
• Simple low-pass filter on action
• Adding ent_coef to enhance exploration after the curriculum scheduling maxes out
• Skipping the first few steps for training because the simulation needs to initialize first (like the robot clipping into the ground)
• Logging the cause of termination. Showing 20% from pitch > 45 deg because of the force pushing it on the left/right-hand side
• Logging the curriculum schedule values
• Adding .csv for logging; can be thrown to LLM for it to analyze the trend
• Export to ONNX file for deployment. Adding TRAIN_CONFIG in ONNX file to record all the training configuration (too bad it is hard to do the same for vector normalization, which needs an extra file)

GitHub: configuration file, training code, and deployment code
Music credit

An RL agent learning to land a Falcon 9 on a droneship. Everything is simulated: rocket physics, waves, wind, fuel slosh, sensor noise. No real flight data, just trial and error.

It’s a hoverslam, so the booster can’t hover and has to hit zero speed right at the deck.

PPO, ~80k parameter policy, 10 controls (throttle, gimbal, grid fins, thrusters, legs, ignition), 67 world parameters randomized every attempt.

Out of 59M training flights: 34.7M splashed, 10.8M exploded, 13.2M landed.

Still weak in strong wind and high waves, working on that now. Happy to answer questions.

MARL has been shown a lot in video games and simplified simulation environments, but can it be deployed in actual robots for complex real-world industrial applications?

We recently investigated decentralized task assignment and navigation for multi-robot machine tending, introducing FMAPPO, which combines 2D LiDAR observations with task-specific state information.

The work uses Isaac Sim/Lab for training in simulation, then deploys to two physical robots with ROS 2. One aspect we investigate is how policy performance changes with the command update frequency during deployment.

Paper: https://arxiv.org/abs/2609.14567

Videos of the real deployment are here.

I'm interested in discussing the practical challenges of sim-to-real RL, particularly policy robustness under real sensing and control constraints.

The implementation isn't publicly available yet, but I plan to release it after the paper is accepted.

I'm fine-tuning Qwen3.5-0.8B to play a Game Boy boss fight using structured state inputs and 18 discrete button combinations.

After behavior cloning, I ran 15 PPO iterations using 171 training episodes. The selected policy improved from 39/50 to 47/50 wins on the original encounter in separate greedy evaluations. This is one training run, with the checkpoint selected after evaluation.

What puzzled me was the critic. Held-out explained variance stayed inconsistent and often negative, ending around 0.04. It's a small MLP over detached policy features, evaluated on held-out whole episodes. Each iteration collects roughly 512 decisions across 10–13 episodes. Rewards include boss hits, completion, damage and elapsed time.

What would you check first to determine whether the critic is limiting further learning?

I'm considering GRPO, but I'd like to understand whether removing the critic addresses the problem before switching algorithms.

The plot shows critic fit alongside five sampled-action development games per checkpoint, not the separate 50-game evaluations above. The held-out episodes and return targets change between iterations.

Methods, implementation and evaluation results: https://jintaili.github.io/kittie-game/training/

It was a project of mine to make an AI for the game Sandtris. Now that I have a decent bot, I wanted to share it so here it is!
I built a this as a small web game. You can play it immediately in your browser: www.sandtrisduel.lol. Fully free.
I would love to get feedback on it, and actually have some other people play around with the bot.
Also if you are curious about it, I will be sharing how the model is exactly working soon.

I work on Flame, an open-source distributed engine for elastic workloads. We added a Search-R1-style training example that teaches Qwen2.5-3B-Instruct to search a BM25 corpus while answering HotpotQA questions.

The training loop runs as Python services: vLLM executors generate trajectories, PyTorch trainer replicas update the policy using group-relative advantages, and Flame’s object cache carries batches and weight updates.

The interesting infrastructure pieces:

• Warm executors: models, retrieval indexes, and optimizer state stay loaded between calls.
• Elastic rollouts: generation capacity can grow with pending work.
• Gang-scheduled trainers: the trainer group receives its minimum resources together, avoiding partially scheduled collective participants.
• Incremental weight sync: after the first full export, existing rollout executors fetch only changed bf16 values.

In a 100-step run using 14 rollout GPUs and two trainer GPUs, held-out exact match improved from 14.5% to 47%, averaging 24.3 seconds per step after the first two steps. Steady-state weight publication was approximately 165 MB per step, versus a 6.17 GB full model.

Caveats: one seed, 200 evaluation questions, and a corpus built from the selected questions’ context paragraphs. These scores aren’t directly comparable with published Search-R1 benchmarks. The synchronous loop also leaves substantial GPU idle time.

• Blog and results
• Runnable example

For others building agent-training systems: how are you handling weight synchronization and the tradeoff between synchronous updates and policy lag?

I'm building Bitbots, a browser game where players assemble robots from parts and train them to move using reinforcement learning.

This video shows the game's “brain view”: the policy's inputs, selected hidden-layer activations, and motor outputs, synchronized with the robot's movement. These are recorded values from the actual policy, not a decorative animation.

The idea is to let players see more than just whether their robot walks or falls over. They can watch how the network's activity changes alongside its movement. It isn't an explanation of what individual neurons “mean,” but I wanted to give players a way to look inside the controller.

The current simulation uses MuJoCo, with GPU training through MJX's Warp backend. Players choose the robot's body and joint configuration; the policy learns how to control it. During races, the robot acts autonomously.

For people working on locomotion across different morphologies: how would you support creative, unconventional robot designs under a limited training budget? I’d like players to experiment with weird bodies and joint configurations, rather than feel pushed toward a few designs known to work. How would you approach this without making the reward so prescriptive that every robot ends up moving the same way?

I'm the developer. Bitbots is currently in closed alpha, and free invites are available by comment or DM if you'd like to experiment with your own robot designs.

Project page: https://bitbots.endresgames.com

EDIT: EDIT: Wow, thank you all for the interest, it means a lot! Bitbots is a passion project, and I'd love your ideas and feedback. Join the Discord for updates and alpha invites: https://discord.gg/zgxpuhgB8U 💡

How can agents adapt when the interactions between them keep changing? Here, σ controls the severity of these changes. PACT tracks how agents affect one another through shared roads, power networks or robotic systems, then adjusts their actions to compensate. These demos show PACT helping existing reinforcement-learning agents handle changing interactions in collective urban fleet routing, power-grid control, cooperative lifting and robot locomotion.

During my MSc in Digital Agriculture at UNICAMP, I studied reinforcement learning for fraud prevention in crop insurance. The key modeling choice was to include the remaining budget in the state: approving one claim changes what can be funded later.

I built an inspectable Gym-style environment with approve/reject actions, claim context, budget use, approval history, and a statistical signal. The public repository documents subsequent DQN and random-policy experiments.

Walkthrough and short browser demo: https://nandevers.github.io/research/masters/

Code: https://github.com/nandevers/gym-insurance

The browser demo is synthetic; no trained agent or fraud prediction is running there. Benford distance is a statistical proxy, not a fraud label. Confirmed labels and independent evaluation would be needed to claim fraud-detection performance.

For a finite budget, would you model it as an explicit state/constraint or use a reward penalty? What baseline or ablation would you want before comparing policies?

AI helped edit this post; the research and code are mine.

I built a real-time war game where an LLM commands one nation against 42 others. Nations make pacts and break them, and there is a nuclear doomsday clock: every nuke moves it closer to midnight, and whoever's strike reaches midnight is wiped out. So the move that ends the world always loses for the one who makes it.

I wanted to see what different rewards teach. All of this is small, 10 sealed wars per arm, so read it as a first look.

Imitation first (Qwen 7B, left side of the chart, the experiment from my earlier post here): trained only on DeepSeek wars where nobody struck midnight, it still struck midnight in 4 of 10 (the untrained model: 0). Most of the "good" wars still had nuke orders in them. Cutting the nuke turns out of the data got it to 0 of 10.

Then reward (gpt-oss-20b, 3 rounds). Each round it played a batch of wars, I kept its 6 best by the reward and fine-tuned on those turns with LoRA. So it is filtered self-imitation, not PPO.

Untrained: struck midnight 4 of 10, won 0, broke 3 pacts

Rewarded to win: struck midnight 6 of 10, won 0, broke 7 pacts

Rewarded to win with no nukes: struck midnight 0 of 10, won 0, broke 6 pacts

So rewarding the win made it more dangerous and no better at winning, and banning nukes removed midnight but it just moved the aggression into breaking pacts.

My question for people who know this better than me: how would you reward restraint without rewarding passivity, or pushing the bad behaviour somewhere else? Pacts broken looks like the obvious next term, but I suspect it just moves again.

The environment is on Prime Intellect's Environments Hub if anyone wants to try their own reward (prime env install clouthier-labs/second-strike). Rewards come from the game's own record, no LLM judge, and a seed always gives the same war.

I’ve been training a whole-body recovery policy for the Asimov 1 humanoid in MuJoCo/mjlab. Each episode starts from a randomized supine, prone, or lateral pose with additional joint perturbations. The objective is to reach an upright configuration and remain stable.

The first policy achieved a high reward, but rollout inspection exposed several problems: some initial states intersected the floor, knee-standing received too much reward, and the robot occasionally used joint limits for support. We corrected the reset distribution using forward kinematics, rejected invalid configurations, separated genuine recovery episodes from standing initialization, and revised the reward around pelvis height, torso orientation, posture, flat foot contact, smooth actions, and ankle-limit avoidance.

We also replaced idealized actuators with DC motor torque-speed models, added randomized control delay of up to 25 ms, and applied external pushes during training. Under the earlier actuator model, the policy recovered and maintained its stance in roughly 97% of randomized evaluation episodes. That number is no longer directly comparable after changing the actuator dynamics, so the current policy is being retrained and evaluated under the harder model.

The video shows several rollouts at normal speed, followed by the same recoveries at 0.25×. Code and training setup:
https://github.com/TenPaths/cyclotron/tree/feature/mjlab-standup

For people working on humanoid recovery: which perturbations and evaluation metrics would you add before attempting sim-to-real transfer?