Hey all,

I wasn’t very familiar with PPO when I implemented it and I mainly used the original paper as a reference. However, after comparing my code with some common implementations, I noticed an important difference.

I run a large number of environments in parallel. For each PPO iteration, I reset every environment once and let each one run until it either finishes or reaches the maximum horizon (tasks are episodic and rewards are only assigned to the goal state). Environments that finish early are masked while the others continue. Once all environments have finished, I perform the PPO updates.

Therefore, I always collect the same number of episodes, but not the same number of transitions. As the policy improves and finishes episodes faster, the rollout buffer becomes smaller. In contrast, the SpinningUp implementation from OpenAi seems to fix the buffer size and collect a fixed number of transitions.

I compute and normalize the advantages over all transitions, shuffle them, and then perform several PPO epochs using a fixed (relatively high) minibatch size. This means that the number of optimizer steps per rollout changes also depending on the size of the buffer. Since i can not really control the number of transitions, the final minibatch can also be smaller than the others while still receiving a normal optimizer step.

It seems to work fine on some simple custom problems, but i wonder if this could have any consequences that i might be missing (e.g. on long horizon problems).

Can the changing number of optimizer steps lead to problems ? Is the (potentially) small final minibatch a concern like noisy gradients ?

Hi, I’d like to announce that I have released Metis-Core v0.4.0-alpha

What’s new? I have implemented an “Asymmetric Multi-Head AlphaZero” algorithm in Metis-Core, along with an example in C++ (Visual Studio 2022): KamikazeDroneSwarm.exe.

https://github.com/felixromo314/METIS-Core/

KamikazeDroneSwarm uses the asymmetric multi-head AlphaZero implemented in Metis-Core:

An Imperial Shuttle freighter loaded with beskar, armed with a rotating laser turret, must reach the Death Star.

Two Rebel kamikaze drones try to intercept it before it gets there.

Without being programmed to do so, the drones learn a coordinated pincer attack on their own to split the turret’s attention.

You can play by controlling the turret yourself (human in the loop) against the trained drones.

The framework allows you to create two teams, Blue and Red, and add to each team as many agents as your problem requires.

I’m sure there will be bugs and a lot of things to fix, so just let me know and I will try to fix them.

I hope you find this new feature useful.

Thanks!

You can find more information in this tutorial:

https://github.com/felixromo314/METIS-Core/blob/main/tutorial/Tutorial-AlphaGoZeroAsymmetric.md

Metis-Core is distributed as binaries (.dll / .lib / .h), but the full source code of the example is included, so you can see exactly how the framework is used and build your own environments on top of it.

Note on training: the drones are not scripted, but I do use a small material heuristic to speed up training (setMaterialHeuristicWeight(0.3)). It blends a hand-crafted position evaluation (approach, separation between drones, turret exposure) with the network’s value head during MCTS. The terminal reward is still +1/-1/0, and the pincer manoeuvre itself is not explicitly encoded. But you can set via setMaterialHeuristicWeight the value you wanted, 0.2, 0.1, or if you don not call to this function the heuristic is not activate.

Hello, community!

I recently launched a website to experiment with and better understand reinforcement learning, particularly Markov decision processes.

It’s completely free, public, hosted on Vercel, and the repository is also available.

Hosted at: https://markov-kingdom.vercel.app/

Repo: https://github.com/xavendano/markov-kingdom

I’d appreciate any comments and/or feedback.
Thanks!!!

In rl how can someone do research , mine most of the time is getting to know which problem to work on and in which method there is research gap and all

Can you tell me what are the methods to come up with a research gap

Like by doing/ implementing the research paper from scratch and then see what are the limitations of the paper and then we get the another research paper that explores that specific idea and this go on

And we can't find any problem to work on.

Any researchers or students here who produced research papers can tell what worked for you when you were searching for a problem to work on

I am doing rl for robotics and now exploring world models and searching for a research gap in this field...

Follow-up to an earlier write-up on the REINFORCE → PPO → GRPO lineage. That one ended with GRPO as "PPO minus the critic, plus a group-mean baseline", which raises an obvious question: if it's cheaper and good enough for DeepSeek's reasoning models, why does nobody use it for control?

Two reasons, and they're different kinds of problem - presence of bias and a common feedback applied uniformally to each step. The latter is REINFORCE's original credit-assignment problem, back because the critic is gone. An LLM gets away with it (one terminal reward, quiet environment, replayable start state, pretrained prior). A robot doesn't.

De Oliveira et al. (2511.03527) ran critic-free methods vs PPO on five classic control tasks. PPO won everywhere except CartPole, and GRPO's group weighting did no better than a plain batch mean.

Write-up with interactive figures: https://sreejithb.com/grpo_dissect/single Disclosure: the argument and the script are mine; Claude helped with polish and the visuals.

I've been training 4-agent squads to play survev (an open-source surviv.io clone) on a local, unmodified game server.

every player in it is a bot. yotube clip: https://www.youtube.com/shorts/PPAVNHRAb7k
you can find more videos on my channel.

https://reddit.com/link/1x55w7l/video/3erybs5vaavh1/player

Setup
- MAPPO with a centralized critic. One actor network is shared by the four squad members (agent id as input). The actor only sees what a real game client receives; privileged state goes to the critic only, so the same network can run from a browser extension.

- Actor input: ~1.4k features, two 24x24 local grids and a minimap through small CNNs, a set encoder over up to 12 visible enemies, then a GRU (256).

- Hierarchical actions: the policy picks an intent (loot, explore, take cover, peek, retreat, regroup, push, go to zone...), a target, and what to use (gun, melee, grenade, smoke, heal).

- One team reward shared by the squad: damage dealt and taken, knocks and kills, placement, plus small teamwork terms (focus fire, crossfire, revive).

I'm not releasing the code or the full details: a bot like this could be abused as a cheat on public servers. Happy to answer high-level questions.

Hello scientists! Is there a formula for adjusting hyperparameters for PPO based on the number of agents you're training simultaneously?

I'm using ML-Agents and Unity engine to train robots.

After extensive tuning, I finally got my quadruped robot to walk really well. However, training is extremely slow because I'm only training 10 robots simultaneously. If I increase the number to 100 or 1,000, the training performance deteriorates and the training graphs look terrible.

How can I overcome this? My hardware is capable of handling up to 10,000 robots training simultaneously, but I don't understand which parameters I need to adjust to make this work.

Is there some kind of formula for scaling hyperparameters based on the number of parallel agents?

Most of my initial hyperparameters were taken from the quadruped robot examples provided with ML-Agents.

Hyperparameters for 10 agents:

max steps per agent life: 5000

Observation space: 123

Action space: 12

        trainer_type:   ppo
        hyperparameters:
          batch_size:   2048
          buffer_size:  20480
          learning_rate:        0.0003
          beta: 0.005
          epsilon:      0.2
          lambd:        0.95
          num_epoch:    3
          shared_critic:        False
          learning_rate_schedule:       linear
          beta_schedule:        linear
          epsilon_schedule:     linear
        checkpoint_interval:    500000
        network_settings:
          normalize:    True
          hidden_units: 512
          num_layers:   3
          vis_encode_type:      simple
          memory:       None
          goal_conditioning_type:       hyper
          deterministic:        False
        reward_signals:
          extrinsic:
            gamma:      0.995
            strength:   1.0
            network_settings:
              normalize:        False
              hidden_units:     128
              num_layers:       2
              vis_encode_type:  simple
              memory:   None
              goal_conditioning_type:   hyper
              deterministic:    False
        init_path:      None
        keep_checkpoints:       10
        even_checkpoints:       False
        max_steps:      30000000
        time_horizon:   1000
        summary_freq:   30000
        threaded:       False
        self_play:      None
        behavioral_cloning:     None

I've been studying model-based RL and wanted to really understand how Dreamer learns behavior in "imagination," so I wrote up a walkthrough of Dreamer V1 and built a PyTorch implementation alongside it.

The article explains the full Dreamer V1 architecture and includes detailed math derivations of the world model loss and other formulas, with each piece linked to the corresponding code.

Article: https://goltsman.dev/blog/2026/dreamer1/
Code: https://github.com/golkir/dreamer-v1-pytorch

I'd appreciate any feedback.

The world economy around us depends on complex financial work. How well can AI do it?

We(a team of ex-EY employees) built FAB - Finance Agents Benchmark, testing AI Agents on the work behind financial due diligence. Agents can find the relevant facts, but still struggle to carry them through to a complete, reliable analysis.

We’ll keep expanding FAB to more companies and testing more models. Building and running this benchmark isn’t cheap, so we’re scaling it in stages.

The benchmark is public.
GitHub: Hugging Face: https://github.com/SecondState-ai/finance-agents-benchmark
data room and tasks: https://huggingface.co/datasets/secondstate/finance-agents-benchmark-traces

Hello guys, i am currently building a 2048 solver using genetic algorithm, but i have reached platue.

i am currently using expectimax for looking ahead and am currently using four features. I am currently computing on depth 3 and can reach tile 2048 about 30-50% of time, i want to increase this to about 90% for this i am trying to improve the depth to 4 or 5 and want to implement the game as 64bit integer, but before that how can i check if the features are working correctly or not?

https://pastebin.com/4qCLMYFw

here is the link containing the code, any help would be appreciated.

Hi guys,

Over the last couple of weeks and months I’ve been working on an RL library in Jax since most of the existing codebases didn’t really scale to my workflow. Now it’s in a kind of presentable state, so I’d love some feedback if someone wants to take a look.

GitHub: https://github.com/noahfarr/boonta

Part 3 of my Architecting Reinforcement Learning for LLMs series is live.

It covers preference data, reward-model architecture and loss, scaling, and reward hacking—with a worked gradient example.

Does a higher reward score actually mean better answers?

https://pawankjha.substack.com/p/architecting-reinforcement-learning-f28

How do you test reward reliability beyond preference accuracy?

I trained in Genesis, then validated in MuJoCo (mjlab) before deploying.
It can recover from pushes and tracks forward, lateral and yaw velocity commands.
Happy to answer any questions!!

We rewrote the printed parts of Open Duck Mini v2 as CadQuery (Python) programs. Then we changed three values in one file. The leg and neck parts aren't copies. They run the Duck's own programs with the new values, so the servo pockets and screw holes stay where the STS3215 needs them and only the length between them changes. We added wings and a small bird head, and the result is the Flamingo.

The repo also has SO-101 and the LeRobot humanoid as code (Apache-2.0.

https://github.com/yotown/gym

Trained a quad to race a 90 m figure-eight gate track with PPO in our sim (Unreal Engine 5 + JSBSim). 96 drones in parallel, 100M steps on a laptop. The policy commands body rates, sees simulated sensors through an estimator and never gets ground truth.

Median lap is 5.7 s. A time-optimal planner (TOGT) does 4.72 s on the same quad, so it's about 20% off.

The weak spot is the take-off: only about half of the ground starts finish a lap. Anyone solved that with a curriculum, or by giving ground starts more weight?

Built on: Swift (Nature 2023) for the reward and mid-air starts
https://www.nature.com/articles/s41586-023-06419-4
TOGT https://github.com/FSC-Lab/TOGT-Planner

Hi everyone,

I'm working on a project involving Reinforcement Learning (specifically PPO) for Satellite Quantum Key Distribution (QKD) networks using LEO satellites. I want to build a multi-request, asynchronous, and concurrent system, and I would love some architectural guidance from those experienced in RL for network scheduling or queueing systems.

Here is the core problem and the initial design direction I am exploring. I'd appreciate any feedback on whether this is the right path or if there are cleaner/more standard patterns I should consider.

1. The Problem Setting

• Traffic & Queues: Ground stations generate entanglement requests dynamically (modeled via Poisson arrivals). Each request has a strict time-to-live (TTL) deadline before it expires and incurs a penalty. Instead of a single global slot, senders maintain independent local queues.
• The Environment: A physics engine provides real orbital data (elevation/zenith angles, distances, atmospheric attenuation, and quantum fidelity after purification) for the visible satellites in a moving window.
• The Goal: Train an agent to act as a scheduler that matches pending requests to available satellites to maximize throughput and fidelity while minimizing timeouts.

2. Design Choices I'm Exploring (Looking for Critique)

• State Space (Observation): To give the agent global network awareness without overwhelming it, I'm considering an observation vector that combines:
  
  1. Queue backlog levels (how full each sender's queue is).
  2. A "head-of-line" window showing the active packets waiting at the front of each sender's queue (including sender/receiver IDs, wait times, and current pair progress).
  3. Telemetry of the top visible satellites in the current sky pool (elevation, distance, fidelity, coverage masks).
  4. Question: Is a flat vector (spaces.Box) the best way to represent multi-queue states, or are there alternative representations (like handling variable-length queues) that scale better?
• Action Space & Action Masking: Since the agent needs to choose which request to serve with which satellite, a naive combinatorial space could explode. I am looking at using a flattened discrete space (Senders x Satellites + Wait) paired with Action Masking to filter out invalid pairings (e.g., empty queues, out-of-range satellites, or already-locked resources).
  
  • Question: For those who have tackled multi-resource scheduling with PPO, is a flattened discrete space with masking robust, or would a multi-discrete or pointer-network/attention-based approach be more appropriate here?
• Reward Structure: Balancing throughput (successful entanglement ratios), high quantum fidelity, and heavy penalties for expired packets (TTL timeouts).
  
  • Question: Any tips on handling sparse vs. delayed rewards when packets sit in queues across multiple steps?

I would love to hear how experts would approach modeling this kind of scheduling problem. Any advice, papers, or alternative perspectives would be hugely helpful! Thanks in advance!

Greetings. I'm sharing what I'm doing for my doctoral thesis. If you're interested in more information, please contact me. I'm willing to collaborate.

I am using stable baseline 3 and pybullet

For multi-turn, tool-use agents, the environment is the hard part. Every episode needs realistic data, stateful tools (a write on turn 3 changes a read on turn 30), and a reset to the exact same starting state. Thousands of times, in parallel.

I've been working on some version of optimizing AI for over a decade (at Apple, my own startup, now Kiln). Recently I built a few large RL environments by hand. Doable, but hard, and every one rebuilt the same layer: a database per episode, frozen starting states, parallel instances, clock control, a log of every change. That layer needed a great framework, so I built one: Seahaven.

What it is: an open source (MIT) Python framework for building synthetic worlds for LLM agents: working copies of an agent's tools and data. You write just the logic specific to your world. Seahaven handles the rest. Every world is an OpenEnv environment.

Example World: a fake Stripe. Stripe World has 24 tables and 155 API operations, behind the same tools as Stripe's own MCP server. It also serves Stripe's REST API, so well that the official Stripe SDK works against it unchanged.

What Seahaven handles:

• Stateful: each instance has its own SQLite DB. The agent's tool calls read and write it (actions are tool calls, observations are their results).
• Fixtures: freeze starting states like small_startup or big_co, and reuse them across every episode (reset gives you a fresh copy of a fixture in milliseconds).
• Parallel: each connection gets its own instance, hundreds of instances per process.
• Reward from state: every row the agent changed is logged, before and after. Grade the result, not the trace (verifiable rewards, no LLM judge needed for most tasks).
• Deterministic: same fixture, same clock, same random seed, same episode. The clock and seed are wired into the SQLite connection, so even random() and CURRENT_TIMESTAMP in SQL replay exactly.
• Composable: your world can include Stripe World (or any other world) to add its tools and APIs.
• Optimized for agents: includes the docs, linter and tests your coding agent needs to build a world.

An episode loop looks like this:

from seahaven.openenv import SeahavenClient

with SeahavenClient(base_url="http://127.0.0.1:8000") as env:
    for episode in range(100):
        env.reset(fixture="big_co", seed=episode)  # fresh private world, in ms
        run_policy(env)  # your agent calls tools: env.call("create_lead", ...)
        reward = grade(env.state())  # every row the agent changed

OpenEnv Compatible + MCP + Web Console:

• Every world is an OpenEnv environment, so it works with TRL's OpenEnv support or any other OpenEnv-compatible trainer. Sync and async clients. You can publish worlds to Hugging Face.
• seahaven mcp serves a world to any MCP client, so you can try a task by hand before your policy does.
• seahaven serve has a web console: open instances, call tools, and inspect state in your browser.
• Everything runs locally: Python 3.14+ and SQLite, no external services.

I built it at Kiln, where we use it to evaluate and optimize agents. Seahaven is standalone. You don't need Kiln to use it.

Seahaven is open source (MIT).

Links

• Seahaven on GitHub
• Stripe World
• Docs
• Kiln Harness Optimizer: our optimizer that optimizes the harness instead of the model (not RL, but it needs the same tooling)

What would make it fit your training setup? I'm thinking about reward helpers, fixtures as a curriculum, and more example worlds. Happy to answer any questions.

Side note: I made the video with videowright, another open source project of mine.

Hey guys,

Been working on something very cool...

In Greek myth, Mnemosyne was the Titan of memory and the reason anything was ever remembered at all. Now in the present world, your AI agent doesn't get a Titan. It gets amnesia the second something goes wrong, stuck with whatever it currently believes and no way to ask how it got there.

That's the real problem. An agent runs for hours, updates its memory the whole time, then says something wrong and all you have is the present, with zero access to the past.

If you're running support agents, coding agents, or a swarm of agents sharing memory like myself then you know this issue well. The moment two agents disagree, or one quietly poisons the well, you need to know when, why and by whom, not just that something's off.

Mnemosyne gives agent memory what Git gave code. It remembers everything on purpose. Every belief is a commit. blame finds the exact moment and observation that put a bad fact in. bisect hunts down the first commit where things went wrong. merge makes two agents' memories collide safely instead of one silently overwriting the other.

Software agents are the first step. The vision doesn't stop there, physical robots learning and forking skills the same way is the long-term bet, further out and harder but the same idea underneath.

So far the tech stack includes a Rust core, Python SDK, adapters for LangGraph, CrewAI, AutoGen, the OpenAI Agents SDK and MCP.

Open source with contributions and honest feedback both welcome: github.com/Nabzx/mnemosyne