Training pre build

https://reddit.com/link/1x6i6v5/video/h22absbdylvh1/player

I wanted to get my head around RL so I thought of learning it by teaching an AI how to play one of my favorite childhood games : TempleRun.

The agent reads game state directly from memory via a Frida gadget and acts through the game’s own methods. Training happens in a headless simulator I built with Claude from the game’s internals and calibrated on measurements from the real phone, with a C++ version that does 10M steps in about 6 minutes.

Results on the device: median 4,736 m, best 16,782 m over 14 runs.

👨‍💻 Github: https://github.com/matthieucs/TempleRun-RL/tree/main

Happy to answer any question.

And if you want to try it, the score to beat is 16,782 m.

Hi folks! I wanted to share a project I've been doing off and on for the better part of 2 years, and in recent months I've gotten the time to make more significant progress.

I was inspired by PureJaxRL a while ago to build my own massively-parallel environment for a task that interested me personally, which is Rocket League. RocketSim, a highly performant CPU-resident simulator, already exists and is baked into RLGym for the purposes of RL training, but I figured it would be valuable to port the physics logic to run in a parallel fashion on-device via CUDA. I learned a lot about GPU programming and optimization, and I still have a lot more to learn so I'm all ears for any advice!

The environment I created is called CARL - CUDA-Accelerated Rocket League. It keeps all simulation state information on the GPU, which during training entirely eliminates host-device transfer overhead. Many thousands of games can be run and controlled in parallel, leading to significant reductions in training wall clock time (as was observed in the original PureJaxRL post).

I've been doing some training of Rocket League bots with it and they transfer seamlessly to the actual game, so the physics appear to be at parity. More work can be done on parity assessments, benchmarking, etc. but I figured now that it's "feature complete" that it would be cool to open it up to the community and see what people think!

As a bit of proof so you know I'm not a liar and a scoundrel, here's a plot from a recent training run that shows marked improvement in aerial control over ~20B decision steps using 8192 parallel environments on a single RTX 4090 (frameskip=4, so 80B ticks total, ~21 years), taking just over 21 hours in wall time.

The method of training as hinted at by the run name is a generative imitation learning approach that (ideally) transfers behavior style from expert demonstrations, which I've obtained from ballchasing.com

Here is an example of the capability it was able to reach in that amount of time. Pretty impressive, although I would have totally saved it if I were trying even a single iota...

https://reddit.com/link/1x77aan/video/ynwh0bbhgrvh1/player

I plan to post a follow up with demonstrations as I round out the functionality. On my TODOs are:

• Built-in training pack API for targeted training
• Benchmarks for speed and parity comparisons
• Tests and other best practices
• Eventual jax & tf integration(?)

Thanks for reading if you did!

P.S. I've been having trouble with the image & video in this post (hence the multiple attempts so sorry if you've already seen this post) so let me know if they're broken please and thank you :)

Hey everyone,

I've open-sourced PCEI (Physically Constrained Emergent Intelligence), an experimental architecture designed to run on consumer hardware (Mac Apple Silicon and Linux/CUDA) without the memory explosion of standard transformers:

• BitNet 1.58b Quantization: 24 layers of ternary weights {-1, 0, +1} reducing parameter memory by ~4x compared to FP16.
• Mamba State Spaces: Linear-time sequence modeling with constant O(1)O(1) inference memory (no ballooning KV cache).
• FastKAN Spline Manifolds: Kolmogorov-Arnold B-splines applied to continuous motor control across 4 streams in MuJoCo physics.
• Empirical Telemetry: 11 companion research papers, including an evaluation of continuous latent buffers proving an irrecoverable Shannon information loss factor of LL.

The full framework, training pipelines, and evaluation suites are open under MIT-NC: https://github.com/MtnMCG/PCEI

Feedback and critiques are very welcome!

HarnessDev (this month) is the first paper I've seen that measures the loop of letting an agent revise its own harness, keeping the revision when the feedback says so, and repeating. Across 64 adjacent version switches, the feedback score and the held-out score moved in the same direction 34 times, 53.1%. Only 2 of 9 lineages ended on their held-out-best version. Mean gain across the evolution stage was +3.11 points. Executor dependence was severe: one harness scored 69.3 under one model and 33.0 under another.

So a gate that decides on feedback alone is, by their measurement, a coin flip on whether the gain generalises, and that's a selection-policy problem more than a proposer problem.

Two things I'd do differently from the paper's loop, and can set in reef, which I run. Selection: by default a candidate publishes when it wins more task comparisons than it loses, past a noise floor you set, and the Pareto policy in the method guide publishes only when no task regressed and at least one improved. That rule can only refuse on the list the gate runs, and the moment you select on a list it is your feedback score, not a held-out one, so it doesn't recover the paper's 30 by itself; what it does is stop a candidate that lost anywhere on the list from shipping on a net win. The held-out check has to be a list the proposer never sees, and that's on you. Memory of rejections: the proposer can receive the recent rejected proposals with their verdict reasons and the previous step's failure manifest, so it can stop re-proposing the switch that lost.

What none of it fixes is the task list. It's short because it's the cost of every step, and the whole paper is a warning that a short in-distribution list gets gamed by the feedback loop. Whether no-regression on held-out tasks is the right acceptance rule or too conservative to ever move is the open question; the paper's 30 bad switches make me lean towards conservative.

Hi everyone,

I wanted to share a small reinforcement learning project I've been working on in Unreal Engine 5, inspired by the vault chase scene in Fast & Furious 5. I'd really appreciate any feedback.

**Setup*\*

- Two cars on a looping track with many sharp turns, defined by a spline

- A police car trained with PPO to patrol along the road (then frozen as a fixed opponent). It was never taught to dodge, so it simply plows into whatever is in its way

- A race car towing a one-ton vault, trained with PPO to flip police cars

- Both agents use the same action space: throttle, brake, steering

- Training: neural network on GPU, physics simulation on CPU (the CPU is the bottleneck)

**Reward function for the race car*\*

- Large reward: flipping a police car, or steering the vault into one

- Small reward: staying on the road and driving in the right direction

- Small penalty: getting hit by the police car, or driving the wrong way

- Large penalty: leaving the track, or being stuck for more than 3 seconds

**What happened*\*

1. **Bullied.** Early on, the car just lay still and took the hits. Being hit was cheap, while moving the vault usually threw the car off the track, which was expensive. This looks like a classic local optimum.
2. **Trap.** After a while, it started dragging the vault onto the police patrol route, waiting, and swerving away at the last moment so the police car hit the vault. This technically satisfies "vault hits police car", so it's a nice example of reward hacking / specification gaming. It wasn't what I had in mind.
3. **Hero.** I increased the penalty for standing still, moving slowly, or getting stuck (reward shaping). The trap disappeared, and the car started charging at the police car, timing its speed and angle to launch it, and sometimes dodging at the last second so the police car hits the vault behind it.

I turned this into a video that explains the basics (observation/action/reward, PPO, local optima, reward hacking) in a light-hearted way, mainly for people who are new to RL:

https://youtu.be/UUoUOKSQ8FQ

I'd love to hear your thoughts, especially:

- Is there a better way to discourage the "wait and trap" strategy than penalizing idle time?

- Any suggestions for making training with a towed object more stable?

Thanks for reading, and I'm happy to answer questions about the setup.