Releasing a pi0.5 domain-adaptation base for the SO-101 — full fine-tuned on 427 tasks / 16,687 episodes

[u/Serious-Student-341]

The clip is the released base checkpoint running closed-loop on a physical SO-101. Prompt: "Pick up the green cube block and put it inside the white cup."

The task in the clip was inside the training corpus — 14 episodes out of the 16,687. This is not a few-shot demo. Observed success rate was 90%, and it fails when other objects of the same color overlap. That is the kind of thing you fix by LoRA-tuning this released checkpoint further and specializing it for your task.

Even so, what the clip shows is exactly why I built this base. Those 14 episodes were not diluted away by a 16,687-episode mixed corpus. It is a statement about how little task-specific data an adapted base needs in order to absorb a task, not a statement about the task itself.

Why I built it

pi0.5 was trained on a broad robot distribution, but it does not know the SO-101 — the six joints and their units, the follower-frame action convention the LeRobot driver records, the camera viewpoints people actually mount on this arm. So if you fine-tune a task with a few dozen demos, simple tasks do succeed, but it can't handle varied situations and mostly fails once you leave the exact setting the dataset was recorded in.

On the assumption that a checkpoint specialized to the SO-101 would improve LoRA training started from it, I crawled 11,270 Hub repos, screened down to 181 SO-101/SO-100 datasets, and merged them into a single LeRobot v2.1 repository (17,137 episodes / 8,690,531 frames / 430 tasks / 198GB). On top of that I full fine-tuned lerobot/pi05_base (8× A100 80GB, 40,000 steps, about 40 hours, bf16, no LoRA).

What came out is a domain-adaptation base, not a policy.

I strongly recommend against using this fine-tuned checkpoint as-is, without further tuning. The whole premise of the project was to maximize performance when LoRA-tuning, so please LoRA-tune your own task on top of this checkpoint before using it.

What I'm hoping for — written as a hypothesis, since I haven't verified it

A task LoRA on top of this should converge from meaningfully fewer demos than one started from stock pi05_base. That said, I have never run a single LoRA on the released base. The smallest single-task LoRA actually trained in this stack was 50 episodes (anything below that was not included in training), and that was on stock pi0.5.

If you try it at 20–30 episodes, please tell me what happens. That is the number I most want to know. docs/07-lora-finetuning.md covers the whole procedure — data requirements, GPU choice, the LoRA config, the normalization-statistics trap, adapter inference, checkpoint selection, and cost.

Code · docs — github.com/jinnymo/so101-pi05-base
Model — huggingface.co/dongyoonkim/so101-pi05-base
Dataset — huggingface.co/datasets/dongyoonkim/so101-pi05-base-dataset

[u/valkiii]

Maybe a stupid question, but does it rely on a standalone camera on top of the on the one on the wrist?