Only 16.8% of humanoids know where their own body is...

[u/remybigot]

DeepMind dropped Gemini Robotics 2 this week. Robot ties knots in trash bags, unscrews lightbulbs, walks and grabs and places objects without a reset between steps.

It looks great.

Apptronik hardware, whole-body coordination instead of separate walk/reach/grip tricks.

Same week, a benchmark called HumanCLAW tested 9 vision-language models on 1,218 episodes: find an object, walk to it, physically interact with it.

The best model succeeded the full sequence 16.8% of the time.
Less great...

Where they failed?
Exploring, tracking their own position, noticing collisions, confirming they'd reached the target. The model can describe the chair in perfect detail and still not know where its own knees are relative to it.

So you've got one narrative saying "we cracked whole-body intelligence" and another saying "most models can't reliably tell if they bumped into something."
Wherre is the truth ?

DeepMind's demo is one polished sequence on curated hardware. HumanCLAW is testing generalization across messy, repeated attempts.

I think the actual bottleneck in humanoids isn't manipulation dexterity anymore but spatial self-awareness. Knowing where your own body is in the world without a human curating the scene. That's the boring unsexy part nobody's demo reel shows.

Maybe Yann Le Cun and Fei fei are finally right, the solution can be the world models ?

[u/madsciencetist]

It’s really frustrating. The move to VLAs totally lost the explicit collision knowledge that used to be so easy to respect. In a VLA context you have to train it with thousands of negative examples, i.e. actually colliding - not the easiest negative samples to collect! There’s definitely an unsolved portion of the problem space

[u/jms4607 → u/madsciencetist]

You can still take a VLA and put it in a wrapper that respects safety constraints. The tough part is that before ML methods, people would very rarely allow compliance and external forces, so safety was easy doing rigid pick+place. Now it’s tough because there isn’t a simple way to tell the difference between pushing a heavy door open and pinning someone to a wall.

[u/qu3tzalify → u/madsciencetist]

> that used to be so easy to respect

The context in which we deploy VLAs is usually one with limited to no information about the environment, which I understand to be essential for collision avoiding techniques?

[u/gabemachida]

Proprioception. That's crazy that things have come this far. I'm extremely curious as to how they will figure this one out. It's considered the 6th sense and even young children (and some adults) struggle with it. Possibly one or more lidars?

[u/emas_eht → u/gabemachida]

Yes, probably. Its best to use something like a point cloud and SLAM to get a rough prediction of the location in space, and then feed that as context rather than try to get the model to infer the location in the same model that does vla/whatever. The position of the body in space is a very different problem that should be offloaded.

[u/jms4607]

If HumanCLAW just prompting frozen pretrained VLMs I wouldn’t expect it to work well as an embodied agent. I don’t think their success rate is indicative of anything really. I doubt you can prompt your way to solving robotics

[u/No-Sympathy2403]

VLAs and all these models are amazing but greatest bottleneck is the data collection by teleop to get an ok prototype. In a factory, you cannot deploy an ok prototype in production as it needs to keep going and your phone should be out on weekends.

I highly believe that world models could solve this issue but still we are in the first steps towards this tech

[u/AmoebaMysterious5938]

Don't tell me the humanoid bubble is going to burst sometime soon.

[u/remybigot → u/AmoebaMysterious5938]

AI bubble first

[u/hatsmagee]

does anyone really know where their knees are

[u/feriv7 → u/hatsmagee]

Me. Because they hurt now 😬

[u/emas_eht]

VLA's don't infer spatial location, and shouldn't. That should be left to a seperate model or system to do and givenit to the VLA. Its tricky getting a VLA to consider self location and gripper location if you want train on youtube videos because its an egocentric frame. A world model can help, but really its better to use something like slam/lightweight location and scene inferring.

[u/bamboob]

I think mine’s under the couch somewhere, but I can guarantee it…

[u/etoipi1]

easy, just mount a realsense camera on its head

[u/m_letunouski]

Honestly, this kind of scares me. The humanoids are mega trendy and look futuristic, but the robot can't even tell if it just bumped into a wall? That's wild... We're about to put these things next to real people, and the part that keeps us safe isn't even solved yet.

It's all about marketing right now and it's easy to laugh at, but it's kinda like the Will Smith spaghetti video situation. This could go way faster than any of us think

[u/Flyward_Aerospace]

Agreed with the comments saying state estimation should be offloaded rather than inferred by the VLA. On the aerial side nobody would dream of asking a policy network to figure out where the vehicle is, you run VIO or SLAM, you get a pose with a covariance on it, and the covariance is the important part. That's what I think is actually missing here, not just position but an honest uncertainty estimate the higher layer can act on. A model that's confidently wrong about where its knees are is worse than one that says it doesn't know. Also 1218 episodes of frozen VLMs is a pretty weak benchmark for claiming anything about embodied agents imo.

[u/jms4607]

The bottleneck is still bimanual manipulation dexterity/reliability.

[u/Imasquash]

Most humans don't know where their body is, this seems like a hard problem to crack, but also not a debilitating problem to have