Abstract
Catching requires generalizing to novel projectiles and adapting in real time to changing visual observations, unseen object properties, and diverse contact dynamics. We present Catch It All!, a sim-to-real catching framework for humanoid robots that generalizes to unseen objects with different geometry, deformability, and elasticity, for both underhand and overhand catches, using only one egocentric camera.
Grasps that adapt to the object's pose and shape come from a fingertip-to-surface distance reward. Safety-aware curricula let the policy act at high frequency near hardware limits, including wrist torque limits as low as 5 N·m. The policy is trained in simulation and deployed zero-shot on a Unitree G1, where it catches 9 novel objects unseen in training: low-rebound plushes, semi-deformable toys and chip bags, and rigid-elastic objects that bounce in unpredicatble ways.
- Real-world success
- 38.9% underhand, 36.1% overhand
- Test objects
- 9, none seen in training
- Sensing
- One head-mounted ZED2i, egocentric.
- Hand
- 20-DoF House of Dextra, 600g
- Teleop baseline
- 6.8% (3 of 44 tosses) @ 15Hz
Submission Video
Egocentric Perspective
All perception is purely egocentric with no third person camera. The robot sees only through an overhead ZED2i stereo camera mounted approximately 0.4m above the neck at −45°. No external motion capture or third-person tracking is used at test time. The videos below show examples of successful catches, demonstrating the robot's real visual input during catching trials.
Overhand Catches
Successful real-world overhand catching examples across objects, played at 0.33× speed.
Underhand Catches
Successful real-world underhand catching examples across objects, played at 0.33× speed.
Real-World Results
We validate the generalization capability of our method through real-world underhand and overhand catches using objects unseen during training. For each real object, for each toss type (underhand or overhand), we perform 1 long (>1.8 m), 2 medium (1.4 m to 1.8 m), and 1 short (1.0 m to 1.4 m) toss. Shorter tosses require fast reaction time, while longer tosses require effective tracking.
| Object class | Underhand | Overhand |
|---|---|---|
| Plush panda, turtle, sandbag | 50.0% | 33.3% |
| Deformable Takis, strawberry, pineapple | 25.0% | 41.7% |
| Rigid-elastic green bowl, ellipsoid, bear | 41.7% | 33.3% |
| All objects | 38.9% | 36.1% |
Failure Examples
Below are examples of failure cases of our method, including reaching failure, delayed interception, and failure to grasp.