Key Idea
Given a single view of an object, ZeroBot uses an image-to-3D model to generate a mesh and rapidly learns to manipulate it in simulation, without human demonstrations or policy pre-training.
Abstract
We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning.
Framework Overview
Our ZeroBot framework has three main stages. First, the robot observes an object from a single view. It generates an object mesh from the RGB image using an image-to-3D model, and aligns it with the depth image to recover scale. Second, the generated mesh is used for simulation. The task is specified by providing a goal pose, which defines a dense reward function. Massively parallel RL is used to learn a policy which can manipulate the object into that goal pose. To enable rapid training, we use a contact action space which selects a high-value contact state using the learned value function. During deployment, the robot moves into this state using motion planning and executes policy actions.
Action Space
An overview of our contact action space on a multi-stage nudge-and-grasp task. (1) First we uniformly sample contact locations from the surface of the generated mesh (a subset of contacts is shown here for visual clarity). Each point shows a sampled gripper pose for nudging (top row) or grasping (bottom row). (2) Then invalid contact states are eliminated, e.g. those colliding with the rest of the object, and the learned value function from RL is used to compute the value of each valid state. Purple states have lower value, and yellow states have higher value. (3) At execution, the max-value state is chosen. (4) The robot moves into contact using motion planning, and then (5) executes policy actions. Meshes shown have been approximated for fast collision checking.
Examples of Generated Meshes
Examples of meshes given by real2sim methods. ZeroBot uses an image-to-3D model which can generate a custom mesh to best explain the input image, even for an object which it has never seen before. Although the robot cannot directly observe the far side of the bowl or the whole tumbler handle, it can still learn to manipulate it in simulation and successfully transfer this zero-shot to the real world, using the powerful visual prior of image-to-3D models to generate complete meshes from partial observations. We also find that for non-prehensile tasks such as rolling a Coca-Cola can onto a ramp, generating smooth and complete geometry is crucial for task success.
The meshes below are interactive: you can click and drag to rotate them, and scroll to zoom in.