How a Boston Dynamics robot sees the world around it

Boston Dynamics has shown how one of its robots sees and perceives the world around it.

The Atlas robot must find a shiny metal part in dim lighting, figure out where it is, carefully grab it and precisely place it in the groove. If you miss by a couple of centimeters, the part will get stuck or fall. And if something goes wrong, Atlas can find the fallen part on the floor and try again.

The secret is in the three-layer vision system. First, the cameras determine what is around in general - racks, obstacles, hazards on the floor. The system highlights key points on industrial structures: green ones show general contours, red ones - the internal structure of shelves and niches. This helps the robot understand where exactly something can be placed.

Then 3D perception turns on. Atlas calculates its position relative to each rack, comparing the found points with an internal map of the environment. If some corners are not visible or the robot is at an angle, the system relies on many internal points between the shelf dividers. Even when the engineer in the video moves the container, Atlas instantly notices the change and adjusts its plans.

SuperTracker is the brain of the object tracking system. It combines data from the robot’s joint sensors with the image from the cameras. When Atlas grabs an object, the system knows exactly where it should be at each movement. If the object disappears from view or slips out of the grip, the robot immediately understands.

To determine the pose of objects, Atlas uses a clever “render-and-compare” algorithm. The system is trained on synthetic data and can work with hundreds of industrial parts, having their 3D models. The robot creates several hypotheses about the object’s position, checks them for consistency and chooses the best option.

But the most important thing is the perfect calibration between what Atlas sees and how it moves. In the video, you can see how the digital model of the robot is precisely superimposed on the real image - the arms, legs and torso are exactly where the robot thinks. This accuracy is achieved through complex calibration procedures that take into account assembly inaccuracies and even temperature changes.