“Self-aware” robots learn from watching themselves

Robots can become "truly useful" if they learn to take care of themselves – something they can apparently achieve by watching their movements through a camera.

Robots can teach themselves about their bodies and movements by observing themselves, according to a new study from engineers at Columbia University.

Using cameras to collect this information, robots can not only plan their actions but also adapt to body damage, researchers said.

"Like humans learning to dance by watching their mirror reflection, robots now use raw video to build kinematic self-awareness," said study lead author Yuhang Hu, a doctoral student at Columbia University's Creative Machines Lab.

"Our goal is a robot that understands its own body, adapts to damage, and learns new skills without constant human programming," Hu said.

Most robots first learn movement through simulations, but researchers believe they should continue learning in the physical world.

“The better and more realistic the simulator, the easier it is for the robot to transition from simulation to reality,” said Hod Lipson, head of the Creative Machines Lab.

However, building an effective simulator requires skilled engineers. To bypass this challenge, the researchers taught a robot to create its own simulator simply by watching its movements through a camera.

“This ability not only saves engineering effort but also allows the simulation to evolve with the robot as it undergoes wear, damage, and adaptation,” Lipson said.

The researchers developed a way for robots to autonomously model their own 3D shapes using a single regular 2D camera. The breakthrough was achieved by three “brain-mimicking” AI systems known as deep neural networks.

The system can also identify changes in the robot’s body and help it adjust its motions to recover from simulated damage.

"Imagine a robot vacuum or a personal assistant bot that notices its arm is bent after bumping into furniture," Hu said.

"Instead of breaking down or needing repair, it watches itself, adjusts how it moves and keeps working. This could make home robots more reliable – no constant reprogramming required.

A different scenario imagined by the researchers involves a robot arm getting knocked out of alignment in a car factory. Instead of halting production, it could watch itself, tweak its movements, and get back to work, increasing efficiency and reducing costs.

"This adaptability could make manufacturing more resilient," Hu said.

Lipson added that “we humans cannot afford to constantly baby these robots, repair broken parts, and adjust performance.”

“Robots need to learn to take care of themselves if they are going to become truly useful,” he said.

The results, published in the journal Nature Machine Intelligence, are part of a series of projects carried out by the Columbia team over the last two decades to improve robots’ use of cameras.