MIT engineers create household robots with “common sense”

Researchers have trained household robots to rely on common sense when faced with obstacles.

Robots are great at mimicking humans but could struggle in situations that were not pre-programmed. For example, when pushed aside while serving a dish, a robot may restart the task from the beginning rather than regroup and act based on the circumstances.

That’s because imitation learning is a mainstream approach to training household robots, according to researchers from the Massachusetts Institute of Technology (MIT). Such robots “blindly” mimic human motions rather than address the situation based on logic.

To tackle the problem, scientists have developed a method that connects robot motion data with the “common sense knowledge” of large language models (LLMs), MIT reported.

“LLMs have a way to tell you how to do each step of a task in natural language. A human’s continuous demonstration is the embodiment of those steps in physical space,” the study’s co-author Yanwei Wang said.

“And we wanted to connect the two so that a robot would automatically know what stage it is in a task and be able to replan and recover on its own,” Wang said.

Researchers demonstrated their approach in experiments with a robotic hand that was tasked to scoop marbles from one bowl and pour them into another. It managed to do so despite intentional efforts to disrupt it.

Experimenters pushed and nudged the robot and even knocked out marbles from its spoon, but each time, the bot was able to recover and move on instead of starting from the beginning. It also made sure it would not carry on if there were no marbles left on its spoon.

According to MIT, researchers first trained the robot by physically guiding it through the task and then had an LLM model to divide it into subtasks. They used their new algorithm to connect the LLM’s defined subtasks with the robot’s motion trajectory data.

The algorithm automatically learned to identify the robot’s current subtask based on its physical position or state, allowing it to continue with the task even when disturbed.

“With our method, when the robot is making mistakes, we don’t need to ask humans to program or give extra demonstrations of how to recover from failures,” Wang said.

This eliminates a “very tedious” job of programming specific responses to every potential error and will lead the way to “robust robot behavior that can do complex tasks, despite external perturbations,” he said.

The results of the study will be presented in detail at the International Conference on Learning Representations (ICLR) in May, according to MIT.