Humanoids might know how you feel and be 95% correct, AI research shows

Robots are learning to be more emotionally intelligent and are reaching accuracy levels as high as almost flawlessly determining what people feel – an achievement likely to transform social robotics as we know it.

Researchers have developed an artificial intelligence (AI) system that allows robots to distinguish between human emotions based on their facial expressions. 95% of the time, they get it right, provided it happens in a controlled environment.

Research conducted by scientists from Universitat Politècnica de Catalunya (UPC) explores how robots can interpret human facial gestures using a computer vision technique called facial emotion recognition (FER). It detects emotions such as happiness, sadness, fear, anger, or surprise.

Lead scientists say that humanoids that are able to recognise, adapt, and react to these feelings accordingly are the key component to developing HRI – a field known as human-robot interactions.

“The potential to provide robots with emotional intelligence with the goal of improving the intuitiveness, sincerity, and naturalness of HRI remains an interesting challenge... The robot’s capacity to sense and comprehend human emotions is essential to achieving this,” say the authors of the study.

According ot them, the need for humanoids to be “emotionally intelligent” arises from their great potential to be companions in hospitals, care homes, and schools. For example, in healthcare, emotionally aware robots could help monitor mental well-being or support patients with dementia and autism. In education, tutors could detect frustration or confusion and adjust their approach.

But how did the researchers measure human feelings and robot emotional intelligence?

In order for robots to “learn” how some emotions appear on human faces, they were shown different sets of public images of people. The process did not involve scanning the faces of people who would have met the robot “in person.”

Images, which in the study are described as datasets, showed people displaying various emotions: happiness, sadness, anger, fear, or surprise.

Each image was labelled so that the AI knows what emotion it represents.

The study used three major datasets:

1. FER2013 – more than 30,000 photos of faces found online, resized to 48×48 pixels, showing seven emotions (angry, disgust, fear, happy, sad, surprise, and neutral).

2. RAF-DB (Real-world Affective Faces Database) – about 30,000 real-life facial photos collected from the internet, showing a wide range of ages, ethnicities, lighting, and postures.

“The images in this database exhibit significant variations in participants’ age, gender, ethnicity, head postures, lighting circumstances, occlusions (e.g., spectacles, facial hair, or self-occlusion), and post-processing techniques (e.g., multiple filters and specialized effects),” the report says.

3. CK+ – 981 images taken in controlled lab conditions, showing actors performing emotional expressions.

As the robots were trained to recognise emotions represented in these groups of images, the CNN (convolutional neural network) learned to recognise facial patterns when people displayed certain emotions. For example, a thing that gave away some emotions was the curvature of a smile or the furrow of a brow.

The researchers were also rotating and flipping images in order for the CNN to become better at recognising expressions from different angles.

The model was then tested on unseen images to see how accurately it could recognise emotions. It achieved 95% accuracy on CK+, 83% on RAF-DB, and 64% on FER2013, showing that the AI performs best in clean, well-lit conditions but struggles with real-world diversity.

Why didn’t AI successfully recognise emotions in more diverse faces or realistic environments?

The experiment results showed that AI’s accuracy declined when CNN was trying to define emotions in real-world conditions due to lighting, head movement, and cultural differences in expression.

“The images in the RAF-DB database exhibit significant variations in participants’ age, gender, ethnicity, head postures, lighting circumstances, occlusions (e.g., spectacles, facial hair, or self-occlusion), and post-processing techniques,” the report states.

The research team plans to improve the AI training system by including larger images from more diverse datasets, as well as using 3D imaging and multimodal sensors that combine facial data with speech and physiological cues.

A the discussion around empathy and other emotions in machines remains rather philosophical and theoretical, researchers from this study argue that their work marks a step toward a future where technology is not only capable of “seeing” emotions but also understanding how humans feel.

Unlock more exclusive Cybernews content on YouTube.