New AI Transfers Emotional Cues from Speech to Facial Expressions
Accepted for CVPR 2026, C-MET transfers emotional cues from speech to facial expressions—even for subtle emotions beyond its training data.
JooHyeon Heo
Abstract Talking face generation has gained significant attention as a core application of generative models. To enhance the expressiveness and realism of synthesized videos, emotion editing in talking face video plays a crucial role. However, existing approaches often limit expressive flexibility and struggle to generate extended emotions. Label-based methods represent emotions with discrete categories, which fail to capture a wide range of emotions. Audio-based methods can leverage emotionally rich speech signals - and even benefit from expressive text-to-speech (TTS) synthesis - but they fail to express the target emotions because emotions and linguistic contents are entangled in emotional speeches. Images-based methods, on the other hand, rely on target reference images to guide emotion transfer, yet they require high-quality frontal views and face challenges in acquiring reference data for extended emotions (eg, sarcasm). To address these limitations, we propose Cross-Modal Emotion Transfer (C-MET), a novel approach that generates facial expressions based on speeches by modeling emotion semantic vectors between speech and visual feature spaces. C-MET leverages a large-scale pretrained audio encoder and a disentangled facial expression encoder to learn emotion semantic vectors that represent the difference between two different emotional embeddings across modalities. Extensive experiments on the MEAD and CREMA-D datasets demonstrate that our method improves emotion accuracy by 14% over state-of-the-art methods, while generating expressive talking face videos - even for unseen extended emotions. Sarcasm, encouragement, and empathy are often conveyed as much through tone of voice as through words themselves. Reproducing these subtle emotional cues in AI-generated talking face videos, however, remains a challenge. Researchers at UNIST have developed a new artificial intelligence (AI) module that transfers emotional information from speech to facial expressions, enabling talking face videos to express a wider range of emotions—including those the system was never explicitly trained to recognize. Led by Professor Taehwan Kim of UNIST Artificial Intelligence Graduate School (AIGS), the team introduced Cross-Modal Emotion Transfer (C-MET) , a modular AI framework that edits facial expressions using emotional cues extracted from speech. Unlike existing approaches, C-MET does not require reference images of the target expression, making it easier to generate natural facial expressions from speech alone. Most existing methods rely on predefined emotion labels or reference images to guide facial expression generation. These approaches often struggle to capture subtle emotional states because spoken language carries both linguistic meaning and emotional information, making the two difficult to disentangle. Image-based approaches also require carefully prepared examples of the desired expression, which are often unavailable for more nuanced emotions such as sarcasm. C-MET addresses these challenges by learning how changes in emotional speech correspond to changes in facial expressions. Rather than matching fixed emotion categories, the system models relationships between speech and facial features, allowing emotional cues to be transferred naturally while preserving both the speaker's identity and spoken content. Because it learns relationships between emotions rather than predefined labels, C-MET can also generate facial expressions for emotional states that were not included in the training data. The researchers evaluated C-MET by integrating it into two state-of-the-art talking face generation models. When incorporated into EDTalk, the module improved emotion accuracy on the MEAD benchmark from 41.99% to 55.91%—an increase of more than 14 percentage points. Applying C-MET to another talking face generation model, PD-FGC, also improved emotion accuracy from 33.36% to 36.82%, while reducing inference time, demonstrating that the module can be readily integrated into different talking face generation frameworks. “Our work overcomes a practical limitation of existing approaches by enabling facial expressions to be edited directly from speech without requiring reference images,” said Professor Kim. "Because C-MET functions as a modular component, it can be incorporated into a wide range of existing systems for applications such as virtual humans, educational avatars, conversational AI, and film and video post-production." The study was led by Chanhyuk Choi of AIGS, as first author. The paper has been accepted for presentation at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2026, one of the world's leading conferences in computer vision. Additional materials, including the source code, model checkpoints, and demonstration videos, are available on the C-MET project page. Journal Reference Chanhyuk Choi, Taesoo Kim, Donggyu Lee, et al. , "Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video," '26 CVPR