Talk at CMU: Spatiotemporal Modeling of Facial Expressions

Maja Pantic
Delft University of Technology

Abstract:

Machine understanding of facial expressions could revolutionize human-machine interaction technologies and fields as diverse as security, behavioral science, medicine, and education. Consequently, computer-based recognition of facial expressions has become an active research area.

Most systems for automatic analysis of facial expressions attempt to recognize a small set of "universal" emotions such as happiness and anger. Recent psychological studies claim, however, that facial expression interpretation in terms of emotions is culture dependent and may even be person dependent. To allow for rich and sometimes subtle shadings of emotion that humans recognize in a facial expression, context-dependent (e.g., user- and task-dependent) recognition of emotions from images of faces is needed.

We propose a case-based reasoning system capable of classifying facial expressions (given in terms of facial muscle actions) into the emotion categories learned from the user. The utilized case base is a dynamic, incrementally self-organizing event-content-addressable memory that allows fact retrieval and evaluation of encountered events based upon the user preferences and the generalizations formed from prior input.

Three systems for automatic recognition of facial muscle actions (i.e., Action Units, AUs) in face video will be presented as well. One of these uses temporal templates as the data representation and a combined k-Nearest-Neighbor and rule-based classifier as the recognition engine. Temporal templates are 2D representations of motion history, that is, they picture where and when motion in the input image sequence has occurred. The other two systems exploit particle filtering to track facial characteristic points in an input face video. One of those systems performs facial-behavior temporal-dynamics recognition in face-profile image sequences using temporal rules. The other employs Support Vector Machines to encode 20 AUs occurring alone or in combination in an input nearly-frontal view face video.

The systems have been trained and tested using two different databases: the Cohn-Kanade facial expression database and our own web-based MMI facial expression database. The recognition results achieved by the proposed systems demonstrated rather high concurrent validity with human coding.

Bio: Maja (Maya) Pantic received the MS and PhD degrees in computer science from Delft University of Technology, in 1997 and 2001. She is currently an associate professor at the Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, The Netherlands, where she is doing research in the area of machine analysis of human interactive cues for achieving a natural, multimodal human-machine interaction. She is the (co-) principal investigator in three large, national, ongoing projects in the area of multimodal, affective, human-machine interaction. She was the organizer and co-organizer of various meetings and symposia on Automatic Facial Expression Analysis and Synthesis and she is the Associate Editor of the IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics, responsible for computer vision and its applications to human-computer interaction. In 2002, for her research on Facial Information For Advanced Interface, she received Innovational Research Award of Dutch Scientific Organization as one of the 7 best young scientists in exact sciences in the Netherlands. She is currently a visiting professor at the Robotics Institute, Carnegie Mellon University. She has published more than 40 technical papers in the areas of machine analysis of facial expressions and emotions, artificial intelligence, and human-computer interaction and has served on the program committee of several conferences in these areas. For more information, please see http://mmi.tudelft.nl/~maja/