System and method for recognition and annotation of facial expressions

Abstract

The innovation disclosed and claimed herein, in aspects thereof, comprises systems and methods of identifying AUs and emotion categories in images. The systems and methods utilized a set of images that include facial images of people. The systems and methods analyze the facial images to determine AUs and facial color due to facial blood flow variations that are indicative of an emotion category. In aspects, the analysis can include Gabor transforms to determine the AUs, AU intensities and emotion categories. In other aspects, the systems and method can include color variance analysis to determine the AUs, AU intensities and emotion categories. In further aspects, the analysis can include convolutional neural networks that are trained to determine the AUs, emotion categories and their intensities.

What is claimed is: 
     
         1 . A computer-implemented method for analyzing an image to determine Action Unit (AU) and AU intensity using color features in the image, comprising:
 identifying changes defining transition of an AU from inactive to active, wherein the changes are selected from the group consisting of chromaticity, hue and saturation, and luminance; and   applying a Gabor transform to the identified transition changes to gain invariance to a timing of the identified transition changes during a facial expression.   
     
     
         2 . The method of  claim 1 , further comprising:
 maintaining, in memory a plurality of trained color features data associated with an AU and/or AU intensity;   receiving the image to be analyzed; and   for each received image:
 determining configural color features of a face in the image due to facial muscle action; and 
 determining none, one or more AU values for the image by comparing the determined configural color features to the plurality of trained color feature data to determine presence of the determined configural color features in one of more of the plurality of trained color feature data. 
   
     
     
         3 . The method of  claim 1 , further comprising:
 analyzing a plurality of faces in images or video frames to determine kernel or face space for a plurality of AU values and AU intensity values, wherein each kernel or face space is associated with at least one AU value and at least one AU intensity value, and wherein each kernel or face space is linear or non-linearly separable to other kernel and face space data, wherein the kernel or face space includes functional color space feature data; and   training a deep neural network with a set of results of the analyzing of the plurality of faces in images or video frames.   
     
     
         4 . The method of  claim 3 , wherein the functional color space feature data is determined by performing a discriminant functional learning analysis on color images each derived from a given image of the plurality of faces in images or video frames. 
     
     
         5 . The method of  claim 1 , further comprising:
 identifying changes defining transition of an emotion from not present to present that are color transitions in a face in a video sequence resulting from blood flow in the face.   
     
     
         6 . The method of  claim 1 , further comprising modifying the analyzed image by:
 modifying at least one of the group consisting of chromaticity, hue and saturation, and luminance of the analyzed image to yield an appearance of a determined expression of an emotion or of a second AU.   
     
     
         7 . The method of  claim 1 , further comprising modifying the analyzed image by:
 modifying at least one of the group consisting of chromaticity, hue and saturation, and luminance to increase or decrease an intensity of a determined emotion or the AU to change a perception of the determined emotion or the AU.   
     
     
         8 . A computer-implemented method for analyzing an image to determine AU and AU intensity in face images, comprising:
 training a deep neural network with a set of training images having faces, wherein the deep neural network is trained to identify AU in the face images; and   identifying local loss and global loss with the deep neural network in the face images to determine AUs in the face images.   
     
     
         9 . The method of  claim 8 , wherein the deep neural network includes comprising:
 detecting a plurality of landmark points with a first part of the deep neural network, wherein the landmark points facilitate calculating the global loss; and   detecting local image changes with a second part of the deep neural network, wherein the landmark points are normalized and concatenated to embed location information of the landmark points into the deep neural network.   
     
     
         10 . The method of  claim 9 , wherein the deep neural network includes a plurality of layers for identifying landmarks in the image and a second plurality of layers to recognize AUs in the face images. 
     
     
         11 . The method of  claim 8 , further comprising:
 determining a neutral face of an image in the face images, the neutral face having color; and   modifying the color of the image of the neutral face to yield an appearance of a determined expression of an emotion or of a second AU.   
     
     
         12 . The method of  claim 8 , further comprising:
 determining a neutral face of an image in the face images, the neutral face characterized by the group consisting of chromaticity, hue and saturation, and luminance; and   modifying at least one of the group consisting of chromaticity, hue and saturation, and luminance of the image of the neutral face to yield an appearance of a determined expression of an emotion or of a second AU.   
     
     
         13 . The method of  claim 8 , further comprising:
 determining an emotion or AU in an image of the face images, the image having color,   determining an intensity of the determined emotion or AU; and   modifying the color of the image to increase or decrease the intensity of the determined emotion or AU to change a perception of the emotion or the AU.   
     
     
         14 . The method of  claim 8 , further comprising:
 determining an emotion or AU in an image of the face images, the image characterized by the group consisting of chromaticity, hue and saturation, and luminance,   determining an intensity of the determined emotion or AU; and   modifying at least one of the group consisting of chromaticity, hue and saturation, and luminance of the image to increase or decrease the intensity of the determined emotion or AU to change a perception of the emotion or the AU.   
     
     
         15 . A non-transitory computer readable medium storing computer-executable instructions that when executed by a computer cause the computer to perform a method for analyzing an image to determine AU and AU intensity in face images, the method comprising:
 training a deep neural network with a set of training images having faces, wherein the deep neural network is trained to identify AU in the face images; and   identifying local loss and global loss with the deep neural network in the face images to determine AUs in the face images.   
     
     
         16 . The non-transitory computer readable medium of  claim 15 , wherein the method further comprises:
 detecting a plurality of landmark points with a first part of the deep neural network, wherein the landmark points facilitate calculating the global loss; and   detecting local image changes with a second part of the deep neural network, wherein the landmark points are normalized and concatenated to embed location information of the landmark points into the deep neural network.   
     
     
         17 . The non-transitory computer readable medium of  claim 16 , wherein the deep neural network includes a plurality of layers for identifying landmarks in the image and a second plurality of layers to recognize AUs in the face images. 
     
     
         18 . The non-transitory computer readable medium of  claim 15 , wherein the method further comprises:
 determining a neutral face of an image in the face images, the neutral face having color; and   modifying the color of the image of the neutral face to yield an appearance of a determined expression of an emotion or of a second AU.   
     
     
         19 . The non-transitory computer readable medium of  claim 15 , wherein the method further comprises:
 determining a neutral face of an image in the face images, the neutral face characterized by the group consisting of chromaticity, hue and saturation, and luminance; and   modifying at least one of the group consisting of chromaticity, hue and saturation, and luminance of the image of the neutral face to yield an appearance of a determined expression of an emotion or of a second AU.   
     
     
         20 . The non-transitory computer readable medium of  claim 15 , wherein the method further comprises:
 determining an emotion or AU in an image of the face images, the image having color,   determining an intensity of the determined emotion or AU; and   
       modifying the color of the image to increase or decrease the intensity of the determined emotion or AU to change a perception of the emotion or the AU.