Multi-modal detection engine of sentiment and demographic characteristics for social media videos

Abstract

A system and method for determining a sentiment, a gender and an age group of a subject in a video while the video is being played back. The video is separated into visual data and audio data, the video data is passed to a video processing pipeline and the audio data is passed to both an acoustic processing pipeline and a textual processing pipeline. The system and method performs, in parallel, a video feature extraction process in the video processing pipeline, an acoustic feature extraction process in the acoustic processing pipeline, and a textual feature extraction process in the textual processing pipeline. The system and method combines a resulting visual feature vector, acoustic feature vector, and a textual feature vector into a single feature vector, and determines the sentiment, the gender and the age group of the subject by applying the single feature vector to a machine learning model.

The invention claimed is: 
     
       1. A system determining a sentiment, a gender and an age group of a subject in a video, the system comprising:
 a video playback device; 
 a display device; and 
 a computer system having circuitry, 
 the circuitry configured to
 while the video is being played back by the video playback device on the display device, separate the video into visual data and audio data, 
 pass the visual data to a video processing pipeline and pass the audio data to both an acoustic processing pipeline and a textual processing pipeline, 
 perform, in parallel, a video feature extraction process in the video processing pipeline to obtain a visual feature vector, an acoustic feature extraction process in the acoustic processing pipeline to obtain an acoustic feature vector, and a textual feature extraction process in the textual processing pipeline to obtain a textual feature vector, 
 combine the visual feature vector, the acoustic feature vector, and the textual feature vector into a single feature vector, and 
 determine the sentiment, the gender and the age group of the subject by applying the single feature vector to a machine learning model, 
 wherein the circuitry extracts the visual features, for a plurality of frames of the video, by:
 detecting the face of the subject, 
 segmenting the detected face from the rest of the video frame, 
 detecting a frontal face of the detected face, 
 cropping and scaling the frontal face to a predetermined number of pixels, 
 extracting local descriptors from each frame which include key points, orientation, and moments, 
 extracting global descriptors which include optical flow for the motion of each pixel between consecutive frames, and 
 generating a histogram of the optical flow. 
 
 
 
     
     
       2. The system of  claim 1 , wherein the age group is a multi-class of four age groups,
 wherein the machine learning model is a plurality of support vector machines including one support vector machine for each of the four age groups, and 
 wherein the circuitry is configured to determine the sentiment, the gender and the age group of the subject by applying the single feature vector to each of the plurality of support vector machines and selecting a result from among the support vector machines for each of the four age groups. 
 
     
     
       3. The system of  claim 1 , wherein the circuitry including the machine learning model is trained using a plurality of visual feature vectors extracted from a plurality of videos of different subjects to classify each subject according to positive or negative sentiment, a plurality of acoustic feature vectors extracted from audio signals associated with the videos, and a plurality of textual feature vectors extracted from text derived from the audio signals. 
     
     
       4. The system of  claim 3 , wherein the circuitry is further configured to
 extract visual features from a portion of each of the plurality of videos to obtain the plurality of visual feature vectors, acoustic features from a portion of an audio signal associated with the videos to obtain the plurality of acoustic feature vectors, and textual features from text for each utterance in the audio signal to obtain the plurality of textual feature vectors, and 
 combine the visual feature vector, acoustic feature vector, and textual feature vector for each of the plurality of videos into a plurality of respective single feature vectors. 
 
     
     
       5. The system of  claim 1 , wherein the circuitry extracts the visual features including determining an angle and a magnitude of each pixel for two consecutive frames of the video. 
     
     
       6. The system of  claim 1 , wherein the circuitry extracts the audio features including extraction of prosodic features and spectral features of the audio signal. 
     
     
       7. The system of  claim 1 , wherein the circuitry extracts the textual features including a preprocessing operation of normalizing Arabic characters that have different forms. 
     
     
       8. The system of  claim 3 , wherein the circuitry including the machine learning model is trained by labeling the age group as one of multiple age group classes including young adults, young-middle-aged, middle-aged, and senior. 
     
     
       9. The system of  claim 1 , wherein the circuitry that extracts the textual features includes a skip-grams neural network that generates a textual feature vector for each utterance. 
     
     
       10. The system of  claim 1 , wherein the circuitry performs reduction of the single feature vector to a vector of reduced dimension, and
 wherein the circuitry determines the sentiment, the gender and the age group of the subject by applying the reduced feature vector to the machine learning model. 
 
     
     
       11. A method for determining a sentiment, a gender and an age group of a subject in a video, the method performed by a computer system having circuitry,
 the method, performed by the circuitry as a background process, comprises: 
 while the video is being played back, 
 separating the video into visual data and audio data; 
 passing the visual data to a video processing pipeline and passing the audio data to both an acoustic processing pipeline and a textual processing pipeline; 
 performing, in parallel, a video feature extraction process in the video processing pipeline to obtain a visual feature vector, an acoustic feature extraction process in the acoustic processing pipeline to obtain an acoustic feature vector, and a textual feature extraction process in the textual processing pipeline to obtain a textual feature vector; 
 combining the visual feature vector, acoustic feature vector, and textual feature vector into a single feature vector; 
 determining the sentiment, the gender and the age group of the subject by applying the single feature vector to a machine learning model; and 
 outputting the sentiment, the gender and the age group of the subject, 
 wherein the extracting, by the circuitry, the visual features includes, for a plurality of frames of the video, 
 detecting the face of the subject; 
 segmenting the detected face from the rest of the video frame; 
 detecting a frontal face of the detected face; 
 cropping and scaling the frontal face to a predetermined number of pixels; 
 extracting local descriptors from each frame which include key points, orientation, and moments; 
 extracting global descriptors which include optical flow for the motion of each pixel between consecutive frames; and 
 generating a histogram of the optical flow. 
 
     
     
       12. The method of  claim 11 , wherein the age group is a multi-class of four age groups,
 wherein the machine learning model is a plurality of support vector machines including one support vector machine for each of the four age group and 
 wherein the method, performed by the circuitry, further comprises 
 determining the sentiment, the gender and the age group of the subject by applying the single feature vector to each of the plurality of support vector machines; and 
 selecting a result from among the support vector machines for each of the four age groups. 
 
     
     
       13. The method of  claim 11 , further comprising training the machine learning model including
 extracting, by the circuitry, a plurality of visual feature vectors from a plurality of videos of different subjects to classify each subject according to positive or negative sentiment; 
 extracting, by the circuitry, a plurality of acoustic feature vectors from audio signals associated with the videos; and 
 extracting, by the circuitry, a plurality of textual feature vectors from text derived from the audio signals. 
 
     
     
       14. The method of  claim 13 , further comprising
 extracting, by the circuitry, visual features from a portion of each of the plurality of videos to obtain the plurality of visual feature vectors, acoustic features from a portion of an audio signal associated with the videos to obtain the plurality of acoustic feature vectors, and textual features from text for each utterance in the audio signal to obtain the plurality of textual feature vectors; and 
 combining, by the circuitry, the visual feature vector, the acoustic feature vector, and the textual feature vector for each of the plurality of videos into a plurality of respective single feature vectors. 
 
     
     
       15. The method of  claim 11 , wherein the extracting the visual features includes determining, by the circuitry, an angle and a magnitude of each pixel for two consecutive frames of the video. 
     
     
       16. The method of  claim 11 , wherein the extracting the audio features includes extracting, by the circuitry, of prosodic features and spectral features of the audio signal. 
     
     
       17. The method of  claim 11 , wherein the extracting the textual features includes normalizing, by the circuitry, Arabic characters that have different forms. 
     
     
       18. The method of  claim 13 , wherein the training of the machine learning model includes labeling the age group as one of multiple age group classes including young adults, young-middle-aged, middle-aged, and senior.