Automated methods and systems for identifying and characterizing face tracks in video

Abstract

The present document is directed to methods and systems that identify and characterize face tracks in one or more videos that include frames that contain images of one or more human faces. In certain implementations, values for attributes, such as age, ethnicity, and gender, are assigned to face-containing subimages identified in frames of the image. The occurrence or presence of a face in a sequence of frames is identified, by comparing attributes and location and dimension parameters assigned to each occurrence of the face in a face-containing subimage within a frame, as a face track that represents a four-dimensional tube or cylinder in space time. Attributes are assigned to each face track based on attributes assigned to the occurrences of subimages of the face in frames within the face track.

The invention claimed is: 
     
       1. A video-processing system comprising:
 a distributed computing system that includes multiple computer systems, each having one or more processors, one or more memories, one or more mass-storage devices, and one or more network interconnections; 
 a still-image face-detection and attribute-assignment subsystem that identifies face-containing subimages within a digital image and assigns values for image attributes for the identified face-containing subimages and that comprises
 an input-image memory for receiving an input image provided by one or more of the multiple computer systems, 
 a set of first-level feature detectors, provided by one or more of the multiple computer systems, that detect face-containing subimages within the input image and, for each face-containing subimage, generate a set of normalized regions, 
 a set of second-level feature detectors that generate a set of feature vectors from each normalized region, 
 a set of attribute classifiers that each outputs an attribute value and associated probability, and 
 a controller that
 submits the input image to the first-level feature detectors in order to generate sets of normalized regions for each face-containing subimage in the input image, 
 submits each set of normalized regions to generate a set of feature vectors, and 
 submits subsets of feature vectors to each attribute classifier in an ordered list of attribute classifiers to assign attributes to each face-containing subimage; and 
 
 
 a controller that
 receives a reference to a video, and 
 processes each chunk of frames in the video to
 identify and characterize face-containing subimages within each frame of the chunk by applying the still-image face-detection and attribute-assignment subsystem to the frame, and 
 identify and characterize face tracks in the video, each face track comprising a sequence of frames in which subimages of a particular face occur, the face track characterized by attribute values generated from the attribute values associated with each identified subimage containing the particular face. 
 
 
 
     
     
       2. The video-processing system of  claim 1  wherein the set of first-level feature detectors comprises:
 a set of multiple face detectors that each receives a reference to the input image and returns parameters that specify a subimage that frames a human face in the input image; and 
 a set of multiple face-subregions detectors that each receives a reference to the input image and a set of parameters that specify a face-containing subimage and that each returns a set of parameters that species a sub-subimage, or region, that contains an image of a facial feature. 
 
     
     
       3. The video-processing system of  claim 2  wherein a set of normalized regions corresponding to a face-containing subimage is generated by one or more of:
 performing rotation-matrix and translation-matrix operations to reorient the face-containing subimage and the corresponding regions that each contains an image of a facial feature to produce a canonical region arrangement; and 
 applying one or more perspective transformations to corresponding regions that each contains a perspective-distorted image of a facial feature. 
 
     
     
       4. The video-processing system of  claim 1  wherein the set of second-level feature detectors that generate a set of feature vectors from each normalized region further comprise:
 multiple feature detectors, including feature detectors selected from one or more of scale-invariant-feature-transform (“SIFT”) feature detectors, histogram-of-gradients (“HoG”) feature detectors, and linear-binary-patterns (“LBP”) feature detectors. 
 
     
     
       5. The video-processing system of  claim 1  wherein each feature vector in the set of feature vectors generated by the second-level feature detectors includes encodings of multiple numeric values. 
     
     
       6. The video-processing system of  claim 1  wherein the set of attribute classifiers includes:
 one or more individual support vector machine (“SVM”) classifiers that each returns, in response to input of one or more feature vectors, one of two possible attribute values; and 
 one or more aggregate classifiers that each includes two or more individual support vector machine (“SVM”) classifiers and that each returns, in response to input of one or more feature vectors, one of multiple possible attribute values. 
 
     
     
       7. The video-processing system of  claim 1  wherein an identified face comprises one or more face-containing subimages within video frames, attributes assigned to which occupying a volume of less than a threshold size within an attribute-value space. 
     
     
       8. The video-processing system of  claim 7  wherein an identified face is associated with a face ID and with position-specifying and size-specifying parameters for each frame of each video in which the identified face occurs. 
     
     
       9. The video-processing system of  claim 7  wherein an identified face is associated with a face profile, the face profile associated with a face-profile ID and multiple derived facial attributes. 
     
     
       10. The video-processing system of  claim 7  wherein the face tracks are stored, within the video-processing system, in relational database tables that include:
 a local-face-tracks table; 
 a face-profiles table; 
 an identified-faces table; 
 a track-profiles table; and 
 a global-tracks table. 
 
     
     
       11. The video-processing system of  claim 10  wherein the face tracks that correspond to a particular individual or group of individuals, stored in the video-processing system, are used to automatically identify, by the video-processing system, human-associated events that have been captured in one or more videos. 
     
     
       12. The video-processing system of  claim 11  wherein the video-processing system processes a frame f by:
 identifying face-containing subimages in the frame; and 
 for each face-containing subimage s in the frame f,
 assigning attributes to the face-containing subimage s; and 
 generating a unique face-profile ID and face ID for the face-containing subimage s, rendering the face-containing subimage identified. 
 
 
     
     
       13. The video-processing system of  claim 7  wherein the controller:
 receives references to multiple videos, video metadata, and a database handle; and 
 for each video v referenced by a received reference,
 for each chunk of frames c in the video v,
 for each frame f in the chunk of frames c,
 processes the frame f; and 
 
 processes the chunk of frames c; and 
 
 processes the video v. 
 
 
     
     
       14. The video-processing system of  claim 13  wherein the video-processing system processes a chunk of frames c by:
 for each identified face in in each frame of the chunk of frames c,
 when there is an active local track corresponding to the identified face,
 adding the identified face and frame containing the identified face to the active local track corresponding to the identified face; and 
 
 when there is no active local track corresponding to the identified face,
 generating a new active local track and adding the identified face and frame containing the identified face to the new active local track. 
 
 
 
     
     
       15. The video-processing system of  claim 1  wherein face tracks include local face tracks and global face tracks. 
     
     
       16. The video-processing system of  claim 15  where local face tracks include:
 single, continuous face tracks that each spans a sequence of video frames, the single, continuous face track comprising a face-containing subimage within each video frame corresponding to an identified face; 
 single face tracks that each includes two of more single, continuous face tracks, each pair of temporally adjacent single, continuous face tracks separated from one another by a set of non-face-containing frames of a cardinality less than a first threshold number of frames; and 
 stitched face tracks that each includes two or more single-continuous-face-track and/or single-face-track component face tracks, each pair of temporally adjacent component face tracks separated from one another by a set of non-face-containing frames of a cardinality less than a second threshold number of frames, wherein the second threshold number of frames is greater than the first threshold number of frames. 
 
     
     
       17. The video-processing system of  claim 16  where local face tracks include two or more local face tracks, including local face tracks from two or more different videos and pairs of temporally adjacent local face tracks separated from one another by a set of non-face-containing frames of a cardinality less than a third threshold number of frames, wherein the third threshold number of frames is greater than the second threshold number of frames. 
     
     
       18. The video-processing system of  claim 15  wherein each face track is associated with a face-track ID, a track-profile ID, and length-specifying and status-specifying parameters. 
     
     
       19. The video-processing system of  claim 15  wherein each face track is associated with a face-track profile, the face-track profile associated with a face-track-profile ID and multiple derived facial attributes.