Mobile face capture and image processing system and method

Abstract

Image processing procedures include receiving at least two side view images of a face of a user. In other aspects, side view images are warped and blended into an output image of a face of a user as if viewed from a virtual point of view. In further aspects, a virtual video is produced in real time of output images from a video feed of side view images.

1 . An image processing method, comprising: 
 receiving at least two side view images of a face of a user;    warping and blending the side view images into an output image of the face of the user as if viewed from a virtual point of view; and    producing a virtual video in real time of output images from a video feed of side view images.    
   
   
       2 . The method of  claim 1 , further comprising: 
 accessing a three-dimensional closed mesh model of points corresponding to salient facial feature points; and    warping and blending the side view images by texture mapping the side view images to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of side view images.    
   
   
       3 . The method of  claim 2 , further comprising instantiating a mesh model for an individual user by obtaining scaling and deformation transformations.  
   
   
       4 . The method of  claim 3 , further comprising obtaining scaling and deformation transformations by choosing special points by hand on a digital frontal and profile photo.  
   
   
       5 . The method of  claim 3 , further comprising obtaining scaling and deformation transformations by choosing special points from the two side images of a neutral facial expression of the user captured by imaging components of a head mounted display worn by the user, and enabling the user wearing the head mounted display to make adjustments that apply various scaling and deformation transformations while viewing a resulting output image rendered by the head mounted display.  
   
   
       6 . The method of  claim 2 , further comprising: 
 receiving a selection of a virtual point of view from which to view a three-dimensional model of the face of the user; and    rendering the three-dimensional model from the virtual point of view based on the selection.    
   
   
       7 . The method of  claim 6 , further comprising applying a lighting model that determines how the three-dimensional model appears to be illuminated.  
   
   
       8 . The method of  claim 2 , further comprising dynamically fitting parameters of the mesh model by optimizing similarity between a three-dimensional model rendered from a virtual point of view corresponding to an actual point of view of a side view image while using the parameters and the side view image.  
   
   
       9 . The method of  claim 8 , further comprising fitting the parameters via hill-climbing so that incremental dynamic updates can be made to the model parameters for sequentially observed side video frames.  
   
   
       10 . The method of  claim 2 , further comprising training a morphable model for dynamic use on a population of users, including capturing side views and a frontal view of a diverse set of training speakers.  
   
   
       11 . The method of  claim 10 , further comprising: 
 hand labeling mesh points for a diverse set of faces and multiframe video recording; and    performing principal components analysis to obtain a minimum spanning dimensionality.    
   
   
       12 . The method of  claim 2 , further comprising texture mapping triangles of the mesh model to stored face data as needed to fill in un-imaged patches.  
   
   
       13 . The method of  claim 1 , further comprising: 
 accessing transformation tables for the side view images, wherein the transformation tables define rules for interpolating regions of the side view images into side portions of the output image;    warping the side view images based on the transformation tables, thereby producing the side portions of the output image; and    blending the side portions of the output image, thereby producing the output image.    
   
   
       14 . The method of  claim 13 , further comprising creating the transformation tables by projecting a grid pattern onto a human face at least as if from the virtual point of view and mapping polygons of left and right calibration face images to corresponding polygons of the grid pattern.  
   
   
       15 . The method of  claim 13 , wherein warping the side view images includes reconstructing coordinates in side portions of the output image by accessing corresponding locations in the transformation tables and retrieving pixels in the side view images using interpolation.  
   
   
       16 . The method of  claim 1 , wherein receiving at least two side view images includes receiving side view images captured via at least two imaging components of a head mounted display worn by the user, said imaging components attached to said head mounted display unit and thereby obtaining fixed positions and orientations relative to the face of the user and adapted to receive at least two side views of the face of the user;  
   
   
       17 . The method of  claim 1 , further comprising linearly smoothing the output image in order to smooth intensity across a vertical midline of the face.  
   
   
       18 . An apparatus, comprising: 
 a head mounted display unit worn by a first user, the display unit rendering to the first user an output image of a face of a second user virtually interacting with the first user in a collaborative, virtual environment, wherein the output image has been formed, based on offset view images of the face of the second user, such that the face of the second user appears as if viewed from a virtual point of view; and    an input port receiving at least one of the following: (a) offset view images of the face of the second user; (b) user-specific scaling and deformation transformations specific to the second user; (c) position of the face of the second user in a common coordinate system of the collaborative, virtual environment; (d) a three-dimensional model of the face of the second user; (e) a selection of a virtual point of view from which to render the three-dimensional model of the face of the second user; and (f) an output image of the face of the second user.    
   
   
       19 . The apparatus of  claim 18 , further comprising: 
 an array of at least two imaging components having fixed positions and orientations relative to a face of the first user and adapted to receive at least two offset views of the face of the first user; and    an output port transmitting at least one of the following: (a) offset view images of the face of the first user; (b) user-specific scaling and deformation transformations specific to the first user; (c) position of the face of the first user in the common coordinate system of the collaborative, virtual environment within which the first user and the second user virtually interact; (d) a three-dimensional model of the face of the first user; (e) a selection of a virtual point of view from which to render the three-dimensional model of the face of the first user; and (f) an output image of the face of the first user.    
   
   
       20 . The apparatus of  claim 19 , further comprising an image processing module accessing a three-dimensional closed mesh model of points corresponding to salient facial feature points of offset view images of the face of the first user, and combining the offset view images of the face of the first user by texture mapping the offset view images of the face of the first user to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images of the face of the first user, thereby forming the three-dimensional model of the face of the first user.  
   
   
       21 . The apparatus of  claim 20 , wherein said image processing module is further adapted to select a virtual point of view from which to view the three-dimensional model of the face of the first user based on positions of faces of the first user and the second user in a common coordinate system of the collaborative environment, and to render the three-dimensional model of the face of the first user from the virtual point of view, thereby forming the output image of the face of the first user.  
   
   
       22 . The apparatus of  claim 21 , wherein said image processing module is adapted to linearly smooth the output image in order to smooth intensity across a vertical midline of the face of the first user.  
   
   
       23 . The apparatus of  claim 21 , wherein said image processing module is adapted to apply a lighting model that determines how the three-dimensional model of the face of the first user appears to be illuminated.  
   
   
       24 . The apparatus of  claim 18 , further comprising an image processing module adapted to select a virtual point of view from which to view the three-dimensional model of the face of the second user based on positions of faces of the first user and the second user in a common coordinate system of the collaborative environment, and to render the three-dimensional model of the face of the second user from the virtual point of view, thereby forming the output image of the face of the second user.  
   
   
       25 . The apparatus of  claim 24 , wherein said imaging module is further adapted to access a three-dimensional closed mesh model of points corresponding to salient facial feature points of offset view images of the face of the second user, and combining the offset view images of the face of the second user by texture mapping the offset view images of the face of the second user to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images of the face of the second user, thereby forming the three-dimensional model of the face of the second user.  
   
   
       26 . The apparatus of  claim 24 , wherein said image processing module is further adapted to apply a lighting model that determines how the three-dimensional model appears to be illuminated.  
   
   
       27 . The apparatus of  claim 18 , further comprising an image processing module adapted to linearly smooth the output image in order to smooth intensity across a vertical midline of the face.  
   
   
       28 . An apparatus, comprising: 
 an array of at least two imaging components having fixed positions and orientations relative to a face of a first user and adapted to receive at least two offset views of the face of the first user; and    an output port transmitting at least one of the following: (a) offset view images of the face of the first user; (b) user-specific scaling and deformation transformations specific to the first user; (c) position of the face of the first user in a common coordinate system of a collaborative, virtual environment within which the first user and a second user virtually interact; (d) a three-dimensional model of the face of the first user; (e) a selection of a virtual point of view from which to render the three-dimensional model of the face of the first user; and (f) an output image of the face of the first user, wherein the output image of the face of the first user has been formed by combining offset view images of the face of the first user into an output image of the face of the first user as if viewed from a virtual point of view.    
   
   
       29 . The apparatus of  claim 28 , further comprising an image processing module accessing a three-dimensional closed mesh model of points corresponding to salient facial feature points of offset view images of the face of the first user, and combining the offset view images of the face of the first user by texture mapping the offset view images of the face of the first user to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images of the face of the first user, thereby forming the three-dimensional model of the face of the first user.  
   
   
       30 . The apparatus of  claim 29 , wherein said image processing module is further adapted to select a virtual point of view from which to view the three-dimensional model of the face of the first user based on positions of faces of the first user and the second user in a common coordinate system of the collaborative environment, and to render the three-dimensional model of the face of the first user from the virtual point of view, thereby forming the output image of the face of the first user.  
   
   
       31 . The apparatus of  claim 30 , wherein said image processing module is adapted to linearly smooth the output image in order to smooth intensity across a vertical midline of the face of the first user.  
   
   
       32 . The apparatus of  claim 29 , wherein said image processing module is adapted to apply a lighting model that determines how the three-dimensional model of the face of the first user appears to be illuminated.  
   
   
       33 . Computer software, comprising: 
 first instructions receiving at least two offset view images of a contoured structure;    second instructions forming, from the offset view images, an output image of the contoured structure as if viewed from a virtual point of view.    
   
   
       34 . The computer software of  claim 33 , wherein said second instructions are adapted to recognize feature points of the contoured structure in the offset view images, to access a three-dimensional closed mesh model of feature points similar to the recognized feature points, and to texture map the offset view images to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images, thereby forming a three-dimensional model of the contoured structure.  
   
   
       35 . The computer software of  claim 33 , wherein said second set of instructions is further adapted to select a virtual point of view from which to view the three-dimensional model, and to render the three-dimensional model from the virtual point of view, thereby forming the output image.  
   
   
       36 . Computer software, comprising: 
 a first set of instructions receiving at least two offset view images of a contoured structure;    a second set of instructions recognizing feature points of the contoured structure in the offset view images, accessing a three-dimensional closed mesh model of feature points similar to the recognized feature points, and texture mapping the offset view images to the three-dimensional closed mesh model based on mappings of vertices of polygons of the mesh model into two-dimensional coordinate spaces of the offset view images, thereby forming a three-dimensional model of the contoured structure.