Method for Facial Animation

Abstract

A method of animating a digital character according to facial expressions of a user, comprising the steps of, (a) obtaining a 2D image and 3D depth map of the face of the user, (b) determining expression parameters for a user expression model so that a facial expression of the user-specific expression model represents the face of the user shown in the 2D image and 3D depth map (c) using the expression parameters and an animation prior to determine animation parameters usable to animate a digital character, wherein the animation prior is a sequence of animation parameters which represent predefined animations of a digital character (d) using the animation parameters to animate a digital character so that the digital character mimics the face of the user.

1 . A method comprising:
 generating, by an electronic device, a user-specific expression model, comprising:   capturing, by one or more sensors of the electronic device, first sensor data of a plurality of expressions of a face of a user,
 determining a set of model parameters from the sensor data, and 
 modifying a generic user-specific expression model using the set of model parameters to obtain the user-specific expression model; and 
   driving an avatar, by the electronic device, using the user-specific expression model, comprising:
 capturing, by the electronic device, second sensor data of the face of the user, and 
 determine, by the electronic device, expression parameters from the second sensor data, 
 wherein the expression parameters and the user-specific expression model are used to animate a digital character. 
   
     
     
         2 . The method of  claim 1 , wherein the first sensor data is captured from a plurality of angles with respect to the user. 
     
     
         3 . The method of  claim 1 , wherein the plurality of expressions comprise a set of predefined expressions. 
     
     
         4 . The method of  claim 3 , wherein the model parameters comprise a polygonal mesh for each expression of the set of predefined expressions. 
     
     
         5 . The method of  claim 1 , wherein the first sensor data comprises image data and depth data. 
     
     
         6 . The method of  claim 1 , wherein the expression parameters and the user-specific expression model are used to animate a digital character by performing an optimization of the expression parameters and animation priors for the digital character. 
     
     
         7 . The method of  claim 1 , wherein the expression parameters and the user-specific expression model are used to animate a digital character by causing the digital character to mimic an expression in the second sensor data. 
     
     
         8 . A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
 generate, by an electronic device, a user-specific expression model, comprising:
 capturing, by one or more sensors of the electronic device, first sensor data of a plurality of expressions of a face of a user, 
 determining a set of model parameters from the sensor data, and 
 modifying a generic user-specific expression model using the set of model parameters to obtain the user-specific expression model; and 
   drive an avatar, by the electronic device, using the user-specific expression model, comprising:
 capturing, by the electronic device, second sensor data of the face of the user, and 
 determine, by the electronic device, expression parameters from the second sensor data, 
 wherein the expression parameters and the user-specific expression model are used to animate a digital character. 
   
     
     
         9 . The non-transitory computer readable medium of  claim 8 , wherein the first sensor data is captured from a plurality of angles with respect to the user. 
     
     
         10 . The non-transitory computer readable medium of  claim 8 , wherein the plurality of expressions comprise a set of predefined expressions. 
     
     
         11 . The non-transitory computer readable medium of  claim 10 , wherein the model parameters comprise a polygonal mesh for each expression of the set of predefined expressions. 
     
     
         12 . The non-transitory computer readable medium of  claim 8 , wherein the first sensor data comprises image data and depth data. 
     
     
         13 . The non-transitory computer readable medium of  claim 8 , wherein the expression parameters and the user-specific expression model are used to animate a digital character by performing an optimization of the expression parameters and animation priors for the digital character. 
     
     
         14 . The non-transitory computer readable medium of  claim 8 , wherein the expression parameters and the user-specific expression model are used to animate a digital character by causing the digital character to mimic an expression in the second sensor data. 
     
     
         15 . A system comprising:
 one or more processors; and   one or more computer readable media comprising computer readable code executable by the one or more processors to:
 generate, by an electronic device, a user-specific expression model, comprising:
 capturing, by one or more sensors of the electronic device, first sensor data of a plurality of expressions of a face of a user, 
 determining a set of model parameters from the sensor data, and 
 modifying a generic user-specific expression model using the set of model parameters to obtain the user-specific expression model; and 
 
 drive an avatar, by the electronic device, using the user-specific expression model, comprising:
 capturing, by the electronic device, second sensor data of the face of the user, and 
 determine, by the electronic device, expression parameters from the second sensor data, 
 wherein the expression parameters and the user-specific expression model are used to animate a digital character. 
 
   
     
     
         16 . The system of  claim 15 , wherein the first sensor data is captured from a plurality of angles with respect to the user. 
     
     
         17 . The system of  claim 15 , wherein the plurality of expressions comprise a set of predefined expressions, and wherein the model parameters comprise a polygonal mesh for each expression of the set of predefined expressions. 
     
     
         18 . The system of  claim 15 , wherein the first sensor data comprises image data and depth data. 
     
     
         19 . The system of  claim 15 , wherein the expression parameters and the user-specific expression model are used to animate a digital character by performing an optimization of the expression parameters and animation priors for the digital character. 
     
     
         20 . The system of  claim 15 , wherein the expression parameters and the user-specific expression model are used to animate a digital character by causing the digital character to mimic an expression in the second sensor data.