Techniques for creating facial animation using a face mesh

Abstract

Techniques for creating facial animation using a face mesh are provided. The techniques include determining a face mesh sequence. A portion of the face mesh sequence is then fitted to an image of the face. For example, a first frame of the face mesh sequence is fitted to the image. The face image is then projected onto the portion of the face mesh based on the fitting. Also, a hair model may be generated and fitted onto the face mesh sequence. Additionally, changes in color differences between sequences of the face mesh may be computed. The face mesh sequence is then played to create a facial animation sequence for the face image. The color differences that were created may also be added to the sequences as they are being played. Accordingly, the playing of the face mesh sequence creates a realistic facial animation of the image of the face.

1. A method for creating facial animation, the method comprising:
 determining a face mesh sequence; 
 selecting a face mesh in the face mesh sequence, the face mesh sequence including the face mesh having a first expression and subsequent face meshes including one or more expression changes from the first expression; 
 fitting the face mesh of the face mesh sequence to a face image based on features of the face image and features of the face mesh; 
 determining displacement values for vertices of the face mesh in the fitting for the face mesh, the displacement values used to deform the face mesh to fit the face image based on the features of the face image and the features of the face mesh; 
 projecting the face image onto the face mesh as texture based on the fitting; 
 adding the displacement values to vertices in subsequent face meshes in the face mesh sequence; 
 projecting the face image on the subsequent face meshes, wherein the face image includes a first image expression that is projected onto the face mesh with the first expression and subsequent face meshes with the one or more expression changes, the first image expression being different from the one or more expression changes on the subsequent face meshes; and 
 playing, using a computing device, the face mesh sequence to create facial animation for the face image that shows the first expression in the face mesh and the one or more expression changes in the subsequent face meshes using the same face image with the first image expression, the facial animation created using the face image used to fit the face mesh, wherein the subsequent face meshes are deformed using the displacement values to fit the face image. 
 
     
     
       2. The method of  claim 1 , further comprising:
 computing a color difference for the subsequent face meshes of the face mesh sequence based on the one or more expression changes for the subsequent face meshes; and 
 adding the computed color difference for each of the subsequent face meshes in the face mesh sequence to the same face image that was projected onto each of the subsequent face meshes in the face mesh sequence. 
 
     
     
       3. The method of  claim 2 , further comprising ignoring any positive color differences. 
     
     
       4. The method of  claim 1 , further comprising:
 creating a hair model from the face mesh; and 
 fitting the hair model on each face mesh in the face mesh sequence. 
 
     
     
       5. The method of  claim 4 , wherein projecting the face image comprises projecting hair of the face image onto the hair model for the face mesh. 
     
     
       6. The method of  claim 1 , wherein the fitting the face mesh comprises determining scale, rotation, and/or translation values by minimizing a distance between first feature points of the face mesh to second feature points of the face image. 
     
     
       7. The method of  claim 6 , wherein the scale, rotation, and/or translation values are applied to the vertices in the subsequent face meshes. 
     
     
       8. The method of  claim 1 , wherein the face mesh sequence is a face mesh recording of a human user. 
     
     
       9. The method of  claim 1 , wherein the portion of the face mesh sequence is a still image that corresponds to the face image. 
     
     
       10. The method of  claim 1 , wherein the face image is an image from a face that was not used to create the face mesh. 
     
     
       11. A device for creating facial animation, the device comprising:
 one or more processors; and 
 logic encoded in one or more computer-readable media for execution by the one or more processors and when executed configured to: 
 determine a face mesh sequence; 
 select a face mesh in the face mesh sequence, the face mesh sequence including the face mesh having a first expression and subsequent face meshes including one or more expression changes from the first expression; 
 fit the face mesh of the face mesh sequence to a face image based on features of the face image and features of the face mesh; 
 determine displacement values for vertices of the face mesh in the fitting for the face mesh, the displacement values used to deform the face mesh to fit the face image based on the features of the face image and the features of the face mesh; 
 project the face image onto the face mesh as texture based on the fitting; 
 add the displacement values to vertices in subsequent face meshes in the face mesh sequence; 
 project the face image on the subsequent face meshes, wherein the face image includes a first image expression that is projected onto the face mesh with the first expression and subsequent face meshes with the one or more expression changes, the first image expression being different from the one or more expression changes on the subsequent face meshes; and 
 play the face mesh sequence to create facial animation for the face image that shows the first expression in the face mesh and the one or more expression changes in the subsequent face meshes using the same face image with the first image expression, the facial animation created using the face image used to fit the face mesh, wherein the subsequent face meshes are deformed using the displacement values to fit the face image. 
 
     
     
       12. The device of  claim 11 , wherein the logic is further configured to:
 compute a color difference for the subsequent face meshes of the face mesh sequence based on the one or more expression changes for the subsequent face meshes; and 
 add the computed color difference for each of the subsequent face meshes in the face mesh sequence to the same face image that was projected onto each of the subsequent face meshes in the face mesh sequence. 
 
     
     
       13. The device of  claim 12 , wherein the logic is further configured to ignore any positive color differences. 
     
     
       14. The device of  claim 11 , wherein the logic is further configured to:
 create a hair model from the face mesh; and 
 fit the hair model on each face mesh in the face mesh sequence. 
 
     
     
       15. The device of  claim 14 , wherein the logic configured to project the face image comprises logic configured to project hair of the face image onto the hair model for the face mesh. 
     
     
       16. The device of  claim 11 , wherein the logic configured to fit the face mesh comprises logic configured to determine scale, rotation, and/or translation values by minimizing a distance between first feature points of the face mesh to second feature points of the face image. 
     
     
       17. The device of  claim 16 , wherein the scale, rotation, and/or translation values are applied to the vertices in the subsequent face meshes. 
     
     
       18. The device of  claim 11 , wherein the face mesh sequence is a face mesh recording of a human user. 
     
     
       19. The device of  claim 11 , wherein the portion of the face mesh sequence is a still image that corresponds to the face image. 
     
     
       20. The device of  claim 11 , wherein the face image is an image from a face that was not used to create the face mesh. 
     
     
       21. Software encoded in one or more computer-readable media for execution by the one or more processors and when executed operable to:
 determine a face mesh sequence; 
 select a face mesh in the face mesh sequence, the face mesh sequence including the face mesh having a first expression and subsequent face meshes including one or more expression changes from the first expression; 
 fit the face mesh of the face mesh sequence to a face image of a face based on features of the face image and features of the face mesh; 
 determine displacement values for vertices of the face mesh in the fitting for the face mesh, the displacement values used to deform the face mesh to fit the face image based on the features of the face image and the features of the face mesh; 
 project the face image onto the face mesh as texture based on the fitting; 
 add the displacement values to vertices in subsequent face meshes in the face mesh sequence; 
 project the face image on the subsequent face meshes, wherein the face image includes a first image expression that is projected onto the face mesh with the first expression and subsequent face meshes with the one or more expression changes, the first image expression being different from the one or more expression changes on the subsequent face meshes; and 
 play the face mesh sequence to create facial animation for the face image that shows the first expression in the face mesh and the one or more expression changes in the subsequent face meshes using the same face image with the first image expression, the facial animation created using the face image used to fit the face mesh, wherein the subsequent face meshes are deformed using the displacement values to fit the face image.