Enhanced animation generation based on video with local phase

Abstract

Embodiments of the systems and methods described herein provide a dynamic animation generation system that can apply a real-life video clip with a character in motion to a first neural network to receive rough motion data, such as pose information, for each of the frames of the video clip, and overlay the pose information on top of the video clip to generate a modified video clip. The system can identify a sliding window that includes a current frame, past frames, and future frames of the modified video clip, and apply the modified video clip to a second neural network to predict a next frame. The dynamic animation generation system can then move the sliding window to the next frame while including the predicted next frame, and apply the new sliding window to the second neural network to predict the following frame to the next frame.

1 - 20 . (canceled) 
     
     
         21 . A computer-implemented method comprising:
 accessing motion capture data associated with an entity in motion;   providing the motion capture data as input to a first machine learning model;   generating pose information associated with the entity using the first machine learning model;   identifying a first plurality of frames of the motion capture data, wherein the first plurality of frames comprises a current frame;   providing the first plurality of frames of the motion capture data as input to a second machine learning model; and   generating a first predicted frame and a first local motion phase corresponding to the first predicted frame using the second machine learning model.   
     
     
         22 . The computer-implemented method of  claim 21 , wherein the computer-implemented method further comprises overlaying the pose information on the motion capture data to generate modified motion capture data. 
     
     
         23 . The computer-implemented method of  claim 21 , wherein the pose information comprises velocity information corresponding to joints of the entity. 
     
     
         24 . The computer-implemented method of  claim 21 , wherein the first plurality of frames comprises sampled frames of the motion capture data at a predefined time threshold. 
     
     
         25 . The computer-implemented method of  claim 21 , wherein the first plurality of frames comprises an equal number of past frames and future frames relative to the current frame. 
     
     
         26 . The computer-implemented method of  claim 21 , wherein the motion capture data is captured from a camera on a mobile computing device. 
     
     
         27 . The computer-implemented method of  claim 21 , wherein at least one of the first machine learning model or the second machine learning model is a neural network. 
     
     
         28 . The computer-implemented method of  claim 21 , wherein the first local motion phase includes phase information for each joint of the entity. 
     
     
         29 . The computer-implemented method of  claim 21 , wherein the first local motion phase includes phase information for each bone of the entity. 
     
     
         30 . The computer-implemented method of  claim 21 , wherein the entity is a real-life person in motion. 
     
     
         31 . A non-transitory computer storage medium storing instructions that when executed by a system of one or more processors, cause the one or more processors to perform operations comprising:
 accessing motion capture data associated with an entity in motion;   providing the motion capture data as input to a first machine learning model;   generating pose information associated with the entity using the first machine learning model;   identifying a first plurality of frames of the motion capture data, wherein the first plurality of frames comprises a current frame;   providing the first plurality of frames of the motion capture data as input to a second machine learning model; and   generating a first predicted frame and a first local motion phase corresponding to the first predicted frame using the second machine learning model.   
     
     
         32 . The non-transitory computer storage medium of  claim 31 , wherein the pose information comprises velocity information corresponding to joints of the entity. 
     
     
         33 . The non-transitory computer storage medium of  claim 31 , wherein the first plurality of frames comprises sampled frames of the motion capture data at a predefined time threshold. 
     
     
         34 . The non-transitory computer storage medium of  claim 31 , wherein at least one of the first machine learning model or the second machine learning model is a neural network. 
     
     
         35 . The non-transitory computer storage medium of  claim 31 , wherein the first local motion phase includes phase information for each joint of the entity. 
     
     
         36 . The non-transitory computer storage medium of  claim 31 , wherein the first local motion phase includes phase information for each bone of the entity. 
     
     
         37 . A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:
 accessing motion capture data associated with an entity in motion;   providing the motion capture data as input to a first machine learning model;   generating pose information associated with the entity using the first machine learning model;   identifying a first plurality of frames of the motion capture data, wherein the first plurality of frames comprises a current frame;   providing the first plurality of frames of the motion capture data as input to a second machine learning model; and   generating a first predicted frame and a first local motion phase corresponding to the first predicted frame using the second machine learning model.   
     
     
         38 . The system of  claim 37 , wherein the pose information comprises velocity information corresponding to joints of the entity. 
     
     
         39 . The system of  claim 37 , wherein at least one of the first machine learning model or the second machine learning model is a neural network. 
     
     
         40 . The system of  claim 37 , wherein the first local motion phase includes phase information for each joint and bone of the entity.