Quantitative, biomechanical-based analysis with outcomes and context

Abstract

Systems and methods are disclosed for generating a 3D avatar using a biomechanical analysis of observed actions with a focus on representing actions through computer-generated 3D avatars. Physical quantities of biomechanical actions can be measured from the observations, and the system can analyze these values, compare them to target or optimal values, and use the observations and known biomechanical capabilities to generate 3D avatars.

What is claimed is: 
     
         1 . A computer-implemented system configured for applying biomechanical analysis to a sequence of images of a user's movement during performance of a swing to generate computer-generated three-dimensional (3D) avatars of the swing based on user-selected kinematic parameters and sequences of motion, the system comprising:
 one or more hardware processors configured by machine-readable instructions to:   provide a movement module configured to:
 quantitatively analyze the sequence of images of the user's movement during performance of the swing to generate a quantitative measurement of the kinematic parameters and sequences of motion, and 
 identify pre-shot routines where the user swings but does not hit a ball; 
   provide a machine learning module configured to determine a user's standard practice routine over time and determine when a new practice routine deviates from the standard practice routine;   provide a 3D avatar engine configured to automatically generate a 3D avatar of the swing based on the quantitative measurement of the kinematic parameters and sequences of motion, where the 3D avatar engine is configured to generate the 3D avatar by tailoring component structures of the 3D avatar based on body points that correspond with pose, movement, or alignment features; and   provide a display generator configured to: i) display the 3D avatar of the swing and animate the 3D avatar to cause movement of the 3D avatar based on the quantitative measurement of the kinematic parameters and sequences of motion; and ii) display the user-selected kinematic parameters with the 3D avatar of the swing.   
     
     
         2 . The computer-implemented system of  claim 1 , wherein the swing is a golf swing. 
     
     
         3 . The computer-implemented system of  claim 1 , wherein the practice swing is a pre-shot routine in golf. 
     
     
         4 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 receive, from at least one image capture device, the sequence of images of the user's movement during the performance of the swing; and   provide a watch list engine configured to enable a user to select via a user interface one or more golf-specific, kinematic parameters and sequences of motion during the swing.   
     
     
         5 . The computer-implemented system of  claim 1 , wherein the movement module is further configured to identify objects other than the user in the sequence of images, wherein the objects include at least a portion of a golf club. 
     
     
         6 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 provide a watch list engine configured to enable creation of one or more watch lists comprising a list of measurable attributes of the swing for which the system is configured to monitor during analysis of the sequence of images.   
     
     
         7 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 provide a watch list engine configured to enable creation of one or more golf-specific watch lists.   
     
     
         8 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 provide a watch list engine is configured to enable creation of one or more golf-specific watch lists and specification of a weighting of kinematic parameters and sequences for analysis of images of swings, wherein the kinematic parameters correspond with measurable attributes of the swings.   
     
     
         9 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 display a graphical user interface with options to enable creation of one or more motion trackers comprising specific kinematic parameters and sequences of motions.   
     
     
         10 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 display a graphical user interface with options to enable creation of one or more golf-specific motion trackers.   
     
     
         11 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 display a graphical user interface with options to enable creation of one or more motion trackers comprising specific kinematic parameters and sequences of motions, the kinematic parameters comprising position, velocity, acceleration, and/or higher-order derivatives of the position with respect to time.   
     
     
         12 . The computer-implemented system of  claim 1 , wherein the 3D avatar engine is further configured to apply recorded 3D motion data from the user to an avatar. 
     
     
         13 . The computer-implemented system of  claim 1 , wherein the 3D avatar engine is further configured to model the user's body as an articulated structure including a set of human skeletal components connected via joints and a measured spatial relationship among the user's skeletal components and animate the model along 3D space based on the measured movement of the user along the 3D space. 
     
     
         14 . The computer-implemented system of  claim 1 , wherein the 3D avatar engine is further configured to customize a model of the user's body based on the golf-specific, kinematic parameters and sequences of motion. 
     
     
         15 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 compare, for the user-selected kinematic parameters and sequences of motion, the measured movements in the performance of the swing by the user to a stored set of target measured movements of a reference performance of the swing.   
     
     
         16 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 compare, for the user-selected kinematic parameters and sequences of motion, the measured movements in the performance of the swing by the user to a stored set of target measured movements of a reference performance of the swing; and   the 3D avatar engine is further configured to simultaneously display an avatar of the swing and animate the 3D avatar to cause movement of the 3D avatar based on the measured, tracked movement from the sequence of images based on the swing during the performance of the swing and a reference avatar based on the reference performance of the swing.   
     
     
         17 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 compare the measured movements in the swing to a stored set of a measured movements of a second user's performance of a second swing.   
     
     
         18 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 compare, for the user-selected kinematic parameters and sequences of motion, the measured movements of the user's performance of the swing by the user to a stored set of a measured movements, including the user selected, kinematic parameters and sequences of motion, of a second user's performance of the swing; and   the 3D avatar engine is further configured to simultaneously display an avatar based on the user's performance of the activity and an avatar based on the second user's performance of a second swing.   
     
     
         19 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 compare the measured movements in a user's first performance of the swing to a stored set of target measured movements of the user's second performance of a second swing.   
     
     
         20 . The computer-implemented system of  claim 1 , wherein the one or more hardware processors are further configured to:
 compare, for the user-selected kinematic parameters and sequences of motion, the measured movements in a user's first performance of the swing by the user to a stored set of target measured movements, including the user selected kinematic parameters and sequences of motion, of the user's second performance of a second swing; and   the 3D avatar engine is configured to simultaneously display an avatar based on the user's first performance of the swing and an avatar based on the user's second performance of the second swing.   
     
     
         21 . The computer-implemented system of  claim 1 , wherein the display generator is further configured to enable a user to playback the user's performances of the swing, including display of a video of the user's performance and the animated 3D avatar display of the user's performance; and further comprising user selectable display elements that enable the user to control a perspective from which the animated 3D avatar is displayed. 
     
     
         22 . The computer-implemented system of  claim 1 , wherein the movement module is further configured to identify one or more objects other than the user in the images, track kinematic parameters of the one or more objects in the images, and generate context data based on the tracked kinematic parameters of the one or more objects. 
     
     
         23 . The computer-implemented system of  claim 1 , wherein the movement module is further configured to identify one or more objects other than the user in the images, track and measure kinematic parameters of the one or more objects in the images and the display generator is further configured to display the 3D avatar relative to the one or more objects based on the measured movements of the user in the images and the kinematic parameters of the one or more objects in the images. 
     
     
         24 . The computer-implemented system of  claim 1 , wherein the movement module is further configured to measure and quantify user-selected movements of the user via a motion tracker, wherein the motion tracker is configured to generate measurements of a predetermined sequence of motions. 
     
     
         25 . The computer-implemented system of  claim 1 , wherein the movement module is further configured to:
 dynamically generate one or more watch lists comprising a list of measurable attributes of a performance for which the system is programmed to monitor in the images of the user's movement during performance of the swing; and   store the watch lists in a memory of the system.   
     
     
         26 . The computer-implemented system of  claim 1 , wherein the movement module is further configured to:
 display a graphical user interface with options to for user selection of one or more components of the user's movement during performance of the swing; and   store the one or more components in a memory of the system as one or more motion trackers.   
     
     
         27 . The computer-implemented system of  claim 1 , wherein the movement module is further configured to track a continuous movement of the swing across a sequence of images. 
     
     
         28 . The computer-implemented system of  claim 1 , wherein the 3D avatar engine is further configured to generate the 3D avatar based on a user selected sequence of measurements. 
     
     
         29 . The computer-implemented system of  claim 1 , wherein the 3D avatar engine is further configured to:
 i) model the user's body as an articulated structure, including a set of human skeletal components connected via joints and a measured spatial relationship among the user's skeletal components;   ii) customize the model of the user's body based on the performance of the swing being golf-specific, the customization including a determination of the human skeletal components and/or joints to include in the model based on at least the swing; and   iii) animate the model along three-dimensional 3D space based on the measured movement of the user along three-dimensional the 3D space.   
     
     
         30 . The computer-implemented system of  claim 1 , wherein the at least one image capture device comprises a single camera and the 3D avatar engine is further configured to automatically generate the 3D avatar of the user from a single-viewpoint recording by the single camera. 
     
     
         31 . The computer-implemented system of  claim 1 , further comprising:
 a watch list engine configured to generate a dynamic watch list;   a user interface configured to present options for a user to make selections for the dynamic watch lists by selecting one or more components, including one or more body parts or context objects; and   one or more motion trackers configured to determine the kinematic parameters of the one or more selected components to assess performance of the swing.   
     
     
         32 . The computer-implemented system of  claim 1 , wherein the 3D avatar engine is further configured to automatically generate the 3D avatar of the user by conversion of a 2D pose estimation from an image into the 3D space. 
     
     
         33 . The computer-implemented system of  claim 1 , wherein the 3D avatar engine is further configured to:
 automatically generate the 3D avatar of the user from the 2D image by modeling the user's body as an articulated structure, including a predetermined set of human skeletal components connected via joints and a measured spatial relationship among the user's skeletal components; and   detecting the joints through 3D pose estimation.   
     
     
         34 . The computer-implemented system of  claim 1 , further comprising:
 a 3D avatar engine configured to apply recorded 3D motion data from the user to the 3D avatar for: 1) playback of the swing to enable the user to watch their own swing from any angle; and ii) apply a second user's recorded kinematic parameter sequence to a second avatar, to enable a coach, whose swing is recorded to model performances for the user.