System and method for real-time automatic video key frame analysis for critical motion in complex athletic movements

Abstract

A system and method train a neural network to process video for the identification of critical points of motion in a movement. Embodiments include a process for training the neural network and a process for identifying the video frames including a critical point based on the criteria of the neural network. An exemplary embodiment includes a software application which may be accessible through a computing device to analyze videos on site.

What is claimed is: 
     
         1 . A method for identifying critical points of motion in a video, comprising:
 using a feature extraction neural network trained to identify features among a plurality of images of movements, wherein the features are associated with a critical point of movement;   receiving a video sequence including a plurality of video frames comprising a motion captured by a camera;   identifying, by the feature extraction neural network, individual frames from the plurality of video frames, wherein the identified individual frames include a known critical point in the captured motion based on identified features associated with a critical point of movement; and   displaying the identified frames which show the critical points in the captured motion.   
     
     
         2 . The method of  claim 1 , further comprising training a temporal neural network based on input from the feature extraction neural network. 
     
     
         3 . The method of  claim 2 , further comprising converting an output of the temporal neural network model into time stamps of the plurality of video frames. 
     
     
         4 . A method of automated identification of frames for video analysis of critical motion, comprising:
 receiving a raw sequence of video data;   receiving selections of the raw sequence of video data, wherein the selections include manual entered labels;   automatically converting the manual entered labels to machine readable labels;   training a neural network feature extractor based on the machine readable labels to identify video frames that include critical points of motion.   
     
     
         5 . The method of  claim 4 , further comprising training a temporal neural network based on an output of the neural network feature extractor. 
     
     
         6 . The method of  claim 4 , further comprising:
 extracting a selected video frame for each unique timestamp;   labeling the extracted video frames;   grouping remaining unlabeled video frames; and   calculating a temporal distance between labeled video frames and unlabeled video frames.   
     
     
         7 . The method of  claim 6 , further comprising associating unlabeled video frames with a labeled video frame based on a shortest temporal distance between each unlabeled video frame to the labeled video frames. 
     
     
         8 . The method of  claim 7 , further comprising generating a similarity score for each labeled video frame and its associated unlabeled video frames based on an image comparison of content in the labeled video frame and the labeled video frame's associated unlabeled video frames. 
     
     
         9 . The method of  claim 8 , further comprising labeling one of the unlabeled video frames in the event the similarity score for the labeled video frame and the labeled video frame's associated unlabeled video frames falls within a threshold range of values. 
     
     
         10 . The method of  claim 9 , further comprising using the unlabeled video frames that has been labeled in training the neural network feature extractor to identify video frames that include critical points of motion. 
     
     
         11 . A computer program product for automated identification of frames for video analysis of critical motion, the computer program product comprising:
 one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising:
 receiving a raw sequence of video data; 
 receiving selections of the raw sequence of video data, wherein the selections include manual entered labels; 
 automatically converting the manual entered labels to machine readable labels; 
 training a neural network feature extractor based on the machine readable labels to identify video frames that include critical points of motion. 
   
     
     
         12 . The computer program product of  claim 11 , wherein the program instructions further comprise training a temporal neural network based on an output of the neural network feature extractor. 
     
     
         13 . The computer program product of  claim 11 , wherein the program instructions further comprise:
 extracting a selected video frame for each unique timestamp;   labeling the extracted video frames;   grouping remaining unlabeled video frames; and   calculating a temporal distance between labeled video frames and unlabeled video frames.   
     
     
         14 . The computer program product of  claim 13 , wherein the program instructions further comprise associating unlabeled video frames with a labeled video frame based on a shortest temporal distance between each unlabeled video frame to the labeled video frames. 
     
     
         15 . The computer program product of  claim 14 , wherein the program instructions further comprise generating a similarity score for each labeled video frame and its associated unlabeled video frames based on an image comparison of content in the labeled video frame and the labeled video frame's associated unlabeled video frames. 
     
     
         16 . The computer program product of  claim 15 , wherein the program instructions further comprise labeling one of the unlabeled video frames in the event the similarity score for the labeled video frame and the labeled video frame's associated unlabeled video frames falls within a threshold range of values. 
     
     
         17 . The computer program product of  claim 16 , wherein the program instructions further comprise using the unlabeled video frames that has been labeled in training the neural network feature extractor to identify video frames that include critical points of motion.