Computerized system and method for fine-grained event detection and content hosting therefrom

Abstract

The disclosed systems and methods provide a novel framework that provides mechanisms for performing cost-effective, accurate and scalable detection and recognition of fine-grained events. The framework functions by training high precision and high recall object/optical character recognition (OCR) models and aligning video frames to text commentaries of the videos (e.g., licensed play-by-play). The disclosed framework operates as a single algorithm that performs multimodal alignments between events/actions within videos and their prescribed text. Thus, the disclosed framework is able to scale to fine-grained action categories across different venues by delving into the key frames and key aspects of a video to identify particular actions performed by particular actors, thereby providing the novelty of fine-granted action detection and recognition.

What is claimed is: 
     
       1. A method comprising:
 identifying a video; 
 analyzing the video, and based on the analysis, identifying information related to a set of recognized actions from content of the video; 
 further analyzing the video, and determining metadata related to the video; 
 identifying play-by-play text of the video, the play-by-play text providing an official time-aligned captioning of segments of the video; 
 determining, by executing a trained optical character recognition (OCR) model, a set of video segments in the video, the determination comprising analyzing a clock present in frames of the video segments, and identifying text content and a continuous time value associated with the clock; 
 aligning the play-by-play text to each of the segments, each of the set of segments being at least a predetermined number of seconds long and in continuous order, such that an entirety of an action is captured during the respective segment; and 
 providing access to the set of video segments. 
 
     
     
       2. The method of  claim 1 , further comprising:
 identifying a set of training videos; 
 determining, for each training video, a presence of other contextual objects within each training video frame; 
 determining, for each training video, surface form consistencies of objects in frames of the training videos; and 
 determining, for each video, temporal consistencies across the frames of each training video. 
 
     
     
       3. The method of  claim 2 , further comprising:
 determining knowledge constraints based on the determinations from the training videos; 
 applying the knowledge constraints to recognized text from the training videos utilizing pretrained models; and 
 training text detection and recognition models based on clean text determined by application of the knowledge constraints. 
 
     
     
       4. The method of  claim 3 , wherein the training of the text detection and recognition models is performed for different domains. 
     
     
       5. The method of  claim 1 , further comprising:
 downloading, over a network, a set of videos; 
 decoding each of the videos in the set of videos, the decoding causing, for each video, a set of frames to be identified; 
 analyzing each of the set of frames, and determining information related to a clock within a subset of each set of frames, the subset comprising every nth frame for a n*x frames per second (fps) video; 
 determining information related to text included within the clock in each frame of the sub set; 
 performing text recognition on the text information inside bounding boxes in each clock; and 
 determining a query based on the text recognition. 
 
     
     
       6. The method of  claim 5 , wherein the information related to the clock corresponds to a spatial position within each frame. 
     
     
       7. The method of  claim 5 , further comprising:
 analyzing the subset, and determining metadata related to the video; and 
 identifying play-by-play text for each video segment. 
 
     
     
       8. The method of  claim 7 , further comprising aligning the play-by-play text to specific video segments. 
     
     
       9. The method of  claim 5 , wherein the information related to a clock and the information related to the text included in the clock corresponds to the bounding box for each. 
     
     
       10. The method of  claim 1 , further comprising:
 analyzing the play-by-play text, and based on the analysis, identifying information related to a time and a specific portion of a game; and 
 mapping, based on a composite key defined by data related to the identified time and game portion information, the video segments to the portions of the play-by-play text. 
 
     
     
       11. The method of  claim 1 , wherein when the video is for a portion of a game, the play-by-play text includes a corresponding portion of the official time-aligned captioning. 
     
     
       12. The method of  claim 1 , wherein the play-by-play text is identified based on an identifier (ID) of a game, wherein the game ID is identified based on at least one of metadata related to the game and a portion of a uniform resource locator (URL) of the video. 
     
     
       13. The method of  claim 1 , wherein the video segment comprises information associated with at least one of a type of action, a particular player, a particular time period, or a particular team. 
     
     
       14. The method of  claim 1 , further comprising:
 requesting, over a network, third party digital content based at least on information related to the video segment; 
 receiving, over the network, the third party digital content; and 
 communicating, over the network, the third party digital content for display along with the video segments. 
 
     
     
       15. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a processor associated with a device, performs a method comprising:
 identifying a video; 
 analyzing the video, and based on the analysis, identifying information related to a set of recognized actions from content of the video; 
 further analyzing the video, and determining metadata related to the video; 
 identifying play-by-play text of the video, the play-by-play text providing an official time-aligned captioning of segments of the video; 
 determining, by executing a trained optical character recognition (OCR) model, a set of video segments in the video, the determination comprising analyzing a clock present in frames of the video segments, and identifying text content and a continuous time value associated with the clock; 
 aligning the play-by-play text to each of the segments, each of the set of segments being at least a predetermined number of seconds long and in continuous order, such that an entirety of an action is captured during the respective segment; and 
 providing access to the set of video segments. 
 
     
     
       16. The non-transitory computer-readable storage medium of  claim 15 , further comprising:
 identifying a set of training videos; 
 determining, for each training video, a presence of other contextual objects within each training video frame; 
 determining, for each training video, surface form consistencies of objects in frames of the training videos; 
 determining, for each video, temporal consistencies across the frames of each training video; 
 applying knowledge constraints to recognized text from the training videos utilizing pretrained models; and 
 training text detection and recognition models based on clean text determined by application of the knowledge constraints, wherein the training of the text detection and recognition models is performed for different domains. 
 
     
     
       17. The non-transitory computer-readable storage medium of  claim 15 , further comprising:
 downloading, over a network, a set of videos; 
 decoding each of the videos in the set of videos, the decoding causing, for each video, a set of frames to be identified; 
 analyzing each of the set of frames, and determining information related to a clock within a subset of each set of frames, the subset comprising every nth frame for a n*x frames per second (fps) video; 
 determining information related to text included within the clock in each frame in the sub set; 
 performing text recognition on the text information inside bounding boxes in each clock; 
 determining a query based on the text recognition; 
 analyzing the subset, and determining metadata related to the video; and 
 identifying a play-by-play text for each video. 
 
     
     
       18. A computing device comprising:
 a processor configured to: 
 identify a video; 
 analyze the video, and based on the analysis, identify information related to a set of recognized actions from content of the video; 
 further analyze the video, and determine metadata related to the video; 
 identify play-by-play text of the video, the play-by-play text providing an official time-aligned captioning of segments of the video; 
 determine, by executing a trained optical character recognition (OCR) model, a set of video segments in the video, the determination comprising analyzing a clock present in frames of the video segments, and identifying text content and a continuous time value associated with the clock; 
 align the play-by-play text to each of the segments, each of the set of segments being at least a predetermined number of seconds long and in continuous order, such that an entirety of an action is captured during the respective segment; and 
 provide access to the set of video segments. 
 
     
     
       19. The computing device of  claim 18 , further comprising:
 identify a set of training videos; 
 determine, for each training video, a presence of other contextual objects within each training video frame; 
 determine, for each training video, surface form consistencies of objects in frames of the training videos; 
 determine, for each video, temporal consistencies across the frames of each training video; 
 apply knowledge constraints to recognized text from the training videos utilizing pretrained models; and 
 train text detection and recognition models based on clean text determined by application of the knowledge constraints, wherein the training of the text detection and recognition models is performed for different domains. 
 
     
     
       20. The computing device of  claim 18 , further comprising:
 download, over a network, a set of videos; 
 decode each of the videos in the set of videos, such that, for each video, a set of frames is identified; 
 analyze each of the set of frames, and determine information related to a clock within a subset of each set of frames, the subset comprising every nth frame for a n*x frames per second (fps) video; 
 determine information related to text included within the clock in each frame of the subset; 
 perform text recognition on the text information inside bounding boxes in each clock; 
 determine a query based on the text recognition; 
 analyze the subset, and determine metadata related to the video; and 
 identify a play-by-play text for each video.