Metadata extraction of non-transcribed video and audio streams

Abstract

A system and computer based method for transcribing and extracting metadata from a source media. A processor-based server extracts audio and video stream from the source media. A speech recognition engine processes the audio and/or video stream to transcribe the audio and/or video stream into a time-aligned textual transcription and to extract audio amplitude by time interval, thereby providing a time-aligned machine transcribed media. The server processor measures the aural amplitude of the extracted audio amplitude and assigns a numerical value that is normalized to a single, normalized, universal amplitude scale. A database stores the time-aligned machine transcribed media, time-aligned video frames and the assigned value from the normalized amplitude scale.

1 - 30 . (canceled) 
     
     
         31 . A computer based method for transcribing and extracting metadata from a non-transcribed source media, comprising the steps of:
 extracting an audio stream from the non-transcribed source media by a processor-based server;   extracting time-aligned audio frames from the audio stream by an audio frame engine;   processing the time-aligned audio frames to extract audio amplitudes by a timed interval, to measure aural amplitudes of the extracted audio amplitudes and assign a numerical value to each extracted audio amplitude to provide time-aligned aural amplitudes by a server processor;   generating an audio histogram of the audio stream by the server processor;   normalizing the audio stream to a single, normalized, universal amplitude scale by determining a loudest frame with a loudest sound and a softest frame with a softest sound within the audio stream by the server processor;   assigning a normalized minimum amplitude value to the softest frame of the audio stream and a normalized maximum amplitude value to the loudest frame of the audio stream;   comparing each frame of the audio stream to the loudest frame and the softest frame by utilizing the audio histogram and assigning a normalized amplitude value between the normalized minimum amplitude value and the normalized maximum amplitude value to said each frame in accordance with a result of the comparison; and   storing the time-aligned audio frames, the time-aligned aural amplitudes and the normalized amplitude value of each frame of the audio stream in a database.   
     
     
         32 . The computer based method of  claim 31 , further comprising the steps of:
 speech recognition processing of the audio stream to transcribe the audio stream into a time-aligned textual transcription by a speech recognition engine to provide a time-aligned machine transcribed media;   processing the time-aligned machine transcribed media by the server processor to extract time-aligned textual metadata associated with the source media; and   storing the time-aligned machine transcribed media and the time-aligned textual metadata in the database.   
     
     
         33 . The computer based method of  claim 31 , further comprising the steps:
 extracting a video stream from the source media by a video frame engine;   extracting time-aligned video frames from the video stream by the video frame engine;   storing the time-aligned video frames in the database; and   processing the time-aligned video frames by the server processor to extract time-aligned visual metadata associated with the source media.   
     
     
         34 . The computer based method of  claim 33 , wherein the step of processing the time-aligned video frames further comprises the steps of:
 an optical character recognition (OCR) analysis on the time-aligned video frames by the server processor to extract time-aligned OCR metadata;   extracting texts from graphics by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined recognized fonts, letters and languages stored in the database; and   receiving one or more matched time-aligned OCR metadata from the database by the server processor.   
     
     
         35 . The computer based method of  claim 33 , wherein the step of processing the time-aligned video frames further comprises the steps of:
 performing a facial recognition analysis on the time-aligned video frames by the server processor to extract time-aligned facial recognition metadata;   extracting facial data points by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined facial data points for individuals stored in the database; and   receiving one or more matched time-aligned facial metadata from the database by the server processor.   
     
     
         36 . The computer based method of  claim 33 , wherein the step of processing the time-aligned video frames further comprises the steps of:
 performing an object recognition analysis on the time-aligned video frames by the server processor to extract time-aligned object recognition metadata;   extracting object data points by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined object data points for a plurality of objects stored in the database; and   receiving one or more matched time-aligned object metadata from the database by the server processor.   
     
     
         37 . The computer based method of  claim 33 , wherein the step of processing the time-aligned video frames further comprises the steps of:
 an optical character recognition (OCR) analysis on the time-aligned video frames by the server processor to extract time-aligned OCR metadata;   performing a facial recognition analysis on the time-aligned video frames by the server processor to extract time-aligned facial recognition metadata; and   performing an object recognition analysis on the time-aligned video frames by the server processor to extract time-aligned object recognition metadata.   
     
     
         38 . A non-transitory computer readable medium comprising computer executable code for transcribing and extracting metadata from a non-transcribed source media, the code comprising instructions for:
 extracting an audio stream from the non-transcribed source media by a processor-based server;   extracting time-aligned audio frames from the audio stream by an audio frame engine;   processing the time-aligned audio frames to extract audio amplitudes by a timed interval, to measure aural amplitudes of the extracted audio amplitudes and assign a numerical value to each extracted audio amplitude to provide time-aligned aural amplitudes by a server processor;   generating an audio histogram of the audio stream by the server processor;   normalizing the audio stream to a single, normalized, universal amplitude scale by determining a loudest frame with a loudest sound and a softest frame with a softest sound within the audio stream by the server processor;   assigning a normalized minimum amplitude value to the softest frame of the audio stream and a normalized maximum amplitude value to the loudest frame of the audio stream;   comparing each frame of the audio stream to the loudest frame and the softest frame by utilizing the audio histogram and assigning a normalized amplitude value between the normalized minimum amplitude value and the normalized maximum amplitude value to said each frame in accordance with a result of the comparison; and   storing the time-aligned audio frames, the time-aligned aural amplitudes and the normalized amplitude value of each frame of the audio stream in a database.   
     
     
         39 . The computer readable medium of  claim 38 , wherein said computer executable code further comprises instructions for:
 speech recognition processing of the audio stream by a speech recognition engine to transcribe the audio stream into a time-aligned textual transcription to provide a time-aligned machine transcribed media;   processing the time-aligned machine transcribed media by the server processor to extract time-aligned textual metadata associated with the source media; and   storing the time-aligned machine transcribed media and the time-aligned textual metadata in the database   
     
     
         40 . The computer readable medium of  claim 38 , wherein said computer executable code further comprises instructions for:
 extracting a video stream from the source media by a video frame engine of a processor-based server;   extracting time-aligned video frames from the video stream by the video frame engine;   storing the time-aligned video frames in the database; and   processing the time-aligned video frames by a server processor to extract time-aligned visual metadata associated with the source media.   
     
     
         41 . The computer readable medium of  claim 40 , wherein said computer executable code further comprises instructions for:
 an optical character recognition (OCR) analysis on the time-aligned video frames by the server processor to extract time-aligned OCR metadata; extracting texts from graphics by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined recognized fonts, letters and languages stored in the database; and   receiving one or more matched time-aligned OCR metadata from the database by the server processor.   
     
     
         42 . The computer readable medium of  claim 40 , wherein said computer executable code further comprises instructions for:
 performing a facial recognition analysis on the time-aligned video frames by the server processor to extract time-aligned facial recognition metadata;   extracting facial data points by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined facial data points for individuals stored in the database; and   receiving one or more matched time-aligned facial metadata from the database by the server processor.   
     
     
         43 . The computer readable medium of  claim 40 , wherein said computer executable code further comprises instructions for:
 performing an object recognition analysis on the time-aligned video frames by the server processor to extract time-aligned object recognition metadata;   extracting object data points by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined object data points for a plurality of objects stored in the database; and   receiving one or more matched time-aligned object metadata from the database by the server processor.   
     
     
         44 . A system for transcribing and extracting metadata from a non-transcribed source media, comprising:
 a processor based server connected to a communications system to receive and extract an audio stream from the source media, the server comprising:
 an audio frame engine to extract time-aligned audio frames from the audio stream; 
 a server processor to:
 process time-aligned audio frames to extract audio amplitudes by a timed interval, 
 measure aural amplitude of the extracted audio amplitudes; 
 assign a numerical value to each extracted audio amplitude to provide time-aligned aural amplitudes; 
 generate an audio histogram of the audio stream; 
 normalize the audio stream to a single, normalized, universal amplitude scale by determining a loudest frame with a loudest sound and a softest frame with a softest sound within the audio stream; 
 assign a normalized minimum amplitude value to the softest frame of the audio stream and a normalized maximum amplitude value to the loudest frame of the audio stream; 
 compare each frame of the audio stream to the loudest frame and the softest frame by utilizing the audio histogram and assign a normalized amplitude value between the normalized minimum amplitude value and the normalized maximum amplitude value to said each frame in accordance with a result of the comparison; and 
 
   a database to store the time-aligned audio frames and the time-aligned aural amplitudes, and to store the normalized amplitude value of each frame of the audio stream.   
     
     
         45 . The system of  claim 44 , wherein the server further comprises a speech recognition engine to process the audio stream to transcribe the audio stream into a time-aligned textual transcription to provide a time-aligned machine transcribed media; wherein the server processor is configured to process the time-aligned machine transcribed media to extract time-aligned textual metadata associated with the non-transcribed source media; and wherein the database stores the time-aligned machine transcribed media and the time-aligned textual metadata associated with non-transcribed source media. 
     
     
         46 . The system of  claim 44 , wherein the server comprises a video frame engine for extracting a video stream from the source media and extracting time-aligned video frames from the video stream; and wherein the server processor processes the time-aligened video frames to extract time-aligned visual metadata associated with the source media; and wherein the database stores the time-aligned video frames. 
     
     
         47 . The system of  claim 46 , wherein the server processor performs an optical character recognition (OCR) analysis on the time-aligned video frames to extract time-aligned OCR metadata by:
 extracting texts from graphics by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined recognized fonts, letters and languages stored in the database; and   receiving one or more matched time-aligned OCR metadata from the database.   
     
     
         48 . The system of  claim 46 , wherein the server processor performs a facial recognition analysis on the time-aligned video frames to extract time-aligned facial recognition metadata by:
 extracting facial data points by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined facial data points for individuals stored in the database; and   receiving one or more matched time-aligned facial metadata from the database.   
     
     
         49 . The system of  claim 46 , wherein the server processor performs an object recognition analysis on the time-aligned video frames to extract time-aligned object recognition metadata by:
 extracting object data points by a timed interval from the time-aligned video frames;   performing database lookups based on a dataset of predefined object data points for a plurality of objects stored in the database; and   receiving one or more matched time-aligned object metadata from the database.