Video Analysis, Archiving and Alerting Methods and Appartus for a Distributed, Modular and Extensible Video Surveillance System

Abstract

Systems and methods are disclosed for analyzing, managing and cataloguing video tracks generated by a video surveillance system. Systems and methods operating in accordance with the present invention receive video tracks from a video surveillance system; analyze them for particular categories of, for example, objects or events, assign unique identifying tags to the video tracks; and save them in highly-cross-referenced databases so that video tracks containing, for example, particular objects or events can be easily searched for and recalled. In addition, the present invention provides browsing facilities for viewing events that have just elapsed, events that are occurring in near real-time.

1 - 128 . (canceled) 
     
     
         129 . A video surveillance system comprising:
 at least one video surveillance system module comprising:
 a video surveillance camera, wherein the video surveillance camera is remotely programmable; 
 a video encoder to encode video gathered by the video surveillance camera; and 
 a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera and to create data derived from the video, where at least a portion of the created data is generated in dependence on at least one user-specified video analysis criterion, where the created data comprises at least one pointer to a portion of the video gathered by the video surveillance system satisfying the specified video analysis criterion; 
   at least one video surveillance system network coupled to the at least one video surveillance system module;   at least one database and application system coupled to the at least one video surveillance system network to store video gathered by the video surveillance camera and data created by the video analysis engine, and to receive and process queries for the video and data;   a video surveillance system control interface coupled to the at least one video surveillance system network to control the operations of the video surveillance system; and   a video surveillance application and control interface coupled to the at least one video surveillance system network for controlling video surveillance activities, the video surveillance application and control interface comprising:
 a program for controlling video surveillance activities; 
 a display for displaying video gathered by the video surveillance camera and data created by the video analysis engine; and 
 at least one input device for creating queries of the database and application system. 
   
     
     
         130 . The video surveillance system of  claim 129 , wherein:
 the video analysis engine further comprises:
 a track identification system for assigning a unique identifier to the at least one track. 
   
     
     
         131 . The video surveillance system of  claim 130 , wherein the track further comprises track summary information. 
     
     
         132 . The video surveillance system of  claim 131 , where the track identification system applies the unique identifier assigned to the at least one track to the track summary information associated with the at least one track. 
     
     
         133 . The video surveillance system of  claim 131 , wherein the track summary information further comprises at least one of timestamp information memorializing a creation time of the track, object class information specifying an object class, object area information, object velocity information, activity class information, activity class information corresponding to loitering activity, a pointer to a video segment and a pointer to a keyframe. 
     
     
         134 . The video surveillance system of  claim 130 , wherein the at least one database and application system further comprises:
 a database server to save the at least one track using the unique identifier.   
     
     
         135 . The video surveillance system of  claim 134 , where the video surveillance application and control interface further comprises: a track query formulation application to formulate queries to locate tracks saved in the database server satisfying a user-specified criterion, where the database and application system further comprises: an event retrieval system to accept user-specified track location queries and to locate tracks satisfying the criterion contained in the user-specified queries; and a track identifier forwarding system to forward track identifiers satisfying the user-specified query to the user requesting the track identifiers. 
     
     
         136 . The video surveillance system of  claim 135 , where the video surveillance application and control interface further comprises: a track information request application to request track information associated with forwarded track identifiers, where the at least one database and application system further comprises: a track information retrieval application to retrieve track information associated with track identifiers; and a transformation application for transforming the track information into a desired output format. 
     
     
         137 . The video surveillance system of  claim 130 , where the video surveillance system further comprises:
 a local storage database associated with the at least one video surveillance system module and coupled to the at least one video surveillance system network for storing data created by the video analysis engine.   
     
     
         138 . The video surveillance system of  claim 130 , wherein the video surveillance system module further comprises:
 a keyframe generation system to create a keyframe of a specific field of view viewed by the video surveillance camera satisfying the specified video analysis criterion.   
     
     
         139 . The video surveillance system of  claim 138 , where the track further comprises a pointer to the keyframe, wherein the keyframe generation system performs the following operation: generating a keyframe satisfying the at least one specified video analysis criterion, wherein the video analysis engine performs the following operation: creating tracks in dependence on the at least one specified video analysis criterion, wherein the at least one database and application system performs the following operation: saving the keyframe and track created when the specified criterion is satisfied. 
     
     
         140 . The video surveillance system of  claim 139 , wherein the video surveillance application and control interface performs the following operations: receiving a query for tracks corresponding to the specified criterion; and transmitting the query for the tracks to the at least one database and application system, wherein the at least one database and application system performs the following operations: receiving the query for tracks corresponding to the specified criterion; and locating tracks corresponding to the specified criterion, wherein the at least one database and application system performs the following operations: locating the keyframe corresponding to the specified criterion by using the tracks; and transmitting the keyframe satisfying the specified criterion to the video surveillance application and control interface. 
     
     
         141 . The video surveillance system of  claim 130 , wherein the video analysis criterion corresponds to at least one of an object appearing in the video gathered by the video surveillance camera, an event appearing in video gathered by the video surveillance camera and an activity appearing in video gathered by the video surveillance camera. 
     
     
         142 . The video surveillance system of  claim 129 , where the at least one database and application system further comprises:
 an alert system, wherein the alert system further comprises:
 an alert criterion specification system to specify an alert based on data created by the video analysis engine; and 
 an alert action specification system to specify alert actions to be taken when the alert criterion is satisfied. 
   
     
     
         143 . The video surveillance system of  claim 142 , where the alert criterion concerns at least one of vehicle speed and loitering activity. 
     
     
         144 . The video surveillance system of  claim 142 , where the alert action comprises at least one of an e-mail issued to a video surveillance analyst and a browser page comprising alert information, where the alert information comprises at least one of a keyframe, track identifiers and a video segment capturing activity that satisfied the alert criterion. 
     
     
         145 . The video surveillance system of  claim 129 , where:
 the video surveillance camera has a multiple programmable view feature, the video surveillance camera further comprising:
 a view programming system for programming views of the video surveillance camera; and 
   the video surveillance system control interface further comprises:
 a video surveillance camera view programming system to program views of the video surveillance camera by interacting with the view programming system of the video surveillance camera. 
   
     
     
         146 . The video surveillance system of  claim 145 , where the video analysis engine further comprises:
 a video surveillance camera view selection system to select a particular view of the video surveillance camera having the multiple programmable view feature and to analyze video associated with the view; and   a track identification system for assigning a unique identifier to the track created by the video analysis engine, where the unique identifier references at least the selected view of the video surveillance camera.   
     
     
         147 . The video surveillance system of  claim 129 , wherein the at least one database and application system performs the following operation:
 saving the at least one track generated when the at least one user-specified video analysis criterion is satisfied.   
     
     
         148 . The video surveillance system of  claim 147 , wherein the video surveillance system application and control interface performs the following operations: receiving a query for tracks corresponding to the at least one user-specified video analysis criterion, and transmitting the query for the tracks to the at least one database and application system; wherein the at least one database and application system performs the following operations: receiving the query for the tracks corresponding to the specified criterion, and locating tracks corresponding to the at least one user-specified video analysis criterion.