Unified system and method for animal behavior characterization with training capabilities

Abstract

In general, the present invention is directed to systems and methods for finding the position and shape of an object using video. The invention includes a system with a video camera coupled to a computer in which the computer is configured to automatically provide object segmentation and identification, object motion tracking (for moving objects), object position classification, and behavior identification. In a preferred embodiment, the present invention may use background subtraction for object identification and tracking, probabilistic approach with expectation-maximization for tracking the motion detection and object classification, and decision tree classification for behavior identification. Thus, the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various objects and the object's movements within the image. The image may be provided in real time or from storage. The invention is particularly useful for monitoring and classifying animal behavior for testing drugs and genetic mutations, but may be used in any of a number of other surveillance applications.

1 - 42 . (canceled) 
     
     
         43 . A user configurable video-based animal behavior analysis system, comprising a computer configured to determine features of an animal from video images and to characterize activity of said animal as one of a set of behaviors based on an analysis of changes in said features over time;
 wherein the set of behaviors comprises at least one predetermined standard behavior and   wherein one or more user-defined behaviors can be added to the set of behaviors.   
     
     
         44 . The system of  claim 43  wherein the user-defined behavior is entered into the system in the form of video images of the user-defined behavior. 
     
     
         45 . The system of  claim 43  wherein the user-defined behavior is entered into the system in the form of a set of rules. 
     
     
         46 . The system of  claim 43 , further comprising: a video camera and a video digitization unit coupled to said computer for capturing said video images and converting said video images from analog to digital format. 
     
     
         47 . The system of  claim 43 , further comprising: an animal identification, segregation, and tracking module receiving said video images. 
     
     
         48 . The system of  claim 47  wherein the at least one user-defined behavior is defined by analysis of video images of the user-defined behavior by the animal identification, segregation and tracking module. 
     
     
         49 . The system of  claim 47 , wherein said computer further includes a behavior identification module for characterizing activity of said animal, said behavior identification module being coupled to said animal identification, segregation, and tracking module. 
     
     
         50 . The system of  claim 49 , wherein said computer further includes an animal behavior storage module for storing information about the predetermined standard behaviors, said animal behavior storage module being coupled to said behavior identification module. 
     
     
         51 . The system of  claim 50  wherein the animal behavior storage module stores information about the at least one user-defined behavior. 
     
     
         52 . The system of  claim 43 , wherein said animal is a mouse. 
     
     
         53 . The system of  claim 43 , wherein said animal is a rat. 
     
     
         54 . A method of characterizing activity of an animal using computer processing of video images, comprising:
 detecting an animal in said video images;   tracking activity of said animal over a plurality of said video images;   classifying the activity of said animal as one of a set of classifiers based on comparing the activity over time to sets of rules of said set of classifiers;   wherein the set of classifiers comprise at least one standard classifier and   wherein one or more user-defined classifiers can be added to the set of classifiers.   
     
     
         55 . The method of  claim 54  wherein the user-defined classifier is entered into the system in the form of video images of the user-defined classifier. 
     
     
         56 . The method of  claim 54  wherein the user-defined classifier is entered into the system in the form of a set of rules. 
     
     
         57 . The method of  claim 54  further comprising:
 detecting an animal in video images of the at least one user-defined classifier; 
 tracking activity of the animal over a plurality of the video images of the at least one user-defined classifier; 
 creating user-defined set of rules based on the changes in activity of the animal over a plurality of the video images of the at least one user-defined classifier; 
 wherein the user-defined set of rules is used to classify the activity of the animal as the at least one user-defined classifier. 
 
     
     
         58 . The method of  claim 54 , further including characterizing activity by comparing the activity of the animal to the set of classifiers. 
     
     
         59 . The method of  claim 58 , wherein said characterizing activity of said animal comprises using statistical shape information. 
     
     
         60 . The method of  claim 58 , wherein said characterizing activity of said animal comprises using contour-based shape information. 
     
     
         61 . The method of  claim 58 , wherein said characterizing activity of said animal includes identifying activity of the animal as one of the set of classifiers, said set of classifiers comprising a horizontal side view classifier, a vertical classifier, a cuddled classifier, a horizontal front/back view classifier, a partially reared classifier, a stretched classifier, a hang vertical classifier, a hang cuddled classifier, an eating classifier, and a drinking classifier. 
     
     
         62 . The method of  claim 54 , further including characterizing activity by aggregating said classifiers into a behavior over a range of images. 
     
     
         63 . The method of  claim 62 , wherein the characterizing activity by aggregating said classifiers into a behavior over a range of images includes identifying patterns of classifiers over a sequence of images. 
     
     
         64 . The method of  claim 62 , wherein the aggregating of said classifiers into a behavior over a range of images includes analyzing temporal ordering of said classifiers. 
     
     
         65 . The method of  claim 64 , wherein said analyzing of temporal ordering of said classifiers is performed with the use of time-series analysis such as Hidden Markov Model (HMM). 
     
     
         66 . The method of  claim 54 , wherein said classifying the activity of said animal as one of the set of classifiers includes analyzing the statistical and contour-based shape information from the image. 
     
     
         67 . The method of  claim 54 , wherein said set of classifiers includes an unknown classifier, wherein said unknown classifier cannot be classified as one of the set of standard or user-defined classifiers. 
     
     
         68 . The method of  claim 54 , wherein the detecting is conducted under night conditions and the images are obtained using red light. 
     
     
         69 . The method of  claim 54 , wherein said detecting an animal includes detecting body parts of said animal. 
     
     
         70 . The method of  claim 69 , wherein said body parts include the forelimbs. 
     
     
         71 . A method of characterizing activity of an animal using computer processing of video images, comprising
 detecting an animal in said video images;   tracking activity of said animal over a plurality of video images;   characterizing the activity based on models or rules,   wherein the models or the rules comprise at least one standard model or standard rule;   wherein one or more user-defined models or user-defined rules can be added to the models or rules.   
     
     
         72 . The method of  claim 71  wherein said characterizing said activity comprises:
 using statistical information. 
 
     
     
         73 . The method of  claim 71  wherein said characterizing activity of the animal comprises using contour-based shape information selected from the group consisting of:
 area of said animal; 
 centroid position of said animal; 
 bounding box and aspect ratio of said bounding box of said animal; 
 eccentricity of said animal; and 
 directional orientation of said animal. 
 
     
     
         74 . The method of  claim 71  wherein characterizing activity of said animal comprises analyzing temporal information regarding the activity of the animal selected from the group consisting of direction and magnitude of movement of the centroid, increase and decrease of the eccentricity, increase and decrease of the area, increase and decrease of the aspect ratio of a bounding box, and change in contour information. 
     
     
         75 . A trainable video-based animal behavior analysis system comprising a computer configured to track an animal over time using video images, to extract features of the animal, and to use the extracted features for training the system and classification of the organism from one frame to another frame. 
     
     
         76 . The trainable video-based animal behavior analysis system of  claim 75 , wherein at least one of the extracted features is the shape of the animal. 
     
     
         77 . The trainable video-based animal behavior analysis system of  claim 75 , wherein classifiers are stored in memory in the computer for later use in analyzing the behavior of the animal. 
     
     
         78 . A trainable animal characterization system comprising a computer having an animal identification module for identifying an animal in an image, an animal tracking module for tracking the animal from one image to another, and an animal classification module for classifying the animal based upon classifiers stored in a memory of the system, wherein the classification module is configured to extract one or more features of the animal for training the classification module to identify new classifiers that are not stored in the memory, the classification module further configured to store in memory the new classifiers for use in subsequent animal characterizations. 
     
     
         79 . The trainable animal behavior analysis system of  claim 78 , wherein the stored classifiers and the new classifiers are stored in a classifier module.