System and method for automatically discovering, characterizing, classifying and semi-automatically labeling animal behavior and quantitative phenotyping of behaviors in animals

Abstract

A method for studying the behavior of an animal in an experimental area including stimulating the animal using a stimulus device; collecting data from the animal using a data collection device; analyzing the collected data; and developing a quantitative behavioral primitive from the analyzed data. A system for studying the behavior of an animal in an experimental area including a stimulus device for stimulating the animal; a data collection device for collecting data from the animal; a device for analyzing the collected data; and a device for developing a quantitative behavioral primitive from the analyzed data. A computer implemented method, a computer system and a nontransitory computer readable storage medium related to the same. Also, a method and apparatus for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area. Further, use of a depth camera and/or a touch sensitive device related to the same.

1 - 15 . (canceled) 
     
     
         16 . A method for analyzing behavior of an animal, the method comprising:
 obtaining a three-dimensional video stream having a plurality of images of an animal, wherein images of the plurality of images comprise area and depth information;   determining contours of the animal in the images of the plurality of images;   extracting parameters from the area and depth information from the images of the plurality of images within the determined contours to provide a plurality of multi-dimensional data points related to a posture of the animal at a plurality of time points, and wherein extracting the parameters is based at least partly on an identification of the animal's head and tail; and   clustering the plurality of multi-dimensional data points to output a set of clusters that are segmented from each other so that each cluster of the set of clusters represents an animal behavior.   
     
     
         17 . The method of  claim 16 , further comprising removing, using an object detection operation, background noise from each image of the plurality of images to generate a plurality of processed images having light and dark areas, and wherein determining the contours of the animal includes determining contours of the light areas in the images of the plurality of processed images. 
     
     
         18 . The method of  claim 16 , wherein obtaining the three-dimensional video stream comprises using a three-dimensional depth camera to obtain the three-dimensional video stream. 
     
     
         19 . The method of  claim 16 , wherein determining the contours of the animal in the images of the plurality of images comprises:
 determining the contours using a contour detection algorithm; and   tracking each detected contour using a Kalman filter.   
     
     
         20 . The method of  claim 16 , further comprising:
 assigning each cluster of the set of clusters a label that represents an animal behavior.   
     
     
         21 . The method of  claim 20 , further comprising:
 outputting a visual representation of the set of clusters and corresponding labels.   
     
     
         22 . The method of  claim 20 , wherein the label for each cluster is requested by a user through a user interface after displaying video data representing each cluster in the user interface. 
     
     
         23 . The method of  claim 16 , wherein the behavior comprises a quantitative behavior primitive. 
     
     
         24 . An apparatus for analyzing behavior of an animal, the apparatus comprising:
 a data processing system comprising a computer processor and a non-transitory computer-readable storage medium storing instructions which, when executed by the computer processor, cause the computer processor to perform a method comprising:
 obtaining a three-dimensional video stream having a plurality of images of an animal, wherein images of the plurality of images comprise area and depth information; 
 determining contours of the animal in the images of the plurality of images; 
 extracting parameters from the area and depth information from the images of the plurality of images within the determined contours to provide a plurality of multi-dimensional data points related to a posture of the animal at a plurality of time points, and wherein extracting the parameters is based at least partly on an identification of the animal's head and tail; and 
 clustering the plurality of multi-dimensional data points to output a set of clusters that are segmented from each other so that each cluster of the set of clusters represents an animal behavior. 
   
     
     
         25 . The apparatus of  claim 24 , further comprising a three-dimensional depth camera configured to generate the three-dimensional video stream. 
     
     
         26 . The apparatus of  claim 24 , wherein the method further comprises removing, using an object detection operation, background noise from each image of the plurality of images to generate a plurality of processed images having light and dark areas, and wherein determining the contours of the animal includes determining contours of the light areas in the images of the plurality of processed images. 
     
     
         27 . The apparatus of  claim 24 , wherein determining the contours of the animal in the images of the plurality of images comprises:
 determining the contours using a contour detection algorithm; and   tracking each detected contour using a Kalman filter.   
     
     
         28 . The apparatus of  claim 24 , wherein the method further comprises:
 assigning each cluster of the set of clusters a label that represents an animal behavior.   
     
     
         29 . The apparatus of  claim 28 , wherein the method further comprises:
 outputting a visual representation of the set of clusters and corresponding labels.   
     
     
         30 . The apparatus of  claim 28 , wherein the label for each cluster is requested by a user through a user interface after displaying video data representing each cluster in the user interface. 
     
     
         31 . The apparatus of  claim 24 , wherein the behavior comprises a quantitative behavior primitive. 
     
     
         32 . At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for analyzing behavior of an animal, the method comprising:
 obtaining a three-dimensional video stream having a plurality of images of an animal, wherein images of the plurality of images comprise area and depth information;   determining contours of the animal in the images of the plurality of images;   extracting parameters from the area and depth information from the images of the plurality of images within the determined contours to provide a plurality of multi-dimensional data points related to a posture of the animal at a plurality of time points, and wherein extracting the parameters is based at least partly on an identification of the animal's head and tail; and   clustering the plurality of multi-dimensional data points to output a set of clusters that are segmented from each other so that each cluster of the set of clusters represents an animal behavior.   
     
     
         33 . The at least one non-transitory computer-readable storage medium of  claim 32 , wherein the method further comprises removing, using an object detection operation, background noise from each image of the plurality of images to generate a plurality of processed images having light and dark areas, and wherein determining the contours of the animal includes determining contours of the light areas in the images of the plurality of processed images. 
     
     
         34 . The at least one non-transitory computer-readable storage medium of  claim 32 , wherein determining the contours of the animal in the plurality of images comprises:
 determining the contours using a contour detection algorithm; and   tracking each detected contour using a Kalman filter.   
     
     
         35 . The at least one non-transitory computer-readable storage medium of  claim 32 , further comprising:
 assigning each cluster of the set of clusters a label that represents an animal behavior; and   outputting a visual representation of the set of clusters and corresponding labels.