Motion detection using depth images

Abstract

A sensor system creates a sequence of depth images that are used to detect and track motion of objects within range of the sensor system. A reference image is created and updated based on a moving average (or other function) of a set of depth images. A new depth images is compared to the reference image to create a motion image, which is an image file (or other data structure) with data representing motion. The new depth image is also used to update the reference image. The data in the motion image is grouped and associated with one or more objects being tracked. The tracking of the objects is updated by the grouped data in the motion image. The new positions of the objects are used to update an application. For example, a video game system will update the position of images displayed in the video based on the new positions of the objects. In one implementation, avatars can be moved based on movement of the user in front of a camera.

1 . A method for using depth images to sense motion, comprising:
 creating a reference image that includes foreground data and background data based on multiple previous depth images;   receiving a new depth image;   creating a motion image based on the new depth image and the reference image;   identifying one or more objects in the motion image;   using position information for the identified one or more objects to update an application; and   updating the reference image based on the new depth image.   
     
     
         2 . The method of  claim 1 , wherein:
 the creating the motion image includes subtracting the new depth image from the reference image to create a set of difference data, identifying difference data greater than a threshold as motion data that includes forward motion data and backward motion data, and discarding the backward motion data.   
     
     
         3 . The method of  claim 1 , wherein:
 the identifying one or more objects in the motion image includes grouping pixels of the motion image to form one or more groups of pixels, associating each of the one or more groups of pixels with one or more objects identified in object history data, and updating the object history data; and   the using position information for the identified one or more objects to update the application includes reporting the one or more objects and positions for the one or more objects to the application.   
     
     
         4 . The method of  claim 3 , wherein:
 the grouping of pixels is based on identifying connecting pixels.   
     
     
         5 . The method of  claim 3 , wherein:
 the grouping of pixels is based on proximity to previously identified objects.   
     
     
         6 . The method of  claim 1 , wherein:
 the identifying one or more objects in the motion image includes grouping pixels of the motion image to form two or more groups of pixels, associating each of the two or more groups of pixels with two or more objects identified in object history data, and updating the object history data; and   the using position information for the identified one or more objects to update the application includes reporting the two or more objects and positions for the two or more objects to the application.   
     
     
         7 . The method of  claim 1 , wherein:
 the identifying one or more objects in the motion image includes grouping pixels of the motion image to an initial group of pixels, determining that the initial group of pixels represents two objects, splitting the initial group of pixels into a first group of pixels and a second group of pixels, associating the first group of pixels with a first object based on object history data, and associating the second group of pixels with a second object based on object history data.   
     
     
         8 . The method of  claim 1 , wherein:
 the identifying one or more objects in the motion image includes grouping pixels of the motion image to form two or more groups of pixels, predicting a trajectory of an object and identifying the group of pixels closest to the predicted trajectory.   
     
     
         9 . The method of  claim 1 , wherein:
 the identifying one or more objects in the motion image includes using object history data and structure information about items in real space in order to identify the one or more objects.   
     
     
         10 . The method of  claim 1 , wherein:
 the updating the reference image includes performing a weighted averaging function on the reference image and the new depth image; and   the creating the reference image include performing the weighted averaging function on previous depth images.   
     
     
         11 . The method of  claim 1 , wherein:
 the using position information for the identified one or more objects to update the application includes changing the position of an image in a video display in response to the position information.   
     
     
         12 . The method of  claim 1 , wherein:
 the creating the motion image includes subtracting the new depth image from the reference image to create a set of difference data, identifying difference data greater than a threshold as motion data that includes forward motion data and backward motion data, and discarding the backward motion data;   the identifying one or more objects in the motion image includes grouping pixels of the motion image to form one or more groups of pixels, associating each of the one or more groups of pixels with one or more objects identified in object history data, and updating the object history data;   the using position information for the identified one or more objects to update the application includes reporting the one or more objects and positions for the one or more objects to the application; and   the using position information for the identified one or more objects to update the application includes changing the position of an image in a video display in response to the position information.   
     
     
         13 . An apparatus that uses depth images to sense motion, comprising:
 a communication interface that receives depth images;   one or more storage devices that store depth images;   a display interface; and   one or more processors in communication with the one or more storage devices and the display interface, the one or more processors access a new depth image received from the communication interface and identify motion based comparing the new depth image to a reference image stored in the one or more storage devices, the one or more processors create a motion image representing identified motion, the one or more processors group pixels of the motion image and associate one or more groups of pixels with one or more objects identified in object history data stored in the one or more storage devices, the one or more processors use position information for the identified one or more objects to update an application running on the apparatus and provide signals on the display interface that indicate the update to the application.   
     
     
         14 . The apparatus of  claim 13 , wherein:
 the display interface connects to a video monitor;   the one or more processors use the position information for the identified one or more objects to update a position of an image on the video monitor; and   the image is displayed by the application.   
     
     
         15 . The apparatus of  claim 13 , wherein:
 the reference image includes foreground and background data based on multiple previous depth images.   
     
     
         16 . The apparatus of  claim 13 , wherein:
 the one or more processors determine that a particular group of pixels represents two objects, split the particular group of pixels into a first group of pixels and a second group of pixels, associate the first group of pixels with a first object based on object history data, and associate the second group of pixels with a second object based on object history data.   
     
     
         17 . The apparatus of  claim 13 , wherein:
 the one or more processors also use structure information about items in real space in order to associate the one or more groups of pixels with the one or more objects.   
     
     
         18 . A method for using depth images to sense motion, comprising:
 receiving a new depth image;   identifying motion based on comparing the new depth image to a reference image;   creating a motion image representing identified forward motion, and discarding identified backward motion when creating the motion image;   identifying one or more objects in the motion image; and   reporting the identified one or more objects in the motion image.   
     
     
         19 . The method of  claim 18 , further comprising:
 creating the reference image that includes foreground data and background data based on multiple previous depth images.   
     
     
         20 . The method of  claim 18 , wherein:
 the reporting the identified one or more objects includes reporting objects and positions of the objects to an application that uses the position to create video.