Scalable real-time motion recognition

Abstract

Human body motion is represented by a skeletal model derived from image data of a user. The model represents joints and bones and has a rigid body portion. The sets of body data are scaled to a predetermined number of sets for a number of periodic units. A body-based coordinate 3-D reference system having a frame of reference defined with respect to a position within the rigid body portion of the skeletal model is generated. The body-based coordinate 3-D reference system is independent of the camera's field of view. The scaled data and representation of relative motion within an orthogonal body-based 3-D reference system decreases the data and simplifies the calculations for determining motion thus enhancing real-time performance for multimedia applications controlled by a user's natural movements.

What is claimed is: 
     
         1 . A method of scalable real-time motion recognition and/or similarity analysis of human body motion based on skeletal model data derived from image data of a user comprising:
 (a) receiving skeletal data from a capture device relating to the position of a user within a scene, the data represented in a 3-D reference system of the capture device;   (b) transforming the data into a body-based coordinate 3-D reference system having a frame of reference of the user, the body-based coordinate 3-D reference system including a plurality of joints described as a single rigid body; and   (c) outputting a computer model of the user based in part on body-based coordinate 3-D reference system.   
     
     
         2 . The method of  claim 1 , further comprising the step (d) of temporally scaling the received skeletal data received in said step (a) by synchronizing the skeletal data to a predetermined number of sets for a number of periodic units wherein the periodic unit is rhythmic. 
     
     
         3 . The method of  claim 2 , wherein the periodic unit is a beat of music. 
     
     
         4 . The method of  claim 1  wherein said step (b) of translating the data into a body-based coordinate 3-D reference system having a frame of reference of the user comprises the step of obtaining three orthogonal axes for the body-based 3-D coordinate reference system by performing a principal component analysis on a set of data points representing the rigid body portion of the skeletal model. 
     
     
         5 . The method of  claim 1 , further comprising the step of translating the skeletal data from a body-based coordinate 3-D reference system to a 3-D camera space system, independent of the camera based on a respective transformation angle defined to transform each of the axes of one system to an axis of the other system. 
     
     
         6 . The method of  claim 5  wherein the model of the human body is a skeleton model of joints and bones including:
 a torso as the rigid body portion, 
 a set of first degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the rigid body torso, and 
 a set of second degree joints in which each respective joint is attached to a bone, another end of which is attached to one of the joints in the set of first degree joints. 
 
     
     
         7 . The method of  claim 6  further comprising representing motion of a joint in the set of first degree joints attached to the rigid body portion in a skeletal data frame by:
 translating the frame of reference for the 3-D body-based coordinate system from an origin of the system on the rigid body to a joint of the rigid body to which an end of a bone attached to the first degree joint is attached; 
 determining the position of the first degree joint in the data frame using spherical coordinate system angles with respect to the body-based 3-D reference system; 
 and storing the position of the first degree joint in the skeletal data frame as the spherical coordinate system angles. 
 
     
     
         8 . The method of  claim 1 , further comprising the step of comparing the output in said step (c) to an oracle of a predefined motion to determine if the output matches the predefined motion above some confidence threshold. 
     
     
         9 . The method of  claim 8 , further comprising the step of determining on a per joint basis a distance between a joint indicated in the output of said step (c) and a corresponding joint in the oracle. 
     
     
         10 . A system for recognizing human motion from skeletal data derived from image data comprising:
 a camera for repeatedly capturing image data sets of a human body;   one or more processors communicatively coupled to the camera for receiving the image data sets and having access to a memory for storing the body data sets;   the one or more processors executing software for representing the human body in the image data sets as a human skeleton model of joints;   the one or more processors executing software for generating a body-based coordinate 3-D reference system having a frame of reference defined with respect to a position of a plurality of joints grouped together as a single rigid body;   the one or more processors executing software for determining positions of first degree joints, adjacent the single rigid body, in relation to a position of the single rigid body;   the one or more processors executing software for determining positions of second degree joints, adjacent the first degree joints, in relation to positions of the first degree joints; and   the one or more processors executing software for determining a motion of at least one body part using the body-based 3-D coordinate reference system and the determined positions of the first and second degree joints.   
     
     
         11 . The system of  claim 10  further comprising:
 the one or more processors executing software for determining the relationship of the body-based 3-D coordinate system to the fixed camera-based 3-D coordinate system for each set of the scaled skeletal data 
 
     
     
         12 . The system of  claim 11  further comprising the one or more processors executing software for scaling the received skeletal data by synchronizing the sets of skeletal data to a predetermined number of sets for a number of periodic units. 
     
     
         13 . The system of  claim 10  wherein the model of the human body is a skeleton model of joints and bones including:
 a torso as the rigid body portion, 
 a set of first degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the rigid body, and 
 a set of second degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the set of first degree joints. 
 
     
     
         14 . The system of  claim 13  wherein determining a motion of at least one body part using the body-based 3-D coordinate reference system further comprises defining for each skeletal data frame
 a set of three transformation angles for transforming the rigid body portion from the rigid body-based 3-D coordinate reference system to the fixed camera-based 3-D coordinate reference system; 
 and spherical coordinate system angles determined for each joint to represent its position using the rigid body-based 3-D coordinate reference system. 
 
     
     
         15 . One or more computer readable storage media having encoded thereon instructions for causing at least one processor to perform a method for recognizing a gesture from image data, the method comprising:
 (a) receiving sets of skeletal data representing a human body in a fixed camera-based three-dimensional (3-D) coordinate reference system having a frame of reference defined with respect to a point in a camera's field of view;   (b) conforming the sets of skeletal data received in said step (a) to a body-based coordinate 3-D reference system having a frame of reference defined with respect to a position within a rigid body portion of a skeletal model, the body-based coordinate 3-D reference system being independent of the camera's field of view;   (c) representing the body-based coordinate 3-D reference system by one of: i) a plurality of angles that rotate the skeletal data back into the fixed camera-based 3-D coordinate reference system and ii) an orientation matrix;   (d) determining a motion of at least one body part using the plurality of angles that rotate the skeletal data back into the fixed camera-based 3-D coordinate reference system; and   (e) determining whether a gesture has been made by at least one body part based on the determined motion of the at least one body part.   
     
     
         16 . The one or more computer readable storage media of  claim 15  wherein the human skeleton model of joints and bones includes:
 a torso as the rigid body portion, 
 a set of first degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the rigid body torso, and 
 a set of second degree joints in which each respective joint is attached to a bone, another end of which is attached to one of the joints in the set of first degree joints. 
 
     
     
         17 . The one or more computer readable storage media of  claim 16 , wherein determining a motion of at least one body part using the body-based 3-D coordinate reference system further comprises:
 representing motion of a joint in the set of first degree joints attached to the rigid body portion in a skeletal data frame by:
 translating the frame of reference for the 3-D body-based coordinate system from an origin of the system on the rigid body to a joint of the rigid body to which an end of a bone attached to the first degree joint is attached; 
 determining the position of the first degree joint in the data frame using spherical coordinate system angles with respect to the body-based 3-D reference system; and 
 storing the position of the first degree joint in the skeletal data frame as the spherical coordinate system angles. 
   
     
     
         18 . The one or more computer readable storage media of  claim 17  wherein determining a motion of at least one body part using the body-based 3-D reference system further comprises:
 representing motion of a joint in the set of second degree joints in a skeletal data frame by
 translating the torso rigid body-based 3-D orthogonal coordinate reference system to a first degree joint to which an end of a bone attached to the second degree joint is attached; 
 generating a first axis of a 3-D orthogonal body-based coordinate reference system with its origin at the first degree joint, the first axis being a bone vector generated by extending a bone attached to a joint of the rigid body at one end and which is attached at the other end to the first degree joint; 
 generating the second 3-D orthogonal coordinate system including the bone vector; 
 defining the bone vector as spherical north; 
 determining the position of the second degree joint in terms of spherical coordinate system angles defined with respect to the body-based 3-D reference system centered at the first degree joint; and 
 storing the position of the second degree joint in the skeletal data frame as the spherical coordinate system angles in a motion determination data set for the second degree joint. 
 
 
     
     
         19 . The one or more computer readable storage media of  claim 18  wherein the motion determination data set for each set of scaled skeletal data includes:
 a set of three transformation angles for transforming the rigid body portion from the rigid body-based 3-D coordinate reference system to the fixed camera-based 3-D coordinate reference system; and 
 the spherical coordinate system angles determined for each joint to represent its position. 
 
     
     
         20 . The one or more computer readable storage media of  claim 19  wherein determining whether a gesture has been made by the at least one body part based on the motion of the at least one body part further comprises:
 generating a set of candidate gestures by determining correlations between data items of the motion determination data sets with model poses representing different gestures; 
 selecting a match gesture based on a criteria; and 
 determining a score representing a deviation of the user's gesture from a gesture standard.