Fusion of inertial and depth sensors for movement measurements and recognition

Abstract

A method of recognizing movement includes measuring, by an inertial sensor, a first unit of inertia of an object. In addition, the method includes measuring a three dimensional shape of the object. Further, the method includes receiving, by a processor, a signal representative of the measured first unit of inertia from the inertial sensor and a signal representative of the measured shape from the depth sensor. Still further, the method includes determining a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

What is claimed is: 
     
         1 . A method of recognizing movement comprising:
 measuring, by an inertial sensor, a first unit of inertia of an object;   measuring a three dimensional shape of the object;   receiving, by a processor, a signal representative of the measured first unit of inertia from the inertial sensor and a signal representative of the measured shape from the depth sensor;   determining a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.   
     
     
         2 . The method of  claim 1 , wherein the determining the type of movement comprises:
 initializing Hidden Markov model (HMM) parameters, including an HMM probability and a transition matrix, for a plurality of HMMs, each of the plurality of HMMs corresponding to a particular type of movement;   determining an observation sequence of the particular type of movement for the particular HMM being trained;   calculating a probability of the observation sequence;   performing a Baum Welch reestimation of the probability of the observation sequence to update the HMM;   calculating a likelihood of probability for each of the plurality of trained HMMs based on the signal representative of the measured first unit of inertia and the signal representative of the measured shape; and   selecting the type of movement corresponding to the trained HMM having the highest likelihood of probability.   
     
     
         3 . The method of  claim 1 , wherein the determining the type of movement comprises:
 training a first, second, and third plurality of Hidden Markov models (HMMs), each of the first plurality of HMMs corresponding to a particular type of movement for the measured first unit of inertia, each of the second plurality of HMMs corresponding to the particular type of movement for a measured second unit of inertia, and each of the third plurality of HMMs corresponding to the particular type of movement for the measured shape;   calculating a first for each of the first plurality of HMMs based on the signal representative of the measured first unit of inertia, second likelihood of probability for each of the second plurality of HMMs based on the signal representative of the measured second unit of inertia, and third likelihood of probability for each of the third plurality of HMMs based on the signal representative of the measured shape;   pooling together the first, second, and third likelihood of probabilities to generate an overall probability for each of the first, second, and third pluralities of HMMs; and   selecting the type of movement corresponding to the trained HMM having the highest overall probability.   
     
     
         4 . The method of  claim 1 , wherein the determining the type of movement comprises:
 extracting a depth feature set from the signal representative of the measured shape;   extracting a inertial feature set from the signal representative of the measured first unit of inertia; and   fusing the depth feature and the inertial feature at a decision-level.   
     
     
         5 . The method of  claim 4 , wherein the extracting the depth feature comprises:
 extracting a foreground containing the object from the signal representative of the measured shape utilizing a background subtraction algorithm to generate a foreground extracted depth image;   generating three two dimensional projected maps corresponding to a front, view of the foreground extracted depth image; and   accumulating a difference between two consecutive projected maps through an entire depth video sequence to generate a depth motion map (DMM).   
     
     
         6 . The method of  claim 4 , wherein the fusing the depth feature and the inertial feature comprises:
 applying a sparse representation classifier (SRC) or collaborative representation classifier (CRC) to the extracted depth feature set and the extracted inertial feature set to generate a first and second basic probability assignments (BPAs) respectively;   combining the first and second BPAs; and   selecting the type of movement.   
     
     
         7 . A non-transitory computer-readable medium storing instructions that when executed on a computing system cause the computing system to:
 receive a signal representative of a measured first unit of inertia from an inertial sensor coupled to an object and a signal representative of a measured shape of the object from a depth sensor; and   determine a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.   
     
     
         8 . The computer-readable medium of  claim 7 , wherein the instructions further cause the computing system to:
 train a plurality of Hidden Markov models (HMMs), each of the plurality of HMMs corresponding to a particular type of movement;   calculate a likelihood of probability for each of the plurality of trained HMMs based on the signal representative of the measured first unit of inertia and the signal representative of the measured shape; and   select the type of movement corresponding to the trained HMM having the highest likelihood of probability.   
     
     
         9 . The computer-readable medium of  claim 7 , wherein the instructions further cause the computing system to:
 train a first, second, and third plurality of Hidden Markov models (HMMs), each of the first plurality of HMMs corresponding to a particular type of movement for the measured first unit of inertia, each of the second plurality of HMMs corresponding to the particular type of movement for a measured second unit of inertia, and each of the third plurality of HMMs corresponding to the particular type of movement for the measured shape;   calculate a first for each of the first plurality of HMMs based on the signal representative of the measured first unit of inertia, second likelihood of probability for each of the second plurality of HMMs based on the signal representative of the measured second unit of inertia, and third likelihood of probability for each of the third plurality of HMMs based on the signal representative of the measured shape;   pool together the first, second, and third likelihood of probabilities to generate an overall probability for each of the first, second, and third pluralities of HMMs; and   select the type of movement corresponding to the trained HMM having the highest overall probability.   
     
     
         10 . The computer-readable medium of  claim 7 , wherein the instructions further cause the computing system to:
 extract a depth feature set from the signal representative of the measured shape;   extract a inertial feature set from the signal representative of the measured first unit of inertia; and   fuse the depth feature and the inertial feature at a decision-level.   
     
     
         11 . The computer-readable medium of  claim 7 , wherein the measured first unit of inertia comprises acceleration data of the object.