Machine learning based activity detection utilizing reconstructed 3d arm postures

Abstract

A method comprises obtaining data from an inertial measurement unit of a user, generating three-dimensional (3D) arm pose estimates from the obtained data, applying the generated 3D arm pose estimates to a machine learning system trained to recognize temporal-spatial patterns of one or more designated activities, and obtaining at least one classification output from the machine learning system. The machine learning system illustratively comprises at least one support vector machine (SVM) model. Applying the generated 3D arm pose estimates to the machine learning system illustratively comprises extracting possible intake gestures of the generated 3D arm pose estimates into respective segments, resampling each of at least a subset of the extracted segments, and utilizing the SVM model to classify whether or not each of one or more of the extracted and resampled segments comprises an intake gesture.

What is claimed is: 
     
         1 . A method comprising:
 obtaining data from an inertial measurement unit of a user;   generating three-dimensional arm pose estimates from the obtained data;   applying the generated three-dimensional arm pose estimates to a machine learning system trained to recognize temporal-spatial patterns of one or more designated activities; and   obtaining at least one classification output from the machine learning system;   wherein the method is performed by at least one processing device comprising a processor coupled to a memory.   
     
     
         2 . The method of  claim 1  wherein the inertial measurement unit is part of a wearable device of the user. 
     
     
         3 . The method of  claim 1  wherein the at least one processing device comprises a wearable device of the user. 
     
     
         4 . The method of  claim 1  wherein the at least one processing device comprises a smartphone of the user that is in communication with a wearable device of the user, the wearable device comprising the inertial measurement unit. 
     
     
         5 . The method of  claim 1  wherein the one or more designated activities comprise eating activities. 
     
     
         6 . The method of  claim 1  wherein the machine learning system is trained on one or more sets of three-dimensional arm pose estimate training data using supervised learning. 
     
     
         7 . The method of  claim 1  further comprising generating at least one control signal based at least in part on the classification output to guide the user toward one or more target activities. 
     
     
         8 . The method of  claim 7  wherein the one or more target activities comprise one or more activities associated with a particular desired health condition of the user. 
     
     
         9 . The method of  claim 1  wherein a given one of the three-dimensional arm pose estimates comprises a combination of an estimate of a wrist orientation relative to a body and an estimate of an elbow orientation relative to the body. 
     
     
         10 . The method of  claim 1  wherein obtaining data from an inertial measurement unit of a user comprises obtaining wrist orientation data from the inertial measurement unit. 
     
     
         11 . The method of  claim 1  wherein generating three-dimensional arm pose estimates from the obtained data comprises, for each of a plurality of body directions, calculating wrist orientation relative to a torso coordinate system using wrist orientation relative to an earth coordinate system and the body direction, looking up the calculated wrist orientation relative to the torso coordinate system in a weighted dictionary generated using actual arm poses to determine a weighted point cloud, and utilizing the weighted point cloud to assign a probability to the body direction based at least in part on weights of the weighted point cloud. 
     
     
         12 . The method of  claim 11  wherein generating three-dimensional arm pose estimates from the obtained data further comprises:
 selecting a particular one of the body directions based at least in part on their respective assigned probabilities; and 
 utilizing the selected body direction to transform the wrist orientation relative to the earth coordinate system to determine a derived wrist orientation relative to the torso coordinate system. 
 
     
     
         13 . The method of  claim 12  wherein generating three-dimensional arm pose estimates from the obtained data further comprises:
 looking up the derived wrist orientation relative to the torso coordinate system in the weighted dictionary to determine a weighted point cloud; and 
 determining a given one of the three-dimensional arm pose estimates based at least in part on weights of the weighted point cloud. 
 
     
     
         14 . The method of  claim 1  wherein the machine learning system comprises at least one support vector machine (SVM) model, and wherein applying the generated three-dimensional arm pose estimates to the machine learning system activities comprises:
 extracting possible intake gestures of the generated three-dimensional arm pose estimates into respective segments; 
 resampling each of at least a subset of the extracted segments; and 
 utilizing the SVM model to classify whether or not each of one or more of the extracted and resampled segments comprises an intake gesture. 
 
     
     
         15 . A system comprising:
 at least one processing device comprising a processor coupled to a memory;   the processing device being configured:   to obtain data from an inertial measurement unit of a user;   to generate three-dimensional arm pose estimates from the obtained data;   to apply the generated three-dimensional arm pose estimates to a machine learning system trained to recognize temporal-spatial patterns of one or more designated activities; and   to obtain at least one classification output from the machine learning system.   
     
     
         16 . The system of  claim 15  wherein generating three-dimensional arm pose estimates from the obtained data comprises:
 for each of a plurality of body directions, calculating wrist orientation relative to a torso coordinate system using wrist orientation relative to an earth coordinate system and the body direction, looking up the calculated wrist orientation relative to the torso coordinate system in a weighted dictionary generated using actual arm poses to determine a weighted point cloud, and utilizing the weighted point cloud to assign a probability to the body direction based at least in part on weights of the weighted point cloud; 
 selecting a particular one of the body directions based at least in part on their respective assigned probabilities; 
 utilizing the selected body direction to transform the wrist orientation relative to the earth coordinate system to determine a derived wrist orientation relative to the torso coordinate system; 
 looking up the derived wrist orientation relative to the torso coordinate system in the weighted dictionary to determine a weighted point cloud; and 
 determining a given one of the three-dimensional arm pose estimates based at least in part on weights of the weighted point cloud. 
 
     
     
         17 . The system of  claim 15  wherein the machine learning system comprises at least one support vector machine (SVM) model, and wherein applying the generated three-dimensional arm pose estimates to the machine learning system comprises:
 extracting possible intake gestures of the generated three-dimensional arm pose estimates into respective segments; 
 resampling each of at least a subset of the extracted segments; and 
 utilizing the SVM model to classify whether or not each of one or more of the extracted and resampled segments comprises an intake gesture. 
 
     
     
         18 . A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code, when executed by at least one processing device comprising a processor coupled to a memory, causes the processing device:
 to obtain data from an inertial measurement unit of a user;   to generate three-dimensional arm pose estimates from the obtained data;   to apply the generated three-dimensional arm pose estimates to a machine learning system trained to recognize temporal-spatial patterns of one or more designated activities; and   to obtain at least one classification output from the machine learning system.   
     
     
         19 . The computer program product of  claim 18  wherein generating three-dimensional arm pose estimates from the obtained data comprises:
 for each of a plurality of body directions, calculating wrist orientation relative to a torso coordinate system using wrist orientation relative to an earth coordinate system and the body direction, looking up the calculated wrist orientation relative to the torso coordinate system in a weighted dictionary generated using actual arm poses to determine a weighted point cloud, and utilizing the weighted point cloud to assign a probability to the body direction based at least in part on weights of the weighted point cloud; 
 selecting a particular one of the body directions based at least in part on their respective assigned probabilities; 
 utilizing the selected body direction to transform the wrist orientation relative to the earth coordinate system to determine a derived wrist orientation relative to the torso coordinate system; 
 looking up the derived wrist orientation relative to the torso coordinate system in the weighted dictionary to determine a weighted point cloud; and 
 determining a given one of the three-dimensional arm pose estimates based at least in part on weights of the weighted point cloud. 
 
     
     
         20 . The computer program product of  claim 18  wherein the machine learning system comprises at least one support vector machine (SVM) model, and wherein applying the generated three-dimensional arm pose estimates to the machine learning system comprises:
 extracting possible intake gestures of the generated three-dimensional arm pose estimates into respective segments; 
 resampling each of at least a subset of the extracted segments; and 
 utilizing the SVM model to classify whether or not each of one or more of the extracted and resampled segments comprises an intake gesture.