Detecting emotions from micro-expressive free-form movements

Abstract

Computerized methods and systems, including computer programs encoded on a computer storage medium, may adaptively predict expression of emotions based on collected biometric data. For example, a computing system may receive first data indicative of a first time-evolving movement of a portion of a body during a collection period, and may obtain second data identifying predictive models that correlate default emotions with second feature values that characterize body movements during prior collection periods. Based on an outcome of the application of the least one pattern recognition algorithm or machine learning algorithm to portions of the first and second data, the system may determine a corresponding one of the default emotions represented by the first time-evolving movement, and may transmit data indicative of the corresponding one of the default emotions to the communications device for presentation to a user.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 receiving, by a computing device, data representing a time-evolving movement of a portion of a body during a collection period from one or more devices associated with the body;   generating, by the computing device, feature data that characterizes the data representing the time-evolving movement of the portion of the body during distinct times within the collection period;   determining, by the computing device, one or more existing predictive models whose inputs correspond to the generated feature data, wherein the determined one or more existing predictive models correspond to a plurality of specified emotions;   applying, by the computing device, a machine learning algorithm to the generated feature data and to the determined one or more existing predictive models;   in response to applying the machine learning algorithm to the generated feature data and to the determined one or more existing predictive models, identifying, by the computing device, a particular emotion of the plurality of specified emotions as indicative of the time-evolving movement of the portion of the body by; and   providing, by the computing device, output data corresponding to the particular emotion.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein receiving data representing the time-evolving movement of the portion of the body during the collection period further comprises:
 receiving, by the computing device, data representing the time-evolving movement of the portion of the body that touches a surface of the one or more devices associated with the body during the collection period.   
     
     
         3 . The computer-implemented method of  claim 1 , wherein receiving data representing the time-evolving movement of the portion of the body during the collection period further comprises:
 receiving, by the computing device, data representing the time-evolving movement of the portion of the body that attaches to the one or more devices during the collection period.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein receiving data representing the time-evolving movement of the portion of the body during the collection period further comprises:
 receiving, by the computing device, data representing the time-evolving movement of the portion of the body that distantly holds the one or more devices during the collection period.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein the one or more devices comprises at least one of a camera, a sensor, a client device, a mouse, a chair, a smart watch, a smart ring, or a smart fabric. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the portion of the body comprises at least one of a hand, a foot, an arm, a hand, a finger, or a whole body. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein receiving the data representing the time-evolving movement of a portion of the body during the collection period further comprises:
 receiving, by the computing device, data that represents at least one of:
 two-dimensional positions of contact between the portion of the body and a surface of a device of the one or more devices during the collection period, 
 a movement of the portion of the body through an environment during the collection period, or 
 three-dimensional positional movements at discrete collection times during the collection period. 
   
     
     
         8 . The computer-implemented method of  claim 7 , wherein generating the feature data that characterizes the data representing the time-evolving movement of the portion of the body during the distinct times within the collection period further comprises:
 generating, by the computing device, differences in the two-dimensional positions of contact position and applied pressure between successive pairs of the distinct times within the collection period; and   based on the generated differences, deriving, by the computing device, the feature data that characterizes the data representing the time-evolving movement of the portion of the body,   wherein the feature data comprises at least one of (i) a speed of the portion of the body at each of the distinct times, (ii) an acceleration of the portion of the body at each of the distinct times, (iii) a displacement of the portion of the body at each of the distinct times, (iv) a direction of the movement of the portion of the body at each of the distinct times, (v) a change in calibrated pressure during successive pairs of the distinct times, (vi) an area of contact between the portion of the body and a surface of a device from the one or more devices, or (v) a shape of a movement of the portion of the body along the surface of the device of the one or more devices.   
     
     
         9 . The computer-implemented method of  claim 1 , wherein the one or more existing predictive models correlate each of the plurality of specified emotions with ranges of second feature values that characterize time-evolving body movements during prior collection periods, and wherein identifying the particular emotion as indicative of the time-evolving movement of the portion of the body further comprises:
 determining, by the computing device, a level of intensity of the particular emotion in response to applying the machine learning algorithm to the generated feature data and to the determined one or more existing predictive models; and   based on the determination of the particular emotion represented by the time-evolving movement of the portion of the body and the level of intensity of the particular emotion, establishing, by the computing device, that the generated feature data correspond to a portion of features of the second feature values, the portion of the features corresponding to one of the particular emotions.   
     
     
         10 . The computer-implemented method of  claim 9 , where providing the output data corresponding to the particular emotion further comprises:
 generating, by the computing device, interface data that represents the particular emotion and the level of intensity of the particular emotion, wherein the interface data comprises at least one of: (i) textual data that identifies the particular emotion represented by the time-evolving movement of the portion of the body and the level of intensity of the particular emotion, (ii) layout data specifying a position of the textual data within a graphical user interface (GUI) presented by a client device, or (iii) interface characteristics that enable a user associated with the client device to visually perceive the particular emotion within the GUI presented by the client device.   
     
     
         11 . The computer-implemented method of  claim 10 , wherein the interface characteristics comprise at least one of text color of the GUI presented by the client device, background color of the GUI presented by the client device, or interface skin of the GUI presented by the client device. 
     
     
         12 . The computer-implemented method of  claim 1 , further comprising:
 receiving, by the computing device, a request from a first user to access a graphical user interface displaying information on identified emotions of a second user;   providing, by the computing device to a device associated with the first user, the graphical user interface displaying information on the identified emotions of the second user;   receiving, by the computing device, information indicative of pleasant engagement of the first user responsive to the identified emotions of the second user; and   generating, by the computing device, a second graphical user interface displaying a plurality of icons representing one or more metrics that provide information indicative of the pleasant engagement of the first user responsive to the identified emotions of the second user.   
     
     
         13 . The computer-implemented method of  claim 1 , further comprising:
 receiving, by the computing device, a request from a first user to access a graphical user interface displaying information on identified emotions of a second user;   providing, by the computing device to a device associated with the first user, the graphical user interface displaying information on the identified emotions of the second user to a device associated with the first user;   receiving, by the computing device, information indicative of thoughtful feedback of the first user responding to the identified emotions of the second user; and   generating, by the computing device, a second graphical user interface displaying a plurality of icons representing one or more metrics that provide information indicative of the thoughtful feedback of the first user responsive to the identified emotions of the second user.   
     
     
         14 . The computer-implemented method of  claim 1 , wherein each of the one or more existing predictive models corresponds to a distinct emotion of the plurality of specified emotions. 
     
     
         15 . A system comprising:
 one or more computers; and   one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
 receiving, by a computing device, data representing a time-evolving movement of a portion of a body during a collection period from one or more devices associated with the body; 
 generating, by the computing device, feature data that characterizes the data representing the time-evolving movement of the portion of the body during distinct times within the collection period; 
 determining, by the computing device, one or more existing predictive models whose inputs correspond to the generated feature data, wherein the determined one or more existing predictive models correspond to a plurality of specified emotions; 
 applying, by the computing device, a machine learning algorithm to the generated feature data and to the determined one or more existing predictive models; 
 in response to applying the machine learning algorithm to the generated feature data and to the determined one or more existing predictive models, identifying, by the computing device, a particular emotion of the plurality of specified emotions as indicative of the time-evolving movement of the portion of the body by; and 
 providing, by the computing device, output data corresponding to the particular emotion. 
   
     
     
         16 . The system of  claim 15 , wherein receiving data representing the time-evolving movement of the portion of the body during the collection period further comprises:
 receiving, by the computing device, data representing the time-evolving movement of the portion of the body that touches a surface of the one or more devices associated with the body during the collection period.   
     
     
         17 . The system of  claim 15 , wherein receiving data representing the time-evolving movement of the portion of the body during the collection period further comprises:
 receiving, by the computing device, data representing the time-evolving movement of the portion of the body that attaches to the one or more devices during the collection period.   
     
     
         18 . The system of  claim 15 , wherein receiving data representing the time-evolving movement of the portion of the body during the collection period further comprises:
 receiving, by the computing device, data representing the time-evolving movement of the portion of the body that distantly holds the one or more devices during the collection period.   
     
     
         19 . The system of  claim 15 , wherein the one or more devices comprises at least one of a camera, a sensor, a client device, a mouse, a chair, a smart watch, a smart ring, or a smart fabric. 
     
     
         20 . A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
 receiving, by a computing device, data representing a time-evolving movement of a portion of a body during a collection period from one or more devices associated with the body;   generating, by the computing device, feature data that characterizes the data representing the time-evolving movement of the portion of the body during distinct times within the collection period;   determining, by the computing device, one or more existing predictive models whose inputs correspond to the generated feature data, wherein the determined one or more existing predictive models correspond to a plurality of specified emotions;   applying, by the computing device, a machine learning algorithm to the generated feature data and to the determined one or more existing predictive models;   in response to applying the machine learning algorithm to the generated feature data and to the determined one or more existing predictive models, identifying, by the computing device, a particular emotion of the plurality of specified emotions as indicative of the time-evolving movement of the portion of the body by; and   providing, by the computing device, output data corresponding to the particular emotion.