Adjusting mental state to improve task performance

Abstract

A method of adjusting mental state includes acquiring video data of an individual, extracting image data and audio data from the video data, extracting semantic text data from the audio data, identifying a first set of features, predicting a baseline mental state, identifying a target mental state, and simulating a predicted path from the baseline mental state to the target mental state. The baseline mental state is predicted based on the first set of features. The predicted path is simulated using a multidimensional mental state model, a plurality of actions, and a first computer-implemented machine learning model. The predicted path comprises one or more actions of the plurality of actions and corresponding changes to at least one of first and second dimensions of the multidimensional mental state model. An indication of the one or more actions of the predicted path is output to the individual.

1 . A method comprising:
 acquiring video data of an individual;   extracting image data and audio data from the video data;   extracting semantic text data from the audio data;   analyzing at least one of the image data, the audio data, and the semantic text data to identify a first set of features;   predicting a baseline mental state of the individual based on the first set of features, wherein:
 the baseline mental state comprises a first mental state value and a second mental state value; 
 the first mental state value corresponds to a first dimension of a multidimensional mental state model; and 
 the second mental state value corresponds to a second dimension of the multidimensional mental state model; 
   identifying a target mental state based on a task performed by the individual in the video data, wherein:
 the target mental state comprises a third mental state value and a fourth mental state value; 
 the third mental state value corresponds to the first dimension of the multidimensional mental state model; and 
 the fourth mental state value corresponds to the second dimension of the multidimensional mental state model; 
   simulating, by a simulator, a predicted path from the baseline mental state toward the target mental state using the multidimensional mental state model, a plurality of actions, and a first computer-implemented machine learning model, wherein:
 the first computer-implemented machine learning model is configured to relate actions of the plurality of actions and changes in value in at least one of the first dimension and the second dimension of the multidimensional mental state model; 
 the predicted path comprises a first set of actions of the plurality of actions and corresponding changes to at least one of the first dimension and the second dimension of the multidimensional mental state model; and 
 the first set of actions includes one or more actions that are performable by the individual; 
   generating a second set of actions based on the first set of features and a relational feature model, wherein:
 the relational feature model relates features and performance of the task; and 
 the second set of actions comprises one or more actions that are performable by the first individual to improve task performance; 
   outputting an indication of the first set of actions to the individual; and   outputting an indication of the second set of actions to the individual.   
     
     
         2 . The method of  claim 1 , wherein predicting the baseline mental state comprises:
 generating, by a second computer-implemented machine learning model, the first mental state value based on the first set of features; and   generating, by a third computer-implemented machine learning model, the second mental state value based on the first set of features.   
     
     
         3 . The method of  claim 3 , wherein the first dimension describes an intensity of a first mental state and the second dimension describes a pleasantness of the first mental state. 
     
     
         4 . The method of  claim 3 , wherein the first dimension describes an intensity of the first mental state, a pleasantness of the first mental state, an importance of information conveyed by the individual, a positivity of the conveyed information, or a subject of the conveyed information. 
     
     
         5 . The method of  claim 3 , wherein the first dimension describes a first mental state, the second dimension describes a second mental state, and the first mental state and the second mental state are selected from a group consisting of tiredness, sleepiness, serenity, satisfaction, calmness, relaxation, contentment, distress, frustration, anger, annoyance, tension, fear, alarm, misery, sadness, depression, gloom, boredom, astonishment, amusement, excitement, happiness, delight, gladness, pleasure, thankfulness, gratitude, confusion, smugness, deliberation, anticipation, cheer, sympathy, trust, humor, envy, melancholy, hostility, resentment, revulsion, and ennui. 
     
     
         6 . The method of  claim 1 , wherein:
 analyzing at least one of the image data, the audio data, and the semantic text data to identify the first set of features comprises:
 analyzing the image data to identify the first set of features; 
 analyzing the to identify a second set of features; and 
 analyzing the semantic text data to identify a third set of features; and 
   the second set of actions is based on the first set of features, the second set of features, the third set of features, and the relational feature model   
     
     
         7 . The method of  claim 6 , wherein:
 the one or more actions of the first set of actions are performable by the individual before, during, or before and during a second iteration of the task to adjust the baseline mental state; and   the one or more actions of the second set of actions are performable by the individual during the second iteration of the task to adjust task performance.   
     
     
         8 . The method of  claim 7 , wherein predicting the baseline mental state comprises:
 generating, by a second computer-implemented machine learning model, the first mental state value based on at least one of the first set of features, the second set of features, and the third set of features; and   generating, by a third computer-implemented machine learning model, the second mental state value based on at least one of the first set of features, the second set of features, and the third set of features.   
     
     
         9 . The method of  claim 8 , wherein the second set of actions comprises at least one of a recommended body language adjustment, a recommended vocal tone adjustment, and one or more recommended spoken words. 
     
     
         10 . The method of  claim 9 , wherein the task is acting, lying, lecturing, public speaking, or teaching. 
     
     
         11 . The method of  claim 10 , wherein predicting the baseline mental state comprises:
 generating, by a second computer-implemented machine learning model, the first mental state value based on the first set of features and the second set of features; and   generating, by a third computer-implemented machine learning model, the second mental state value based on the first set of features and the second set of features.   
     
     
         12 . The method of  claim 11 , wherein:
 the baseline mental state comprises a fifth mental state value corresponding to a third dimension of the multidimensional mental state model;   the target mental state comprises a sixth mental state value corresponding to the third dimension of the multidimensional mental state model;   the first computer-implemented machine learning model is configured to relate actions of the plurality of actions and changes in value in at least one of the first dimension, the second dimension, and the third dimension of the multidimensional mental state model; and   the predicted path comprises one or more actions of the plurality of actions and corresponding changes to at least one of the first dimension, the second dimension, and the third dimension of the multidimensional mental state model.   
     
     
         13 . The method of  claim 10 , wherein predicting the baseline mental state further comprises generating, by a fourth computer-implemented machine learning model, the fifth mental state value based on the third set of features. 
     
     
         14 . The method of  claim 11 , wherein the first dimension describes an intensity of a first mental state, the second dimension describes a pleasantness of the first mental state, and the third dimension describes an importance of information conveyed by the individual. 
     
     
         15 . The method of  claim 12 , wherein:
 the second computer-implemented machine learning model is configured to relate intensity of the first mental state with features of the first set of features and the second set of features;   the third computer-implemented machine learning model is configured to relate pleasantness of the first mental state with features of the first set of features and the second set of features; and   the fourth computer-implemented machine learning model is configured to relate an importance of information conveyed by the individual with features of the third set of features.   
     
     
         16 . The method of  claim 15 , further comprising generating, before acquiring video of the first individual, the relational feature model using a fifth computer-implemented machine learning model. 
     
     
         17 . The method of  claim 16 , wherein generating the relational feature model comprises:
 generating a training set of video data, wherein the labeled training video data depicts preferred task performance;   generating a training set of features from the training set of video data;   training the fifth computer-implemented machine learning model with the training set of features;   selecting features of the training set of features having, as determined by the trained fifth computer-implemented machine learning model, predictive accuracy above a pre-determined threshold; and   generating the relational feature model based on the selected features and the predictive accuracies of the selected features.   
     
     
         18 . The method of  claim 17 , wherein:
 analyzing the image data to identify the first set of features comprises analyzing the image data with a computer vision model;   analyzing the audio data to identify the second set of features comprises converting the audio data to a spectrogram and analyzing the spectrogram with a sixth computer-implemented machine learning model; and   analyzing the semantic text data to identify the third set of features comprises analyzing the semantic text data with a natural language understanding model.   
     
     
         19 . The method of  claim 1 , wherein the simulating the predicted path comprises:
 generating a first plurality of intermediate points based on the changes in value in at least one of the first dimension and the second dimension and the baseline mental state, wherein each intermediate point of the first plurality of intermediate points corresponds to an action of the one or more actions;   measuring a first plurality of Euclidean distances between the first plurality of intermediate points and the target mental state;   determining a first preferred intermediate point of the first plurality of intermediate points, the first preferred intermediate point having a shortest Euclidean distance of the first plurality of Euclidean distances to the target mental state; and   storing, as a first step of the predicted path, the change in value in at least one of the first dimension and the second dimension used to generate the first preferred intermediate point and the corresponding action of the one or more actions.   
     
     
         20 . A system for adjusting task performance, the system comprising:
 a processor;   a user interface; and   a memory encoded with instructions that, when executed, cause the processor to:
 acquire video data of an individual; 
 extract image data and audio data from the video data; 
 extract semantic text data from the audio data; 
 analyze at least one of the image data, the audio data, and the semantic text data to identify a first set of features; 
 predict a baseline mental state of the individual based on the first set of features, wherein:
 the baseline mental state comprises a first mental state value and a second mental state value; 
 the first mental state value corresponds to a first dimension of a multidimensional mental state model; and 
 the second mental state value corresponds to a second dimension of the multidimensional mental state model; 
 
 identify a target mental state based on a task performed by the individual in the video data, wherein:
 the target mental state comprises a third mental state value and a fourth mental state value; 
 the third mental state value corresponds to the first dimension of the multidimensional mental state model; and 
 the fourth mental state value corresponds to the second dimension of the multidimensional mental state model; 
 
 simulate, by a simulator, a predicted path from the baseline mental state toward the target mental state using the multidimensional mental state model, a plurality of actions, and a first computer-implemented machine learning model, wherein:
 the first computer-implemented machine learning model is configured to relate actions of the plurality of actions and changes in value in at least one of the first dimension and the second dimension of the multidimensional mental state model; 
 the predicted path comprises a first set of actions of the plurality of actions and corresponding changes to at least one of the first dimension and the second dimension of the multidimensional mental state model; and 
 the first set of actions include one or more actions that are performable by the individual; 
 
 generate a second set of actions based on the first set of features and a relational feature model, wherein:
 the relational feature model relates features and performance of the task; and 
 the second set of actions include one or more actions that are performable by the first individual to improve task performance; 
 
 cause the user interface to output an indication of the first set of actions to the individual; and 
 cause the user interface to output an indication of the second set of actions to the individual.