Method for objectively tracking and analyzing the social and emotional activity of a patient

Abstract

A method and system for objectively tracking and analyzing the social and emotional activity of a patient using an augmented reality computing device is provided. A patient is permitted to manually manipulate a target object in the physical world while viewing an augmented version showing a unique animated character representing either an abstract language, emotions, or social skills, depending on the module. The present system tracks and records the active face and the time spent on the active face, where the active face is the face upon which the patient's focus is automatically estimated, through calculation, to be trained upon. An observer views the session, the data recorded, and an automatically generated graphical representation of the data, which permits the observer to speak to patient regarding the character or scene rendered on the face which is determined to be the active face, helping the student engage in the session.

The invention claimed is: 
     
       1. A method for objectively tracking and analyzing the social and emotional activity of a patient using a computing device, the method comprising the steps of:
 implementing an augmented reality tracking application in memory on the computing device; 
 upon the augmented tracking application being initialized on the computing device, detecting by at least one camera a target object manipulated by the patient, the target object comprising a first face and a second face, a first fiducial marker positioned on the first face and a second fiducial marker positioned on the second face; 
 detecting at least one of the first fiducial marker and the second fiducial marker; 
 displaying on a screen of the computing device a first augmented image overlaid on the target object, the first augmented image comprising a first image associated with the first fiducial marker and a second image associated with the second fiducial marker; 
 displaying a second augmented image in place of displaying the first augmented image upon receiving an input activating a first image link, the second augmented image comprising a third image associated with the first fiducial marker and a fourth image associated with the second fiducial marker; 
 displaying a third augmented image in place of displaying the first augmented image upon receiving a second input activating a second image link, the third augmented image comprising a fifth image associated with the first fiducial marker and a sixth image associated with the second fiducial marker; 
 detecting a position and a rotation relative to the camera of the target object for each point in time for which a data point is collected; 
 calculating a coordinate data set of the target object based on the position and the rotation detected; 
 determining an active face for each point in time using the coordinate data set and a parameter indicative of the patient focusing attention on one of the first image and the second image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and 
 storing as a data set the data point for each point in time, the data point comprising the coordinate data set, the active face, and a time. 
 
     
     
       2. The method of  claim 1  further comprising the step of associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject. 
     
     
       3. The method of  claim 2  further comprising the steps of transmitting the data set to a second computing device; and displaying to a user at least a portion of the data set on a second screen of the second computing device. 
     
     
       4. The method of  claim 3  further comprising the steps of recording a video of the patient; transmitting the video to the second computing device; and displaying the video on the second screen of the second computing device. 
     
     
       5. The method of  claim 3  further comprising the steps of transmitting a mirror video of the screen of the first computing device to the second computing device; and displaying the mirror video on the second screen of the second computing device. 
     
     
       6. The method of  claim 3  further comprising the steps of retrieving from memory one or more talking prompts comprising a first talking prompt associated with the first therapeutic subject and a second talking prompt associated with the second therapeutic subject; and communicating the talking prompts through the second computing device to a user. 
     
     
       7. The method of  claim 3  further comprising the steps of establishing a videotelephony session between the patient and the user to permit the transmission in real-time and audio-video signal. 
     
     
       8. The method of  claim 3  further comprising the steps of processing the data set to automatically create a graphical representation of the data set; and displaying the graphical representation on the second screen of the second computing device. 
     
     
       9. The method of  claim 1  wherein the target object is a cube comprising the first face, the second face, a third face with a third fiducial marker positioned on the third face, a fourth face with a fourth fiducial marker positioned on the fourth face, a fifth face with a fifth fiducial marker positioned on the fifth face, and a sixth face with a sixth fiducial marker positioned on the sixth face. 
     
     
       10. The method of  claim 1  further comprising the steps of:
 determining, for the second augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the third image and the fourth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and 
 determining, for the third augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the fifth image and the sixth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time. 
 
     
     
       11. The method of  claim 1  further comprising the steps of:
 associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject; 
 associating the third image with a first subject first subtopic of the first therapeutic subject and the fourth image with a first subject second subtopic of the first therapeutic subject, the data point further comprising the first subject first subtopic and the first subject second subtopic; and 
 associating the fifth image with a second subject first subtopic of the second therapeutic subject and the sixth image with a second subject second subtopic of the first therapeutic subject, the data point further comprising the second subject first subtopic and the second subject second subtopic. 
 
     
     
       12. A method for objectively tracking and analyzing the social and emotional activity of a patient using a computing device, the method comprising the steps of:
 implementing an augmented reality tracking application in memory on the computing device; upon the augmented tracking application being initialized on the computing device, detecting by at least one camera a target object manipulated by the patient, the target object comprising a first face and a second face, a first fiducial marker positioned on the first face and a second fiducial marker positioned on the second face; 
 detecting at least one of the first fiducial marker and the second fiducial marker; 
 displaying on a screen of the computing device a first augmented image overlaid on the target object, the first augmented image comprising a first image associated with the first fiducial marker and a second image associated with the second fiducial marker; 
 detecting a position and a rotation relative to the camera of the target object for each point in time for which a data point is collected; 
 calculating a coordinate data set of the target object based on the position and the rotation detected; 
 determining an active face for each point in time using the coordinate data set and a parameter indicative of the patient focusing attention on one of the first image and the second image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and 
 storing as a data set the data point for each point in time, the data point comprising the coordinate data set, the active face, and a time, 
 wherein the step of determining the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the first image and the second image, wherein the parameter is one or more of determining which of the first face and the second face are oriented more upright compared to the other, determining which of the first face and the second face has a larger screen area, and determining which of the first face and the second face has received a touch input. 
 
     
     
       13. The method of  claim 12  further comprising the step of associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject. 
     
     
       14. The method of  claim 13  further comprising the steps of transmitting the data set to a second computing device; and displaying to a user at least a portion of the data set on a second screen of the second computing device. 
     
     
       15. The method of  claim 14  further comprising the steps of recording a video of the patient;
 transmitting the video to the second computing device; and displaying the video on the second screen of the second computing device. 
 
     
     
       16. The method of  claim 14  further comprising the steps of transmitting a mirror video of the screen of the first computing device to the second computing device; and displaying the mirror video on the second screen of the second computing device. 
     
     
       17. The method of  claim 14  further comprising the steps of retrieving from memory one or more talking prompts comprising a first talking prompt associated with the first therapeutic subject and a second talking prompt associated with the second therapeutic subject; and
 communicating the talking prompts through the second computing device to a user. 
 
     
     
       18. The method of  claim 14  further comprising the steps of establishing a videotelephony session between the patient and the user to permit the transmission in real-time and audio-video signal. 
     
     
       19. The method of  claim 14  further comprising the steps of processing the data set to automatically create a graphical representation of the data set; and displaying the graphical representation on the second screen of the second computing device. 
     
     
       20. The method of  claim 12 , wherein the target object is a cube comprising the first face, the second face, a third face with a third fiducial marker positioned on the third face, a fourth face with a fourth fiducial marker positioned on the fourth face, a fifth face with a fifth fiducial marker positioned on the fifth face, and a sixth face with a sixth fiducial marker positioned on the sixth face. 
     
     
       21. The method of  claim 12  further comprising the steps of:
 displaying a second augmented image in place of displaying the first augmented image upon receiving an input activating a first image link, the second augmented image comprising a third image associated with the first fiducial marker and a fourth image associated with the second fiducial marker; and 
 displaying a third augmented image in place of displaying the first augmented image upon receiving a second input activating a second image link, the third augmented image comprising a fifth image associated with the first fiducial marker and a sixth image associated with the second fiducial marker. 
 
     
     
       22. The method of  claim 21  further comprising the steps of:
 determining, for the second augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the third image and the fourth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and 
 determining, for the third augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the fifth image and the sixth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time. 
 
     
     
       23. The method of  claim 21  further comprising the steps of:
 associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject; 
 associating the third image with a first subject first subtopic of the first therapeutic subject and the fourth image with a first subject second subtopic of the first therapeutic subject, the data point further comprising the first subject first subtopic and the first subject second subtopic; and 
 associating the fifth image with a second subject first subtopic of the second therapeutic subject and the sixth image with a second subject second subtopic of the first therapeutic subject, the data point further comprising the second subject first subtopic and the second subject second subtopic.