Apparatus, method, and system of cognitive communication assistant for enhancing ability and efficiency of users communicating comprehension

Abstract

A communication apparatus, a method, a computer readable medium, and a system providing communication with cognitive and visual assistance. The cognitive assistance and visual assistance is provided during a communication between a first communication apparatus with at least one second communication apparatus via a network. The first communication apparatus captures communication data comprising visual and audio information obtained from the communication and captures synchronized cognitive and emotional data generated from the user during the communication with the second communication apparatus. The communication data and the synchronized cognitive and emotional data is stored and converted into a visual form comprising at least one of synchronized text, symbols, sketches, images, and animation. The visual form is displayed on a display of the first communication apparatus.

What is claimed is: 
     
         1 . A method of providing cognitive assistance during a communication comprising:
 establishing, by a first communication apparatus operated by a user, the communication with a second communication apparatus of a participant;   receiving, by the first communication apparatus, first multimedia data generated by the second communication apparatus during the established communication;   capturing, by the first communication apparatus, second multimedia data generated by the user during the established communication;   extracting, during the established communication, audio content from the first multimedia data and the second multimedia data;   dividing the audio content to generate a plurality of audio data blocks;   converting each of the plurality of audio data blocks into a text format to generate a respective text block; and   during the established communication:
 displaying, in a first display area of the first communication apparatus, the respective text block for each of the first multimedia data and the second multimedia data in a form of a plurality of scripts, and 
 displaying, in a second display area of the first communication apparatus, the first multimedia data including the participant and the second multimedia data including the user. 
   
     
     
         2 . The method of  claim 1 , further comprising:
 displaying, during the established communication, in the first display area, third multimedia data generated during the established communication,   wherein the third multimedia data includes an environment of the user or the participant, or data downloaded via a network.   
     
     
         3 . The method of  claim 1 , wherein displaying the respective text block for each of the first multimedia data and the second multimedia data includes:
 displaying, during the established communication, a plurality of text blocks of preceding and current audio data, wherein the plurality of text blocks are color coded based on a cognitive state of the user.   
     
     
         4 . The method of  claim 1 , further comprising:
 receiving, via a user interface of the first communication apparatus, user input related to whether to start converting each of the plurality of audio data blocks into the respective text block.   
     
     
         5 . The method of  claim 1 , further comprising:
 displaying, after the established communication, in the first display area, the respective text block for each of the first multimedia data and the second multimedia data and a corresponding section for adding notes by the user.   
     
     
         6 . The method of  claim 5 , further comprising:
 displaying the plurality of scripts and the notes in a distinguishable manner from one another in the first display area.   
     
     
         7 . The method of  claim 1 , further comprising:
 displaying the plurality of scripts such that each of a first text block corresponding to the first multimedia data is visually distinguishable from a second text block corresponding to the second multimedia data.   
     
     
         8 . The method of  claim 1 , further comprising:
 based on user input, switching the first communication apparatus from a communication mode in which the established communication occurs, to a review mode in which the established communication has ended.   
     
     
         9 . An apparatus for providing cognitive assistance during a communication, the apparatus comprising:
 a network interface configured to establish the communication with a communication apparatus of a participant and to receive first multimedia data generated by the communication apparatus during the communication;   an image capturer configured to capture second multimedia data generated by a user of the apparatus during the communication;   a memory configured to store computer executable instructions;   a processor configured to execute the computer executable instructions, which when executed by the processor causes the processor to:
 extract, during the communication, audio content from the first multimedia data and the second multimedia data, 
 divide the audio content to generate a plurality of audio data blocks, and 
 convert each of the plurality of audio data blocks into a text format to generate a respective text block; and 
   at least one display configured to display, during the communication:
 the respective text block for each of the first multimedia data and the second multimedia data in a form of a plurality of scripts, in a first display area, and 
 the first multimedia data including the participant and the second multimedia data including the user, in a second display area. 
   
     
     
         10 . The apparatus of  claim 9 , wherein the at least one display is further configured to display, during the communication, in the first display area, third multimedia data generated during the communication and wherein the third multimedia data includes an environment of the user or the participant, or data downloaded via a network. 
     
     
         11 . The apparatus of  claim 9 , wherein the at least one display is configured to display the respective text block for each of the first multimedia data and the second multimedia data by:
 displaying, during the communication, a plurality of text blocks of preceding and current audio data, the plurality of text blocks being color coded based on a cognitive state of the user.   
     
     
         12 . The apparatus of  claim 9 , further comprising:
 a user interface configured to receive user input related to whether to start converting each of the plurality of audio data blocks into the respective text block.   
     
     
         13 . The apparatus of  claim 9 , further comprising:
 a user interface configured to receive input from a user in a form of notes or comments,   wherein the at least one display is further configured to display, after the communication, in the first display area, the respective text block for each of the first multimedia data and the second multimedia data and a section with the notes that are input by the user, and   wherein each one of the notes is displayed corresponding to the respective text block.   
     
     
         14 . The apparatus of  claim 13 , wherein the at least one display is further configured to display the plurality of scripts and the notes in a distinguishable manner from one another in the first display area. 
     
     
         15 . The apparatus of  claim 9 , wherein the at least one display is further configured to display the plurality of scripts such that each of a first text block corresponding to the first multimedia data is visually distinguishable from a second text block corresponding to the second multimedia data. 
     
     
         16 . The apparatus of  claim 9 , further comprising:
 a user interface configured to receive user input for switching the apparatus from a communication mode in which the communication occurs, to a review mode in which the communication has ended.   
     
     
         17 . A non-transitory computer readable medium configured to store instructions for providing cognitive assistance during a communication, the instructions are executed by a processor and cause the processor to execute the following operations:
 establish the communication with a communication apparatus of a participant;   receive first multimedia data generated by the communication apparatus during the established communication;   control an image capturer to capture second multimedia data generated by a user during the established communication;   extract, during the established communication, audio content from the first multimedia data and the second multimedia data;   divide the audio content to generate a plurality of audio data blocks;   convert each of the plurality of audio data blocks into a text format to generate a respective text block; and   during the established communication, control at least one display to display:
 the respective text block for each of the first multimedia data and the second multimedia data in a form of a plurality of scripts, in a first display area, and 
 the first multimedia data including the participant and the second multimedia data including the user, in a second display area. 
   
     
     
         18 . The non-transitory computer readable medium of  claim 17 , wherein the instructions further cause the processor to control the at least one display to display, during the established communication, in the first display area, third multimedia data generated during the established communication, wherein the third multimedia data includes an environment of the user or the participant, or data downloaded via a network. 
     
     
         19 . The non-transitory computer readable medium of  claim 17 , wherein the instructions further cause the processor to control the at least one display to display the respective text block for each of the first multimedia data and the second multimedia data by displaying, during the established communication, a plurality of text blocks of preceding and current audio data, and
 wherein the plurality of text blocks are color coded based on a cognitive state of the user.   
     
     
         20 . The non-transitory computer readable medium of  claim 17 , wherein the instructions further cause the processor to receive, via a user interface, user input related to whether to start converting each of the plurality of audio data blocks into the respective text block.