System and method of handling problematic input during context-sensitive help for multi-modal dialog systems

Abstract

A method of presenting a multi-modal help dialog move to a user in a multi-modal dialog system is disclosed. The method comprises presenting an audio portion of the multi-modal help dialog move that explains available ways of user inquiry and presenting a corresponding graphical action performed on a user interface associated with the audio portion. The multi-modal help dialog move is context-sensitive and uses current display information and dialog contextual information to present a multi-modal help move that is currently related to the user. A user request or a problematic dialog detection module may trigger the multi-modal help move.

1 . In a multi-modal dialog system, a method of presenting a multi-modal help dialog move to a user in a problematic dialog situation, the method comprising:
 receiving a plurality of utterances that do not meet a successful recognition threshold;   in response to the received plurality of utterances, presenting an audio portion of a multi-modal help dialog move that explains available non-audio ways of user inquiry; and   presenting a corresponding graphical action performed on a user interface associated with the audio portion.   
   
   
       2 . The method of  claim 1 , wherein the presented audio portion and presented graphical action relate to what is currently being displayed to the user. 
   
   
       3 . The method of  claim 1 , wherein the presented graphical action further comprises at least one graphical button corresponding to choices available to the user. 
   
   
       4 . The method of  claim 3 , wherein the at least one graphical button relates to different kinds of help available to the user. 
   
   
       5 . The method of  claim 1 , wherein the number of the plurality of user utterances that does not meet the successful recognition threshold is adjustable based on user experience. 
   
   
       6 . The method of  claim 1 , wherein the number of the plurality of user utterances that does not meet the successful recognition threshold is adjustable based on automatic speech recognition performance. 
   
   
       7 . The method of  claim 1 , further comprising:
 receiving at least one graphical user input that does not meet a graphical user input recognition threshold.   
   
   
       8 . The method of  claim 7 , wherein the presented audio portion and presented graphical action relate to what is currently being displayed to the user. 
   
   
       9 . The method of  claim 7 , wherein the presented graphical action further comprises at least one graphical button corresponding to choices available to the user. 
   
   
       10 . The method of  claim 9 , wherein the at least one graphical button relates to different kinds of help available to the user. 
   
   
       11 . The method of  claim 1 , wherein the number of plurality of utterances is set according to user experience and automatic speech recognition parameters. 
   
   
       12 . A multi-modal dialog system for presenting a multi-modal help dialog move to a user, in a problematic dialog situation, the system comprising:
 a module receiving a plurality of utterances that do not meet a successful recognition threshold;   in response to the received plurality of utterances, a module presenting an audio portion of a multi-modal help dialog move that explains available non-audio ways of user inquiry; and   a module presenting a corresponding graphical action performed on a user interface associated with the audio portion.   
   
   
       13 . The system of  claim 12 , wherein the presented audio portion and presented graphical action relate to what is currently being displayed to the user. 
   
   
       14 . The system of  claim 12 , wherein the presented graphical action further comprises at least one graphical button corresponding to choices available to the user. 
   
   
       15 . The system of  claim 14 , wherein the at least one graphical button relates to different kinds of help available to the user. 
   
   
       16 . The system of  claim 12 , wherein the number of the plurality of user utterances that does not meet the successful recognition threshold is adjustable based on user experience. 
   
   
       17 . The system of  claim 11 , wherein the number of the plurality of user utterances that does not meet the successful recognition threshold is adjustable based on automatic speech recognition performance. 
   
   
       18 . A computer readable medium storing a computer program for controlling a computing device to present a multi-modal help dialog move to a user in a problematic dialog situation, the computer program having instructions comprising:
 receiving a plurality of utterances that do not meet a successful recognition threshold;   in response to the received plurality of utterances, presenting an audio portion of a multi-modal help dialog move that explains available non-audio ways of user inquiry; and   presenting a corresponding graphical action performed on a user interface associated with the audio portion.   
   
   
       19 . The computer readable medium of  claim 18 , wherein the presented audio portion and presented graphical action relate to what is currently being displayed to the user. 
   
   
       20 . The computer readable medium of  claim 18 , wherein the presented graphical action further comprises at least one graphical button corresponding to choices available to the user.