Multi-modal interaction with intelligent assistants in voice command devices

Abstract

A method comprising detecting an activation of an intelligent assistant on an electronic device, waking up the intelligent assistant from a sleep mode in response to the activation, and determining an amount of vocabulary the intelligent assistant acts upon during a listening mode based on a type of the activation.

What is claimed is: 
     
         1 . A refrigerator comprising:
 a motion sensor;   a microphone;   a display;   a main body including one or more chambers;   at least one chamber door coupled to the main body; and   at least one processor configured to:
 based on a voice agent of the refrigerator being switched to the activation state from a deactivation state by a voice input including a wake word received via the microphone, control the display to output a first visual feedback indicating the activation state of the voice agent, 
 based on the voice agent being switched to the activation state from the deactivation state by an input received from the motion sensor, control the display to output a second visual feedback indicating the activation state of the voice agent, wherein the second visual feedback is different from the first visual feedback, and 
 based on a voice command being received via the microphone while the voice agent of the refrigerator is in the activated state, perform an action associated with the voice command in a case that the voice command is recognized. 
   
     
     
         2 . The refrigerator of  claim 1 , further comprising:
 a speaker.   
     
     
         3 . The refrigerator of  claim 2 , wherein, in a case that the voice agent is activated based on the voice input, the at least one processor is further configured to:
 control the speaker to output a first audio feedback indicating that the voice agent is in the activation state.   
     
     
         4 . The refrigerator of  claim 3 ,
 wherein, in a case that the voice agent is activated based on the input received from the motion sensor, the at least one processor is further configured to:   control the speaker to output a second audio feedback indicating the activation state of the voice agent, and   wherein the second audio feedback is different from the first audio feedback.   
     
     
         5 . The refrigerator of  claim 3 , wherein, in a case that the voice agent is activated based on the input received from the motion sensor, the at least one processor is further configured to:
 prevent the speaker from outputting the first audio feedback.   
     
     
         6 . The refrigerator of  claim 2 ,
 wherein the at least one processor is further configured to:
 obtain one or more error feedback rules, and 
 in a case that the voice agent encounters an error, determine whether to output, via the speaker, an audio feedback indicating the error based on the one or more error feedback rules, and 
 wherein the one or more error feedback rules considers whether the voice agent is activated based on the voice input or the input received from the motion sensor. 
   
     
     
         7 . The refrigerator of  claim 2 , wherein the at least one processor is further configured to:
 in a case that the voice command received via the microphone after the voice agent is switched to the activated state by the voice input is not recognized, control the speaker to output a first audio feedback indicating that the voice command is not recognized, and   in a case that the voice command received via the microphone after the voice agent is switched to the activated state by the input from the motion sensor is not recognized, prevent the speaker from outputting a second audio feedback indicating that the second voice command is not recognized.   
     
     
         8 . The refrigerator of  claim 2 , further comprising:
 a door sensor for detecting a change in a state of the at least one chamber door.   
     
     
         9 . The refrigerator of  claim 8 ,
 wherein the at least one processor is further configured to:
 identify whether the voice command is recognized, and 
 identify a current state of the refrigerator, and 
 wherein the action associated with the voice command is performed based on the voice command being recognized and the current state of the refrigerator. 
   
     
     
         10 . The refrigerator of  claim 9 , wherein the current state of the refrigerator includes a current state of the at least one chamber door of the refrigerator. 
     
     
         11 . The refrigerator of  claim 8 , wherein the at least one processor is further configured to:
 based on an input indicating a change in state of the at least one chamber door, trigger a transition between the activation state of the voice agent and the deactivation state of the voice agent.   
     
     
         12 . The refrigerator of  claim 1 , wherein the at least one processor is further configured to:
 identify whether to perform the action associated with the voice command received via the microphone after the voice agent is switched to the activated state by the voice input based on a method used to activate the voice agent, and   identify whether to perform the action associated with the voice command received via the microphone after the voice agent is switched to the activated state by the input from the motion sensor based on the method used to activate the voice agent.   
     
     
         13 . The refrigerator of  claim 1 , wherein, to perform the action associated with the voice command, the at least one processor is further configured to control one or more mechanical parts of the refrigerator by control data associated with the voice command. 
     
     
         14 . The refrigerator of  claim 1 , wherein the input received from the motion sensor corresponds to an object being detected within a predetermined distance of the refrigerator. 
     
     
         15 . The refrigerator of  claim 1 , wherein the at least one processor is further configured to:
 extract speech data from the voice command,   compare the extracted speech data to speech data stored in a memory of the refrigerator, and   recognize the voice command as a valid command based on the extracted speech data and the speech data stored in the memory.   
     
     
         16 . A method performed by a refrigerator, the method comprising:
 based on a voice agent of a refrigerator being switched to the activation state from a deactivation state by a voice input including a wake word received via a microphone of the refrigerator, controlling a display of the refrigerator to output a first visual feedback indicating the activation state of the voice agent;   based on the voice agent being switched to the activation state from the deactivation state by an input received from a motion sensor of the refrigerator, controlling the display to output a second visual feedback indicating the activation state of the voice agent, wherein the second visual feedback is different from the first visual feedback; and   based on voice command being received via the microphone while the voice agent of the refrigerator is in the activated state, performing an action associated with the voice command in a case that the voice command is recognized.   
     
     
         17 . The method of  claim 16 , further comprising:
 controlling a speaker of the refrigerator to output a first audio feedback indicating that the voice agent is in the activation state.   
     
     
         18 . The method of  claim 17 , further comprising:
 in a case that the voice agent is activated based on the input received from the motion sensor, controlling the speaker to output a second audio feedback indicating the activation state of the voice agent,   wherein the second audio feedback is different from the first audio feedback.   
     
     
         19 . The method of  claim 17 , further comprising:
 in a case that the voice agent is activated based on the input received from the motion sensor, preventing the speaker from outputting the first audio feedback.   
     
     
         20 . The method of  claim 17 , further comprising:
 obtaining one or more error feedback rules; and   in a case that the voice agent encounters an error, determining whether to output, via the speaker, an audio feedback indicating the error based on the one or more error feedback rules,   wherein the one or more error feedback rules considers whether the voice agent is activated based on the voice input or the input received from the motion sensor.   
     
     
         21 . The method of  claim 17 , further comprising:
 in a case that the voice command received via the microphone after the voice agent is switched to the activated state by the voice input is not recognized, controlling the speaker to output a third audio feedback indicating that the voice command is not recognized, and   in a case that the voice command received via the microphone after the voice agent is switched to the activated state by the input from the motion sensor is not recognized, preventing the speaker from outputting a fourth audio feedback indicating that the voice command is not recognized.   
     
     
         22 . The method of  claim 17 , further comprising:
 identifying whether the voice command is recognized, and   identifying a current state of the refrigerator based on an input received from a door sensor of the refrigerator,   wherein the action associated with the voice command is performed based on the voice command being recognized and the current state of the refrigerator.   
     
     
         23 . The method of  claim 22 , wherein the current state of the refrigerator includes a current state of at least one chamber door of the refrigerator. 
     
     
         24 . The method of  claim 23 , further comprising:
 based on the input received from the door sensor of the refrigerator, triggering a transition between the activation state of the voice agent and the deactivation state of the voice agent.   
     
     
         25 . The method of  claim 16 , further comprising:
 identifying whether to perform the action associated with the voice command received via the microphone after the voice agent is switched to the activated state by the voice input based on a method used to activate the voice agent; and   identifying whether to perform the action associated with the voice command received via the microphone after the voice agent is switched to the activated state by the input from the motion sensor based on the method used to activate the voice agent.   
     
     
         26 . The method of  claim 16 , wherein the input received from the motion sensor corresponds to an object being detected within a predetermined distance of the refrigerator. 
     
     
         27 . The method of  claim 16 , further comprising:
 extracting speech data from the voice command;   comparing the extracted speech data to speech data stored in a memory of the refrigerator; and   recognizing the voice command as a valid command based on the extracted speech data and the speech data stored in the memory.