Multimodal aggregating unit

Abstract

In a voice processing system, a multimodal request is received from a plurality of modality input devices, and the requested application is run to provide a user with the feedback of the multimodal request. In the voice processing system, a multimodal aggregating unit is provided which receives a multimodal input from a plurality of modality input devices, and provides an aggregated result to an application control based on the interpretation of the interaction ergonomics of the multimodal input within the temporal constraints of the multimodal input. Thus, the multimodal input from the user is recognized within a temporal window. Interpretation of the interaction ergonomics of the multimodal input include interpretation of interaction biometrics and interaction mechani-metrics, wherein the interaction input of at least one modality may be used to bring meaning to at least one other input of another modality.

1 . A voice processing system, comprising: 
 a plurality of modality input devices; and    a multimodal aggregating unit comprising a life cycle determining device for receiving input from the plurality of modality input devices, whereby the multimodal aggregating unit provides an aggregated output based on the input from the plurality of modality input devices and temporal constraints determined by the life cycle determining device.    
     
     
         2 . The system of  claim 1 , wherein the life cycle determining device determines a temporal window for aggregating the input of the plurality of modality input devices based on life cycles of each input.  
     
     
         3 . The system of  claim 2 , wherein the plurality of modality input devices include a first modality input device and a second modality input device, wherein input from the second modality input device is received within the temporal window to bring meaning to input from the first modality input device.  
     
     
         4 . The system of  claim 3 , wherein the temporal window is determined as the overlap between the life cycles of the input from the first modality input device and the input from the second modality input device.  
     
     
         5 . The system of  claim 1 , wherein the multimodal aggregating unit interprets interaction ergonomics of the input from the plurality of modality input devices and an input of at least one modality may be used to bring meaning to at least one other input of another modality.  
     
     
         6 . The system of  claim 4 , wherein interpreting the interaction ergonomics comprises interpretation of interaction biometrics and interaction mechani-metrics.  
     
     
         7 . The system of  claim 1 , wherein the input from the plurality of modality input devices comprises at least one of touch, speech, gestures, eye movements, face direction, and the like.  
     
     
         8 . A multimodal aggregating unit, comprising: 
 an input unit that receives an input from a plurality of modality input devices; and    a decoder comprising a life cycle determining device for providing an aggregated output based on the input from the plurality of modality input devices and temporal constraints determined by the life cycle determining device.    
     
     
         9 . The multimodal aggregating unit of  claim 8 , wherein the life cycle determining device determines a temporal window for aggregating the input of the plurality of modality input devices based on life cycles of each input.  
     
     
         10 . The multimodal aggregating unit of  claim 9 , wherein the input includes input from a first modality input device and input from a second modality input device, wherein the input from the second modality input device is received within the temporal window to bring meaning to the input from the first modality input device.  
     
     
         11 . The multimodal aggregating unit of  claim 10 , wherein the temporal window is determined as the overlap between the life cycles of the input from the first modality input device and the input from the second modality input device.  
     
     
         12 . The multimodal aggregating unit of  claim 8 , wherein the multimodal aggregating unit interprets interaction ergonomics of the input from the plurality of modality input devices and an input of at least one modality may be used to bring meaning to at least one other input of another modality.  
     
     
         13 . The multimodal aggregating unit of  claim 12 , wherein interpreting the interaction ergonomics comprises interpretation of interaction biometrics and interaction mechani-metrics.  
     
     
         14 . The multimodal aggregating unit of  claim 8 , wherein the input from the plurality of modality input devices comprises at least one of touch, speech, gestures, eye movements, face direction, and the like.  
     
     
         15 . A voice processing method, comprising: 
 receiving an input from a plurality of modality input devices; and    providing an aggregated output based on the input from the plurality of modality input devices and temporal constraints determined for aggregating the input of the plurality of modality input devices.    
     
     
         16 . The method of  claim 15 , further comprising determining a temporal window based on life cycles of each input of the modality input devices to determine the temporal constraints for aggregating the input of the plurality of modality input devices.  
     
     
         17 . The method of  claim 16 , wherein the input from the plurality of modality input devices includes a first modality input and a second modality input, wherein the second modality input is received within the temporal window to bring meaning to the first modality input.  
     
     
         18 . The method of  claim 17 , further comprising determining the temporal window as the overlap between the life cycles of the first modality input and the second modality input.  
     
     
         19 . The method of  claim 15 , further comprising interpreting interaction ergonomics of the input from the plurality of modality input devices, wherein an input of at least one modality may be used to bring meaning to at least one other input of another modality.  
     
     
         20 . The method of  claim 19 , wherein interpreting the interaction ergonomics comprises interpretation of interaction biometrics and interaction mechani-metrics.