Method and apparatus for classifying and ranking interpretations for multimodal input fusion

Abstract

A method used in an electronic equipment ( 100 ) generates a set of joint multimodal interpretations ( 125 ) from a set of multimodal interpretations ( 115 ) generated by one or more modalities ( 105 ) during a turn, generates a set of integrated multimodal interpretations ( 135 ) including an integrated multimodal interpretation formed from each joint multimodal interpretation by unifying the type feature structure of each multimodal interpretation in the joint multimodal interpretation, and generates a multilevel confidence score for each integrated multimodal interpretation based on at least one of a context score, a content score, and a confidence score of the integrated multimodal interpretation. The method classifies the multimodal interpretations and generates a set of joint multimodal interpretations that comprises essentially all possible joint multimodal interpretations. The multilevel confidence scoring is based on up to eleven factors, which provides for an accurate ranking of the integrated multimodal interpretations.

1 . A method of classifying and ranking multimodal interpretations, the method comprising:
 generating a set of joint multimodal interpretations from a set of multimodal interpretations generated by one or more modalities during a turn;   generating a set of integrated multimodal interpretations comprising an integrated multimodal interpretation formed from each joint multimodal interpretation by unifying type feature structures of each multimodal interpretation in the joint multimodal interpretation; and   generating a multilevel confidence score for each integrated multimodal interpretation based on at least one of a context score, a content score, and a confidence score of the integrated multimodal interpretation.   
   
   
       2 . The method of classifying and ranking multimodal interpretations according to  claim 1 , wherein generating a set of joint multimodal interpretations from a set of multimodal interpretations generated by one or more modalities during a turn comprises:
 receiving a set of multimodal interpretations that are generated within a turn by a set of modalities, wherein each multimodal interpretation is one of either a non-ambiguous interpretation or an ambiguous interpretation of a subset of ambiguous interpretations of the set of multimodal interpretations, and wherein each multimodal interpretation is identified with the modality that has generated the multimodal interpretation, and wherein each multimodal interpretation is assigned an interpretation type and confidence score; and   generating a set of joint multimodal interpretations comprising all possible joint multimodal interpretations, each of which is formed from one of all possible combinations of multimodal interpretations of the set of multimodal interpretations such that each joint multimodal interpretation has no more than one ambiguous interpretation from each subset of ambiguous interpretations, and such that the multimodal interpretations of each joint multimodal interpretation satisfy an interpretation type relationship of a defined set of interpretation type relationships based on the interpretation type and modality of each multimodal interpretation.   
   
   
       3 . The method of classifying and ranking multimodal interpretations according to  claim 1 , wherein the context score of the integrated multimodal interpretation is generated based on at least one of:
 a quantity of turns ago an integrated multimodal interpretation was generated that had a type that is related to the type of the integrated multimodal interpretation;   a type relationship score determined by a type of relationship between the integrated multimodal interpretation and the integrated multimodal interpretation generated the quantity of turns ago; and   a set of type relationship scores determined by types of relationships between the integrated multimodal interpretation and each of a set of integrated multimodal interpretations that have been predicted to be generated within the turn.   
   
   
       4 . The method of classifying and ranking multimodal interpretations according to  claim 3 , wherein the context score is inversely related to the quantity of turns. 
   
   
       5 . The method of classifying and ranking multimodal interpretations according to  claim 3 , wherein the types of relationships comprise direct super, direct sub, direct contained, direct container, same, none, indirect super, indirect sub, indirect contained, and indirect container. 
   
   
       6 . The method of classifying and ranking multimodal interpretations according to  claim 1 , wherein the content score of the integrated multimodal interpretation is generated based on at least one of:
 a quantity of attributes included in the type feature structure of the integrated multimodal interpretation;   a quantity of the attributes that were given at least one value in the turn;   a quantity of the attributes that were given redundant values in the turn;   a quantity of the attributes that were given an explicit reference and not given a value in the turn; and   a confidence score for each attribute that was given a value in the turn.   
   
   
       7 . The method of classifying and ranking multimodal interpretations according to  claim 1 , wherein the confidence score for the integrated multimodal interpretation is generated based on at least one of:
 a confidence score generated for each multimodal interpretation that is included in a set of multimodal interpretations that formed the integrated multimodal interpretation;   a set of type relationship scores determined by a type of relationship between each pair of multimodal interpretations within the set of multimodal interpretations that formed the integrated multimodal interpretation; and   a reliability score of each of the one or more modalities that generated the set of multimodal interpretations that formed the integrated multimodal interpretation.   
   
   
       8 . The method of classifying and ranking multimodal interpretations according to  claim 7 , wherein the reliability score of a modality is based on a history of confidence scores of the modality and a current environment type. 
   
   
       9 - 14 . (canceled)