Discriminating Between Natural Language and Keyword Language Items

Abstract

This disclosure pertains to a classification model, and to functionality for producing and applying the classification model. The classification model is configured to discriminate whether an input linguistic item (such as a query) corresponding to either a natural language (NL) linguistic item or a keyword language (KL) linguistic item. An NL linguistic item expresses an intent using a natural language, while a KL linguistic item expresses the intent using one or more keywords. In a training phase, the functionality produces the classification model based on query click log data or the like. In an application phase, the functionality may, among other uses, use the classification model to filter a subset of NL linguistic items from a larger set of items, and then use the subset of NL linguistic items to train a natural language interpretation model, such as a spoken language understanding model.

What is claimed is: 
     
         1 . A computer system for performing actions based on input linguistic items, comprising:
 an NL-KL classifier module configured to:
 receive an input linguistic item; and 
 use an NL-KL classification model, produced using a machine learning technique, to produce a label which reflects whether the input linguistic item corresponds to one of:
 (a) a natural language (NL) linguistic item, which expresses an intent using a natural language; or 
 (b) a keyword language (KL) linguistic item, which expresses the intent using one or more keywords; and 
 
   an action-taking module configured to perform an action on the input linguistic item based on the label,   the NL-KL classifier module and the action-taking module performing their respective functions using one or more processing devices, in conjunction with storage resources.   
     
     
         2 . The computer system of  claim 1 ,
 wherein the NL-KL classifier module is configured to receive a set of input linguistic items, and classify each of the set of input linguistic items as either an NL linguistic item or a KL linguistic item, yielding, overall, a subset of NL linguistic items and a subset of KL linguistic items, and   wherein the action-taking module is configured to use a machine learning technique to generate a natural language interpretation model, based on the subset of NL linguistic items.   
     
     
         3 . The computer system of  claim 2 , wherein the natural language interpretation model is a spoken language understanding model. 
     
     
         4 . The computer system of  claim 1 , further comprising a data harvesting module configured to produce a collection of linguistic items that match a filtering condition specified in a filter request, and
 wherein the NL-KL classifier module is configured classify each of the collection of linguistic items as either an NL linguistic item or a KL linguistic item, yielding, overall, a subset of NL linguistic items and a subset of KL linguistic items.   
     
     
         5 . The computer system of  claim 4 , wherein the data harvesting module is configured to use a knowledge resource to generate the collection of linguistic items, the knowledge resource specifying a plurality of entities and relations among the entities. 
     
     
         6 . The computer system of  claim 1 ,
 wherein the NL-KL classifier module is configured to receive a set of input linguistic items, and classify each of the set of input linguistic items as either an NL linguistic item or a KL linguistic item, yielding, overall, a subset of NL linguistic items and a subset of KL linguistic items, and   further comprising a data harvesting module configured to filter the subset of NL linguistic items and/or the subset of KL linguistic items based on a filtering condition specified in a filter request.   
     
     
         7 . The computer system of  claim 1 , wherein the action-taking module is configured to perform a first action when the input linguistic item is an NL linguistic item, and perform a second action when the input linguistic item is a KL linguistic item. 
     
     
         8 . The computer system of  claim 7 ,
 wherein the first action comprises processing the input linguistic item with a first processing resource, and   wherein the second action comprises processing the input linguistic item with a second processing resource.   
     
     
         9 . The computer system of  claim 7 ,
 wherein the first action comprises routing the input linguistic item to a first destination, and   wherein the second action comprises routing the input linguistic item to a second destination.   
     
     
         10 . The computer system of  claim 1 , further including a training system for producing the NL-KL classification model, including:
 a domain-independent identification module configured to:
 receive a set of domain-labeled linguistic items; and 
 determine a set of domain-independent linguistic items based on the set of domain-labeled linguistic items; 
   a pair-generating module configured to:
 determine a set of matching linguistic item that contain at least one domain-independent linguistic item from the set of domain-independent linguistic items; 
 determine a set of linked linguistic items that are linked to the set of matching linguistic items, based on common actions taken by users, as reflected in log data; and 
 formulate a set of pair items based on the set of matching linguistic items and the set of linked linguistic items, each pair item in the set of pair items including: an NL linguistic item expressed in the natural language; and a related KL linguistic item expressed in the keyword language; and 
   a training module configured to:
 apply a first machine learning technique to the set of pair items, to produce an initial classification model; 
 receive a set of new linguistic items; 
 use the initial classification model to determine whether each new linguistic item in the set of new linguistic items is an NL linguistic item or a KL linguistic item, to produce a set of initially-labeled linguistic items; 
 receive additional labels provided by human annotators, to produce a set of refined linguistic items; and 
 apply a second machine learning technique to the set of refined linguistic items to produce the NL-KL classification model. 
   
     
     
         11 . A computer readable storage medium for storing computer readable instructions, the computer readable instructions providing an NL-KL classifier module when executed by one or more processing devices, the computer readable instructions comprising:
 logic configured to receive an input linguistic item; and   logic configured to use an NL-KL classification model, produced using a machine learning technique, to produce a label which reflects whether the input linguistic item corresponds to one of:
 (a) a natural language (NL) linguistic item, which expresses an intent using natural language; or 
 (b) a query language (KL) linguistic item, which expresses the intent using one or more keywords. 
   
     
     
         12 . A method, implemented using one or more computing devices, for producing an NL-KL classification model, comprising:
 producing and storing a set of pair items, each pair item in the set of pair items including:
 (a) a natural language (NL) linguistic item, which expresses an intent using natural language; and 
 (b) a keyword language (KL) linguistic item, which expresses the intent using one or more keywords, and is related to the NL linguistic item; and 
   using a machine learning technique to generate and store an NL-KL classification model based, in part, on the set of pair of items,   the NL-KL classification model being configured to identify whether an input linguistic item corresponds to either an NL linguistic item or a KL linguistic item.   
     
     
         13 . The method of  claim 12 , wherein said producing of the set of pair items comprises:
 receiving a set of domain-labeled linguistic items;   determining a set of domain-independent linguistic items based on the set of domain-labeled linguistic items;   determining a set of matching linguistic items that contain at least one domain-independent linguistic item from the set of domain-independent linguistic items;   determining a set of linked linguistic items that are linked to the set of matching linguistic items, based on common actions taken by users, as reflected in log data; and   formulating the set of pair items based on the set of matching linguistic items and the set of linked linguistic items.   
     
     
         14 . The method of  claim 13 , wherein the set of linked linguistic items are associated with the set of matching linguistic items based on common selections made by users in response to submitting the set of matching linguistic items and the set of linked linguistic items. 
     
     
         15 . The method of  claim 12 , wherein generating of the NL-KL classification model comprises:
 applying a first machine learning technique to the set of pair items, to produce an initial classification model;   receiving a set of new linguistic items;   using the initial classification model to determine whether each new linguistic item in the set of new linguistic items is an NL linguistic item or a KL linguistic item, to produce a set of initially-labeled linguistic items;   removing noise from the set of initially-labeled linguistic items, to produce a set of filtered linguistic items;   receiving additional labels for at least part of the set of filtered linguistic items provided by human annotators, to produce a set of refined linguistic items; and   applying a second machine learning technique to the set of refined linguistic items to produce the NL-KL classification model.   
     
     
         16 . The method of  claim 15 ,
 wherein the first machine learning technique uses a first set of features, and the second machine learning techniques uses a second set of features,   wherein the first set of features includes lexical features and structural features,   and wherein the second set of features includes lexical features, structural features, and semantic features.   
     
     
         17 . The method of  claim 16 , wherein at least one semantic feature identifies a presence of non-NL content in an input linguistic item, where that non-NL content resembles NL content. 
     
     
         18 . The method of  claim 12 , further comprising applying the NL-KL classification model by:
 receiving a set of input linguistic items;   using the NL-KL classification model to classify each of the input linguistic items as either an NL linguistic item or a KL linguistic item, to produce, overall, a subset of NL linguistic items and a subset of KL linguistic items, and   using a machine learning technique to generate a natural language interpretation model, based on the subset of NL linguistic items.   
     
     
         19 . The method of  claim 12 , further comprising applying the NL-KL classification model by:
 receiving a filter request that specifies a filtering condition;   producing a collection of linguistic items that match the filtering condition, based, at least in part, on a knowledge resource; and   classifying each of the collection of linguistic items as either an NL linguistic item or a KL linguistic item, to produce, overall, a subset of NL linguistic items and a subset of KL linguistic items.   
     
     
         20 . The method of  claim 12 , further comprising applying the NL-KL classification model by:
 receiving an input linguistic item;   using the NL-KL classification model to classify the input linguistic item as either an NL linguistic item or a KL linguistic item;   performing a first action when the input linguistic item is an NL linguistic item; and   performing a second action when the input linguistic item is a KL linguistic item.