Interactive Concept Editing In Computer-Human Interactive Learning

Abstract

A collection of data that is extremely large can be difficult to search and/or analyze. Relevance may be dramatically improved by automatically classifying queries and web pages in useful categories, and using these classification scores as relevance features. A thorough approach may require building a large number of classifiers, corresponding to the various types of information, activities, and products. Creation of classifiers and schematizers is provided on large data sets. Exercising the classifiers and schematizers on hundreds of millions of items may expose value that is inherent to the data by adding usable meta-data. Some aspects include active labeling exploration, automatic regularization and cold start, scaling with the number of items and the number of classifiers, active featuring, and segmentation and schematization.

1 . One or more hardware computer-storage device having embodied thereon computer-usable instructions that, when executed, implement a method of providing a user interface for interactively generating a machine-learning feature comprising a list of n-grams corresponding to examples of a concept, the method comprising:
 presenting the user interface which receives user input that is utilized for generating the list of n-grams corresponding to examples of the concept;   presenting on the user interface an input field that receives user-input n-grams as examples of the concept;   presenting on the user interface a field that displays suggested n-grams that are generated based on a generalized concept determined from the user-input n-grams, wherein the suggested n-grams are examples of the generalized concept; and   presenting on the user interface a field that displays n-grams added to the list of n-grams corresponding to examples of the concept, comprising the user-input n-grams.   
     
     
         2 . The one or more hardware computer-storage device of  claim 1 , the method further comprising:
 receiving a first user-input n-gram that is an example of the concept;   presenting a list of suggested n-grams that represent the generalized concept generated based at least on the first user-input n-gram.   
     
     
         3 . The one or more hardware computer-storage device of  claim 2 , the method further comprising:
 subsequent to presenting the list of suggested n-grams, receiving a second user-input n-gram that is a second example of the concept;   refining the list of suggested n-grams based at least on a combination of the first user-input n-gram and the second user-input n-gram; and   presenting the refined list of suggested n-grams that represent a refined generalized concept.   
     
     
         4 . The one or more hardware computer-storage device of  claim 2 , the method further comprising:
 subsequent to presenting the list of suggested n-grams, receiving a second user-input n-gram that is designated as a negative example of the concept;   refining the list of suggested n-grams based at least on a combination of the first user-input n-gram and the second user-input n-gram; and   presenting the refined list of suggested n-grams that represent a refined generalized concept.   
     
     
         5 . The one or more hardware computer-storage device of  claim 2 , the method further comprising:
 receiving a user selection of one or more of the suggested n-grams; and   including the one or more suggested n-grams as user-input examples of the concept.   
     
     
         6 . The one or more hardware computer-storage device of  claim 1 , wherein each n-gram in the list of n-grams is assigned a respective weight. 
     
     
         7 . The one or more hardware computer-storage device of  claim 6 , wherein the respective weight of each n-gram is scaled by a function of frequency, and a function of size of the n-gram list, based on training data during generation of the n-gram list. 
     
     
         8 . The one or more hardware computer-storage device of  claim 7 , wherein the scaled weights are related by a regularization constraint that adjusts the weights of the n-grams that have less training data toward a value determined by the n-grams that have more training data. 
     
     
         9 . A method of providing a user interface for interactively generating a machine-learning feature comprising a list of n-grams corresponding to a concept, the method comprising:
 presenting the user interface which receives user input that is utilized for generating the list of n-grams corresponding to the concept, wherein the n-grams are examples of the concept;   receiving one or more first user-input n-grams that are examples of the concept;   determining a first generalized concept based on the one or more first user-input examples;   based on the first generalized concept, generating a first set of suggested n-grams that are examples of the first generalized concept; and   displaying the first set of suggested n-grams on the user interface.   
     
     
         10 . The method of  claim 9 , further comprising:
 generating a refinement of the first set of suggested n-grams that represents a refinement of the first generalized concept based on at least one or more additional user-input n-grams; and   presenting the refinement of the first set of suggested n-grams on the user-interface.   
     
     
         11 . The method of  claim 10 , wherein the one or more additional user-input examples are received as a selection by the user from the first set of suggested n-grams. 
     
     
         12 . The method of  claim 10 , further comprising:
 repeating the steps of generating the refinement and presenting the refinement until an indication is received that the user has finished editing the list of n-grams; and   saving the first user-input n-grams and the one or more additional user-input n-grams in the list of n-grams corresponding to the concept.   
     
     
         13 . The method of  claim 9 , further comprising:
 generating a second set of suggested n-grams that represent a second generalized concept based on the one or more first user-input n-grams; and   presenting the second set of suggested n-grams.   
     
     
         14 . The method of  claim 13 , wherein the first set of suggested n-grams is generated utilizing a first source. 
     
     
         15 . The method of  claim 14 , wherein the first source includes one or more of:
 a previously-stored list of n-grams;   a click graph representing queries and visited web pages;   content of a table found on the world-wide web; or   semantic representations of n-grams.   
     
     
         16 . The method of  claim 14 , wherein the second set of suggested n-grams is generated utilizing a second source. 
     
     
         17 . The method of  claim 16 , wherein the second source includes one or more of:
 a previously-stored list of n-grams;   a click graph representing queries and visited web pages;   content of a table found on the world-wide web; or   semantic representations of n-grams.   
     
     
         18 . A system for providing a user interface for interactively editing a machine-learning feature comprising a list of n-grams corresponding to a concept, the system comprising:
 one more memory devices; and   one or more processors configured to:
 generate an interface for editing the list of n-grams that corresponds to the concept, wherein the n-grams are examples of the concept; 
 receive, via the interface, one or more user-input n-grams as examples of the concept; 
 determine a first generalized concept from the one or more user-input n-grams; 
 based on the first generalized concept, generate a set of suggested n-grams that are examples of the first generalized concept; 
 present the set of suggested n-grams on the user interface; 
 receive a user selection of a first suggested n-gram from the set of suggested n-grams; 
 include the selected first suggested n-gram as a user-input n-gram that is an example of the concept; 
 refine the set of suggested n-grams based at least on the selected first suggested n-gram, wherein the refined set of suggested n-grams represents a refined generalized concept; 
 present the refined set of suggested n-grams on the user interface; 
 receive an indication that the user has finished editing the list of n-grams; and 
 save the user-input n-grams in the list of n-grams corresponding to the concept. 
   
     
     
         19 . The system of  claim 18 , the one or more processors further configured to:
 present on the interface one or more input fields configured to receive one or more of:
 a user-selection that indicates whether trainable parameters are associated with each n-gram in the list of n-grams, or whether there is one trainable parameter associated with the entire list of n-grams, 
 a user selection that indicates whether a feature value associated with the list of n-grams is a binary value based on a quantity of n-grams in the list of n-grams or whether the feature value is a pre-determined function of a frequency of n-grams in the list of n-grams, 
 a user selection that indicates whether n-gram frequencies are normalized, 
 a user selection that indicates a regularization threshold that regulates a degree of interrelatedness between respective weights assigned to n-grams in the list of n-grams, or 
 a user selection that indicates whether the feature value is higher when multiple n-grams from the list of n-grams appear within a document or when a single n-gram from the list of n-grams appears multiple times in the document. 
   
     
     
         20 . The system of  claim 18 , wherein each n-gram in the list of n-grams is assigned a respective weight, and wherein the weights are related by a regularization constraint that adjusts the weights of n-grams that have less training data toward a value determined by n-grams that have more training data.