Scalable curation system

Abstract

A system and method is disclosed for improving automated question-answering using real-world knowledge from a knowledge base. The system can be used to answer questions from various users. The system can generate answers to these questions using data stored in a knowledge base. In some embodiments, the system is provided with a list of questions separate from any user questions. The system can be trained using these questions in advance of a user question. The system can process the presented questions to determine that it can generate high-quality correct answers. The system can take various steps to determine a high-quality answer to a question. The system can utilize human intelligence providers to improve this process, such as through a human interactive task system. Human intelligence can be used to determine that a question is understood, that a question is answered, and that the answer is of high quality.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 under control of one or more computing devices configured with specific computer-executable instructions,
 receiving, from an electronic device, text representing a question; 
 identifying a first pattern of objects included in the text, wherein the first pattern identifies an ordering of the objects; 
 comparing the first pattern to a template of ordered object types; 
 determining a type for each of at least a portion of the objects included in the first pattern differs from the ordered object types of the template; 
 generating a second pattern of objects, the second pattern including at least one object from the first pattern; 
 determining a type for each of at least a portion of the objects included in the second pattern corresponds to the ordered object types of the template; 
 retrieving, from a knowledge base, information associated with the question, the retrieving based in part on the second pattern; and 
 transmitting at least a portion of the information associated with the question to the electronic device. 
   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 identifying a format of the question represented by the text based on a location of the object within the ordering, wherein identifying the first pattern is based at least in part on the format of the question.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the format includes a carrier slot identifying a type of information the question is soliciting and a content slot including the object. 
     
     
         4 . The computer-implemented method of  claim 2 , wherein the format includes a relationship between a first object and a second object. 
     
     
         5 . The computer-implemented method of  claim 1 , further comprising:
 transmitting a textual representation of the second pattern of linguistic features to a natural language understanding system; and   receiving at least one text response and a confidence metric for the at least one text response.   
     
     
         6 . The computer-implemented method of  claim 5 , further comprising:
 comparing the confidence metric to a confidence threshold, wherein the retrieving, from the knowledge base, further comprises determining the confidence metric meets or exceed the confidence threshold.   
     
     
         7 . The computer-implemented method of  claim 5 , further comprising:
 determining the confidence metric indicates a confidence less than a confidence threshold; and   receiving, from a data source, additional information associated with the question, wherein the at least a portion of the information associated with the question includes the additional information.   
     
     
         8 . The computer-implemented method of  claim 5 , wherein the retrieving, from the knowledge base, further comprises retrieving, from the knowledge base, the information associated with the question based on the at least one text response. 
     
     
         9 . The computer-implemented method of  claim 1 , further comprising, prior to transmitting the at least a portion of the information:
 generating a validation question including at least the portion of the information;   transmitting text representing the validation question to at least one of the one or more computing devices;   receiving a response to the validation question; and   determining the response corresponds to an entity identified in the question.   
     
     
         10 . A system comprising:
 a computer-readable memory storing executable instructions; and   one or more processors in communication with the computer-readable memory, wherein the one or more processors are programmed by the executable instructions to at least:
 receive, from an electronic device, text representing a question; 
 identify a first pattern of objects included in the text, wherein the first pattern identifies an ordering of the objects; 
 compare the first pattern to a template of ordered object types; 
 determine a type for each of at least a portion of the objects included in the first pattern differs from the ordered object types of the template; 
 generate a second pattern of objects, the second pattern including at least one object from the first pattern; 
 determine a type for each of at least a portion of the objects included in the second pattern corresponds to the ordered object types of the template; 
 retrieve, from a knowledge base, information associated with the question based in part on the second pattern; and 
 transmit at least a portion of the information associated with the question to the electronic device. 
   
     
     
         11 . The system of  claim 10 , wherein the one or more processors are further programmed by the executable instructions to at least:
 identify a format of the question represented by the text based on a location of the object within the ordering, wherein the one or more processors are programmed to identify the first pattern based at least in part on the format of the question.   
     
     
         12 . The system of  claim 11 , wherein the format includes a carrier slot identifying a type of information the question is soliciting and a content slot including the object. 
     
     
         13 . The system of  claim 11 , wherein the format includes a relationship between a first object and a second object. 
     
     
         14 . The system of  claim 10 , wherein the one or more processors are further programmed by the executable instructions to at least:
 transmit a textual representation of the second pattern of linguistic features to a natural language understanding system; and   receive at least one text response and a confidence metric for the at least one text response.   
     
     
         15 . The system of  claim 14 , wherein the one or more processors are further programmed by the executable instructions to at least:
 compare the confidence metric to a confidence threshold, wherein the one or more processors are programmed to retrieve, from the knowledge base are further programmed to at least determine that the confidence metric meets or exceed the confidence threshold.   
     
     
         16 . The system of  claim 14 , wherein the one or more processors are further programmed by the executable instructions to at least:
 determine the confidence metric indicates a confidence less than a confidence threshold; and   receive, from a data source, additional information associated with the question, wherein the at least a portion of the information associated with the question includes the additional information.   
     
     
         17 . The system of  claim 16 , wherein the data source comprises a human interactive tasking system. 
     
     
         18 . The system of  claim 16 , wherein the one or more processors are further programmed by the executable instructions to at least:
 transmit a message to the electronic device indicating that the additional information is being acquired, wherein the one or more processors are programmed to transmit the at least a portion of the information associated with the question to the electronic device in response to receiving the additional information.   
     
     
         19 . The system of  claim 14 , wherein the one or more processors are programmed to retrieve, from the knowledge base, the information based on the at least one text response. 
     
     
         20 . The system of  claim 10 , wherein the one or more processors are further programmed by the executable instructions to at least, prior to transmitting the at least a portion of the information:
 generate a validation question including at least the portion of the information;   transmit text representing the validation question to at least one of the one or more computing devices;   receive a response to the validation question; and   determine the response corresponds to an entity identified in the question.