Validating that a user is human

Abstract

A method of validating that a user is human. A user question is generated using a computerized device. The user question is output to a user. A user response to the user question is received from the user. The user response is validated as having been provided by a human.

1 . A computerized device for determining that a user of a question-answer system is human, comprising:
 a question-answer system comprising software for performing a plurality of question answering processes;   a receiver receiving a question into said question-answer system;   a processor connected to said question-answer system, said processor:
 generating a plurality of candidate answers to said question, 
 evaluating sources of evidence used to generate said plurality of candidate answers to identify marginal evidence, said marginal evidence only partially contributing to a candidate answer, 
 determining a confidence score for each of said plurality of candidate answers, 
 identifying information not provided by said marginal evidence that could further develop said confidence score, and 
 generating at least one follow-on inquiry based on said information; and 
   a network interface outputting said at least one follow-on inquiry to external sources separate from said question-answer system to obtain responses to said follow-on inquiry and receives from said external sources at least one response to said at least one follow-on inquiry,   said processor evaluating whether said at least one response to said at least one follow-on inquiry was provided by a human respondent based on one of:   a previous determination that a respondent is human, and   syntactic or semantic characteristics of said at least one response.   
     
     
         2 . The computerized device of  claim 1 , said processor evaluating whether said at least one response to said at least one follow-on inquiry was provided by a human respondent comprising one of
 comparing said at least one response with a response of a trusted source, said at least one response being valid if said at least one response matches said response of said trusted source;   comparing said at least one response with a consensus from an online community, said at least one response being valid if said at least one response matches said consensus, said consensus being determined based on a threshold number of responses from an online community matching; and   determining if said at least one response is in natural language format.   
     
     
         3 . The computerized device of  claim 1 , said processor evaluating whether said at least one response to said at least one follow-on inquiry was provided by a human respondent comprising determining if said at least one response is in natural language format by parsing said at least one response into syntactic elements and determining if said at least one response conforms to syntactic rules of a language 
     
     
         4 . The computerized device of  claim 1 , said processor identifying information not provided by said marginal evidence further comprising:
 identifying missing information from a corpus of data, said missing information comprising any information that improves said confidence score for a candidate answer to said question; and   generating said at least one follow-on inquiry in natural language format, said at least one follow-on inquiry prompting said external sources to provide said missing information.   
     
     
         5 . The computerized device of  claim 4 , said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule. 
     
     
         6 . The computerized device of  claim 1 , said processor generating at least one follow-on inquiry comprising forming a natural-language inquiry; and
 said processor evaluating whether said at least one response to said at least one follow-on inquiry was provided by a human respondent comprising one of
 assessing accuracy of said at least one response to said at least one follow-on inquiry, 
 determining whether or not said at least one response to said at least one follow-on inquiry is in natural language format, and 
 any combination of said assessing accuracy of said at least one response to said at least one follow-on inquiry and said determining whether or not said at least one response to said at least one follow-on inquiry is in natural language format. 
   
     
     
         7 . The computerized device of  claim 1 , said processor generating at least one follow-on inquiry comprising forming an inquiry that humans have a known particular manner of responding; and
 said processor evaluating whether said at least one response to said at least one follow-on inquiry was provided by a human respondent comprising determining that said respondent's response matches said known particular manner of responding.   
     
     
         8 . A computer system for verifying that a user is human, comprising:
 an automated question answering (QA) system comprising:
 a corpus of data, 
 a processor, said processor having software for performing a plurality of question answering processes; and 
 a network interface; 
   said processor generating a user question from said corpus of data,   said network interface outputting said user question to a user separate from said QA system and receiving a user response to said user question from said user,   said processor determining whether said user is human based on syntactic or semantic characteristics of said response to said user question.   
     
     
         9 . The computer system of  claim 8 , said processor determining whether said user is human comprising one of
 comparing said user response with a response of a trusted source, said user response being valid if said user response matches said response of said trusted source;   comparing said user response with a consensus from an online community, said user response being valid if said user response matches said consensus, said consensus being determined based on a threshold number of responses from an online community matching; and   determining if said user response is in natural language format.   
     
     
         10 . The computer system of  claim 8 , said processor determining whether said user is human comprising determining if said user response is in natural language format by parsing said user response into syntactic elements and determining if said user response conforms to syntactic rules of a language 
     
     
         11 . The computer system of  claim 8 , said processor generating said user question further comprising:
 identifying missing information from said corpus of data, said missing information comprising any information that improves a confidence score for a candidate answer to a first question; and   generating said user question in natural language format, said user question prompting said user to provide said missing information,
 said missing information comprising a data, a fact, a syntactical relationship, a grammatical relationship, a logical rule, or a taxonomy rule. 
   
     
     
         12 . The computer system of  claim 11 , said processor identifying missing information further comprising:
 receiving said first question into a question-answer system;   generating said candidate answer to said first question; and   evaluating a piece of evidence relevant to said candidate answer in relation to said first question in order to identify said missing information.   
     
     
         13 . The computer system of  claim 12 , said processor evaluating a piece of evidence further comprising:
 parsing said first question into a collection of elements;   parsing said piece of evidence into a collection of elements;   analyzing said collection of elements for said first question and said piece of evidence in order to determine a relationship between an element of said collection of elements for said first question and an element of said collection of element for said piece of evidence;   locating a missing relationship between said elements, wherein said missing relationship is said missing information, said relationship between elements comprising a lexical relationship, a grammatical relationship, or a semantic relationship.   
     
     
         14 . A question answering (QA) system comprising:
 a processor;   an evidence analysis module, said evidence analysis module being connected to said processor;   a first interface connected to said processor;   a second interface connected to said processor; and   a corpus of data connected to said evidence analysis module,   said first interface receiving a first question to be answered by said QA system,   said processor creating a collection of candidate answers to said first question from said corpus of data, each said candidate answer having supporting evidence and a confidence score generated by said processor,   said evidence analysis module producing a second question based on said supporting evidence,   said processor presenting said second question through said second interface to one or more external sources separate from said QA system to obtain responses to said second question,   said processor receiving at least one response or knowledge item from said one or more external sources through said second interface, and   said processor determining whether a respondent is human based on one of:
 a previous determination that a respondent is human, and 
 syntactic or semantic characteristics of said at least one response or knowledge item obtained for said second question. 
   
     
     
         15 . The question answering system of  claim 14 , said corpus of data further comprising a plurality of passages, said processor comparing said first question to said plurality of passages to provide said collection of candidate answers. 
     
     
         16 . The question answering system of  claim 14 , said corpus of data further comprising a total amount of evidence, said evidence analysis module classifying said total amount of evidence as one of good evidence and marginal evidence,
 said good evidence having an evidence score above a previously established evidence threshold value, and enabling said QA system to provide a candidate answer to said first question with a confidence score above a previously established confidence threshold value, and   said marginal evidence having an evidence score below said previously established evidence threshold value, enabling said QA system to provide a candidate answer to said first question with a confidence score below said previously established confidence threshold value, and requiring said QA system to obtain additional information to improve said confidence score for said candidate answer to said first question.   
     
     
         17 . The question answering system of  claim 14 , said corpus of data further comprising answers to previous questions received from external sources previously known to be humans,
 said processor comparing said at least one response received from a respondent to said tracked answers and determining that said respondent is human when said at least one response received from said respondent matches said tracked answers.   
     
     
         18 . The question answering system of  claim 14 , said evidence analysis module producing a second question based on said supporting evidence comprising forming a natural-language inquiry; and
 said processor determining whether said respondent is human comprising one of
 assessing accuracy of said at least one response or knowledge item, 
 determining whether or not said at least one response or knowledge item is in natural language format, and 
 any combination of said assessing accuracy of said at least one response or knowledge item and said determining whether or not said at least one response or knowledge item is in natural language format. 
   
     
     
         19 . The question answering system of  claim 14 , said evidence analysis module producing a second question based on said supporting evidence comprising forming an inquiry that humans have a known particular manner of responding; and
 said processor determining whether said respondent is human comprising determining that said respondent's response matches said known particular manner of responding.   
     
     
         20 . A non-transitory computer readable storage medium readable by a computerized device, said computerized device comprising a question-answer system comprising question answering processes, said non-transitory computer readable storage medium storing instructions executable by said computerized device to perform a method comprising:
 generating a user question;   outputting said user question to a user;   receiving a user response to said user question from said user; and   validating said user response as having been provided by a human.   
     
     
         21 . The non-transitory computer readable storage medium of  claim 20 , said validating said user response comprising one of
 comparing said user response with a response of a trusted source, said user response being valid if said user response matches said response of said trusted source;   comparing said user response with a consensus from an online community, said user response being valid if said user response matches said consensus, wherein said consensus is determined based on a threshold number of responses from an online community matching; and   determining if said user response is in natural language format.   
     
     
         22 . The non-transitory computer readable storage medium of  claim 20 , said method further comprising:
 parsing said user response into syntactic elements and determining if said user response conforms to syntactic rules of a language.   
     
     
         23 . The non-transitory computer readable storage medium of  claim 20 , said generating said user question further comprising:
 identifying missing information from a corpus of data, wherein said missing information is any information that improves a confidence for a candidate answer to a first question; and   generating said user question in natural language format, wherein said user question prompts said user to provide said missing information.   
     
     
         24 . The non-transitory computer readable storage medium of  claim 23 , said identifying missing information further comprising:
 receiving said first question into a question-answer system;   generating said candidate answer to said first question; and   evaluating a piece of evidence relevant to said candidate answer in relation to said first question in order to identify said missing information.   
     
     
         25 . The non-transitory computer readable storage medium of  claim 24 , said identifying missing information further comprising:
 parsing said first question into a collection of elements;   parsing said piece of evidence into a collection of elements;   analyzing said collection of elements for said first question and said piece of evidence in order to determine a relationship between an element of said collection of elements for said first question and an element of said collection of element for said piece of evidence;   locating a missing relationship between said elements, wherein said missing relationship is said missing information.