Fact checking

Abstract

The present invention relates to a method and system for verification scoring and automated fact checking. More particularly, the present invention relates to a combination of automated and assisted fact checking techniques to provide a verification score. According to a first aspect, there is a method of verifying input data, comprising the steps of: receiving one or more items of input data; determining one or more pieces of information to be verified from the or each item of input data; determining which of the one or more pieces of information are to be verified automatically and which of the one or more pieces of information require manual verification; determining an automated score indicative of the accuracy of the at least one piece of information which is to be verified automatically; and generating a combined verification score which gives a measure of confidence of the accuracy of the information which forms the or each item of input data.

1 . (canceled) 
     
     
         2 . A method comprising:
 determining an automated score indicative of an accuracy of a first set of pieces of information, of one or more items, to be verified by automatic verification;   providing a user with a fact-checking tool to determine a manual score indicative of the accuracy of a second set of pieces of information, of the one or more items, to be verified by manual verification, the fact-checking tool being configured to find correct evidence for the second set of pieces of information and to present a task list for a human fact-checker to check the second set of pieces of information, the manual score being provided by assistance of the human fact-checker; and   combining the automated score and the manual score to generate a combined verification score, the combined verification score giving a measure of confidence of the accuracy of the information which forms the one or more items.   
     
     
         3 . The method of  claim 2 , wherein the first set of pieces of information comprising at least one of a sentence, a paragraph, an article, or a full news story. 
     
     
         4 . The method of  claim 2 , wherein the determining of the automated score comprises using at least one classifier module to identify fake content or misleading content. 
     
     
         5 . The method of  claim 4 , wherein the at least one classifier module comprises at least one of:
 a clickbait detection module;   a stance detection module; or   a content-density module.   
     
     
         6 . The method of  claim 2 , wherein the determining of the automated score comprises using at least one of natural language processing or another computational method to generate a probabilistic score. 
     
     
         7 . The method of  claim 2 , wherein the automated score is determined in accordance with one or more weightings from multiple classifier modules. 
     
     
         8 . The method of  claim 2 , wherein the second set of pieces of information comprising a statement, and wherein the statement forms part of any one of an online post, a paragraph, an article, or a full news story. 
     
     
         9 . The method of  claim 2 , wherein the determining of the manual score comprises comparing a piece of information, from the second set of pieces of information, against one or more public databases or against reference information. 
     
     
         10 . The method of  claim 2 , wherein the determining of the manual score comprises detecting one or more statements from one or more pieces of the second set of pieces of information. 
     
     
         11 . The method of  claim 10 , wherein the detecting of the one or more statements comprises semantic parsing of the one or more pieces. 
     
     
         12 . The method of  claim 2 , wherein the determining of the manual score comprises:
 accessing an expert score for each human fact-checker;   allocating a claim to a most suitable fact-checker based on an accessed expert score of the most suitable fact-checker; and   using machine learning to automatically gather, for the most suitable fact-checker, one or more supporting or negating arguments for the claim from at least one of reference information or a public database.   
     
     
         13 . The method of  claim 12 , wherein the determining of the manual score comprises:
 receiving from the most suitable fact-checker at least one of a counter-hypothesis for the claim, a counter-argument for the claim, a fact-checker step-by-step reasoning for the claim; and   providing a reasoned conclusion or statements for the claim based on the at least one of the counter-hypothesis for the claim, the counter-argument for the claim, or the fact-checker step-by-step reasoning for the claim from the most suitable fact-checker.   
     
     
         14 . The method of  claim 12 , wherein the expert score for each human fact-checker is indicative of reliability of each human fact-checker. 
     
     
         15 . The method of  claim 12 , wherein the expert score for each human fact-checker is determined through an analysis of at least one of fact-checker bias, fact-checker credibility, fact-checker profile, or content generated by the human fact-checker. 
     
     
         16 . The method of  claim 2 , comprising storing the combined verification score on a real-time content quality database. 
     
     
         17 . The method of  claim 16 , wherein the real-time content quality database is adapted for a specific user type. 
     
     
         18 . A computer program operable to perform operations comprising:
 determining an automated score indicative of an accuracy of a first set of pieces of information, of one or more items, to be verified by automatic verification;   providing a user with a fact-checking tool to determine a manual score indicative of the accuracy of a second set of pieces of information, of the one or more items, to be verified by manual verification, the fact-checking tool being configured to find correct evidence for the second set of pieces of information and to present a task list for a human fact-checker to check the second set of pieces of information, the manual score being provided by assistance of the human fact-checker; and   combining the automated score and the manual score to generate a combined verification score, the combined verification score giving a measure of confidence of the accuracy of the information which forms the one or more items.   
     
     
         19 . The computer program of  claim 18 , wherein the determining of the manual score comprises:
 accessing an expert score for each human fact-checker;   allocating a claim to a most suitable fact-checker based on an accessed expert score of the most suitable fact-checker; and   using machine learning to automatically gather, for the most suitable fact-checker, one or more supporting or negating arguments for the claim from at least one of reference information or a public database.   
     
     
         20 . The computer program of  claim 19 , wherein the determining of the manual score comprises:
 receiving from the most suitable fact-checker at least one of a counter-hypothesis for the claim, a counter-argument for the claim, a fact-checker step-by-step reasoning for the claim; and   providing a reasoned conclusion or statements for the claim based on the at least one of the counter-hypothesis for the claim, the counter-argument for the claim, or the fact-checker step-by-step reasoning for the claim from the most suitable fact-checker.   
     
     
         21 . A system comprising:
 a media monitoring engine configured to receive one or more items of input data;   an automated misleading content detection algorithm configure to determine one or more pieces of information to be verified from the input data;   a claim detector configured to determine a first set of pieces of information to be verified automatically and a second set of pieces of information to be verified manually;   a set of classifier modules configured to determine an automated score indicative of accuracy of at least one piece of information in the first set;   a fact-checking tool configured to provide a user with a fact-checking tool to determine a manual score indicative of the accuracy of at least one piece of information in the second set of pieces of information, the fact-checking tool being configured to find correct evidence for the second set of pieces of information and to present a task list for a human fact-checker to check the second set of pieces of information, the manual score being provided by assistance of the human fact-checker; and   a truth score generator configured to combine the automated score and manual score to generate a combined verification score, the combined verification score giving a measure of confidence in the accuracy of the information in the input data.