Comprehensive human computation framework

Abstract

Technologies for a human computation framework suitable for answering common sense questions that are difficult for computers to answer but easy for humans to answer. The technologies support solving general common sense problems without a priori knowledge of the problems; support for determining whether an answer is from a bot or human so as to screen out spurious answers from bots; support for distilling answers collected from human users to ensure high quality solutions to the questions asked; and support for preventing malicious elements in or out of the system from attacking other system elements or contaminating the solutions produced by the system, and preventing users from being compensated without contributing answers.

1 . A method performed on a computing device, the method comprising:
 selecting, by the computing device, a common-sense problem from a first source;   receiving, by the computing device, answers to the common-sense problem from a second source;   identifying, by the computing device, any of the received answers that are arbitrary answers;   removing, by the computing device, the identified arbitrary answers from the received answers; and   designating, by the computing device in response to the removing, as final answers any remaining received answers.   
     
     
         2 . The method of  claim 1  further comprising sending, in response to the designating, the final answers to the first source. 
     
     
         3 . The method of  claim 1  where the computing device is configured for performing the method without a priori knowledge of the common-sense problem. 
     
     
         4 . The method of  claim 1  further comprising inhibiting compensation to a source that does not contribute an answer to the common-sense problem. 
     
     
         5 . The method of  claim 1  where the first source and the second source are the same source. 
     
     
         6 . The method of  claim 1  where the second source comprises at least one human. 
     
     
         7 . The method of  claim 1  where the identifying the arbitrary answers is based on modeling the arbitrary answers as a uniform distribution. 
     
     
         8 . A computing device comprising:
 memory;   a processor coupled to the memory and via which the computing device:
 orders answers according to their frequency of occurrence; 
 determines a relative difference for each neighboring pair of the ordered answers, the relative distance based on the frequency of occurrence of each ordered answer of the each neighboring pair; and 
 designates as final answers any of the ordered answers that have a frequency of occurrence that is greater than a frequency of occurrence of an ordered answer of a neighboring pair that has a greatest relative distance of the neighboring pairs. 
   
     
     
         9 . The computing device of  claim 8  where the relative distance is determined based on calculating a slope. 
     
     
         10 . The computing device of  claim 8  where the answers are directed to labeling an image. 
     
     
         11 . The computing device of  claim 10  where the labeling comprises a process including a plurality of refining stages. 
     
     
         12 . The computing device of  claim 11  where the plurality of refining stages comprise collecting candidate labels. 
     
     
         13 . The computing device of  claim 12  where the plurality of refining stages comprise further refining the candidate labels based on multiple choices. 
     
     
         14 . The computing device of  claim 12  where the plurality of refining stages comprise further refining based on locating an object in the image that corresponds to at least one of the refined candidate labels 
     
     
         15 . At least one computer storage device that comprises computer-executable instructions that, based on execution by a computing device, configure the computing device to perform actions comprising:
 ordering, by a computing device, answers according to their frequency of occurrence;   determining, by a computing device, a relative difference for each neighboring pair of the ordered answers, the relative distance based on the frequency of occurrence of each ordered answer of the each neighboring pair; and   designating, by a computing device, as final answers any of the ordered answers that have a frequency of occurrence that is greater than a frequency of occurrence of an ordered answer of a neighboring pair that has a greatest relative distance of the neighboring pairs.   
     
     
         16 . The at least one computer storage device of  claim 15  where the determining the relative distance comprises calculating a slope. 
     
     
         17 . The at least one computer storage device of  claim 15  where the answers are directed to labeling an image. 
     
     
         18 . The at least one computer storage device of  claim 17  where the labeling comprises a process including a plurality of refining stages. 
     
     
         19 . The at least one computer storage device of  claim 18  where the plurality of refining stages comprise collecting candidate labels. 
     
     
         20 . The at least one computer storage device of  claim 19  where the plurality of refining stages comprise further refining the candidate labels based on multiple choices, and further refining based on locating an object in the image that corresponds to at least one of the refined candidate labels.