System and method for determining quality metrics for a question set

Abstract

A computer-implemented method is provided for determining quality metrics for a question set. In an implementation, a test question set model may be produced based upon calculated quality metrics of a test question set with respect to a test corpus, and including features representing quality metrics. The test question set model may be compared to a baseline question set model based on a distance calculated between one or more projected model features of the baseline question set model and one or more runtime model features of the test question set model. Contents of the test question set may be adjusted based upon the calculated distance.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 producing, by a processor, a test question set model based upon, at least in part, calculated quality metrics of a test question set with respect to a test corpus, and including a plurality of test question set model features representing quality metrics for the test question set in the test question set model to define coverage between the test question set and the test corpus based on one or more possible candidate answers to the test question set identified from the test corpus, wherein the calculated quality metrics comprise one or more of corpus coverage metrics, corpus non-coverage metrics, a weak coverage rate, an accuracy rate of the test question set, a recall rate of the test question set, breadth metrics, and depth metrics;   comparing, by the processor, the test question set model to a baseline question set model based on calculating a distance between one or more projected model features of the baseline question set model and one or more runtime model features of the test question set model, wherein the test question set model and the baseline question set model each comprise vectors and the distance comprises a vector distance difference;   adjusting, by the processor, contents of the test question set based upon, at least in part, the calculated distance between the projected model features of the baseline question set model and the runtime model features of the test question set model; and   testing a question answering computer system based on the adjusted contents of the test question set.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the baseline question set model is produced based on calculated quality metrics of a baseline question set with respect to a baseline corpus and includes a plurality of baseline question set model features representing quality metrics for the baseline question set. 
     
     
         3 . The computer-implemented method of  claim 2 , wherein the baseline question set model is selected based upon, at least in part, a domain distance between the baseline corpus and the test corpus. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein the calculated quality metrics for the test question set model are calculated using a static question set analysis tool. 
     
     
         5 . The computer-implemented method of  claim 1 , further including:
 projecting the test question set accuracy from the runtime model features of the baseline question set by analyzing the distance between the baseline question set model and the test question set model.   
     
     
         6 . The computer-implemented method of  claim 1 , further including:
 applying machine learning to tune the test question set model by rewarding prominent features of the test question set and penalizing less prominent features of the test question set.   
     
     
         7 . The computer-implemented method of  claim 1 , further including:
 identifying a level of coverage for the test question set; and   identifying a level of non-coverage for the test question set.   
     
     
         8 . A system comprising:
 at least one processor device and at least one memory architecture coupled with the at least one processor device, the at least one processor device configured for:
 producing a test question set model based upon, at least in part, calculated quality metrics of a test question set with respect to a test corpus, and including a plurality of test question set model features representing quality metrics for the test question set in the test question set model to define coverage between the test question set and the test corpus based on one or more possible candidate answers to the test question set identified from the test corpus, wherein the calculated quality metrics comprise one or more of corpus coverage metrics, corpus non-coverage metrics, a weak coverage rate, an accuracy rate of the test question set, a recall rate of the test question set, breadth metrics, and depth metrics; 
 comparing the test question set model to a baseline question set model based on calculating a distance between one or more projected model features of the baseline question set model and one or more runtime model features of the test question set model, wherein the test question set model and the baseline question set model each comprise vectors and the distance comprises a vector distance difference; 
 adjusting contents of the test question set based upon, at least in part, the calculated distance between the projected model features of the baseline question set model and the runtime model features of the test question set model; and 
 testing a question answering computer system based on the adjusted contents of the test question set. 
   
     
     
         9 . The system of  claim 8 , wherein the baseline question set model is produced based on calculated quality metrics of a baseline question set with respect to a baseline corpus and includes a plurality of baseline question set model features representing quality metrics for the baseline question set. 
     
     
         10 . The system of  claim 9 , wherein the baseline question set model is selected based upon, at least in part, a domain distance between the baseline corpus and the test corpus. 
     
     
         11 . The system of  claim 8 , wherein the calculated quality metrics for the test question set model are calculated using a static question set analysis tool. 
     
     
         12 . The system of  claim 8 , wherein the at least one processor device is further configured for:
 projecting the test question set accuracy from the runtime model features of the baseline question set by analyzing the distance between the baseline question set model and the test question set model.   
     
     
         13 . The system of  claim 8 , wherein the at least one processor device is further configured for:
 applying machine learning to tune the test question set model by rewarding prominent features of the test question set and penalizing less prominent features of the test question set.   
     
     
         14 . The system of  claim 8 , wherein the at least one processor device is further configured for:
 identifying a level of coverage for the test question set; and   identifying a level of non-coverage for the test question set.   
     
     
         15 . A computer program product comprising a non-transitory computer readable medium having a plurality of instructions stored thereon, which, when executed by a processor, cause the processor to perform operations including:
 producing a test question set model based upon, at least in part, calculated quality metrics of a test question set with respect to a test corpus, and including a plurality of test question set model features representing quality metrics for the test question set in the test question set model to define coverage between the test question set and the test corpus based on one or more possible candidate answers to the test question set identified from the test corpus, wherein the calculated quality metrics comprise one or more of corpus coverage metrics, corpus non-coverage metrics, a weak coverage rate, an accuracy rate of the test question set, a recall rate of the test question set, breadth metrics, and depth metrics;   comparing the test question set model to a baseline question set model based on calculating a distance between one or more projected model features of the baseline question set model and one or more runtime model features of the test question set model, wherein the test question set model and the baseline question set model each comprise vectors and the distance comprises a vector distance difference, wherein the baseline question set model is produced based on calculated quality metrics of a baseline question set with respect to a baseline corpus and includes a plurality of baseline question set model features representing quality metrics for the baseline question set;   adjusting contents of the test question set based upon, at least in part, the calculated distance between the projected model features of the baseline question set model and the runtime model features of the test question set model; and   testing a question answering computer system based on the adjusted contents of the test question set.   
     
     
         16 . The computer program product of  claim 15 , wherein the baseline question set model is selected based upon, at least in part, a domain distance between the baseline corpus and the test corpus. 
     
     
         17 . The computer program product of  claim 15 , wherein the calculated quality metrics for the test question set model are calculated using a static question set analysis tool. 
     
     
         18 . The computer program product of  claim 15 , further including instructions for:
 projecting the test question set accuracy from the runtime model features of the baseline question set by analyzing the distance between the baseline question set model and the test question set model.   
     
     
         19 . The computer program product of  claim 15 , further including instructions for:
 applying machine learning to tune the test question set model by rewarding prominent features of the test question set and penalizing less prominent features of the test question set.   
     
     
         20 . The computer program product of  claim 15 , further including instructions for:
 identifying a level of coverage for the test question set; and   identifying a level of non-coverage for the test question set.