Determining Comprehensiveness of Question Paper Given Syllabus

Abstract

A mechanism is provided in a data processing system for determining comprehensiveness of a question paper given a syllabus of topics. An answer and evidence generator of a question answering system executing on the data processing system finds one or more answers based on the syllabus of topics for each question in the question paper. The answer and evidence generator identifies evidence for the one or more answers in the syllabus for each question in the question paper. A concept identifier of the question answering system identifies a set of concepts in the syllabus corresponding to the evidence for each question in the question paper to form a plurality of sets of concepts. The mechanism determines a value for a comprehensiveness metric for the question paper with respect to the syllabus of topics based on the plurality of sets of concepts.

1 . A method, in a data processing system, for determining comprehensiveness of a question paper given a syllabus of topics, the method comprising:
 finding, by an answer and evidence generator of a question answering system executing on the data processing system, one or more answers based on the syllabus of topics for each question in the question paper;   identifying, by the answer and evidence generator, evidence for the one or more answers in the syllabus for each question in the question paper;   identifying, by a concept identifier of the question answering system, a set of concepts in the syllabus corresponding to the evidence for each question in the question paper to form a plurality of sets of concepts; and   determining a value for a comprehensiveness metric for the question paper with respect to the syllabus of topics based on the plurality of sets of concepts.   
     
     
         2 . The method of  claim 1 , wherein finding the one or more answers in the syllabus of topics for each question in the question paper comprises using the question answering system to find one or more answers having a highest confidence score for each question in the question paper. 
     
     
         3 . The method of  claim 1 , wherein determining the set of concepts for a given question comprises determining all evidence concepts in the evidence of the one or more answers and all support concepts that support the evidence concepts, wherein the set of concepts comprises the evidence concepts and the support concepts. 
     
     
         4 . (canceled) 
     
     
         5 . (canceled) 
     
     
         6 . The method of  claim 1 , wherein the comprehensiveness metric comprises a difficulty metric, wherein determining the value for the comprehensiveness metric comprises:
 mapping the sets of concepts to topics in the syllabus;   for each given question in the question paper, building a tree of topics contributing to an answer to the given question, each tree of topics comprising a root node representing a cent al topic, at least one child node representing a topic is having concepts that help in understanding concepts of the central topic, and at least one leaf node representing a topic having fundamental concepts;   building a forest of the trees corresponding to the questions of the question paper; and   determining value to difficulty metric for the question paper to be equal to a depth of the forest.   
     
     
         7 . A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
 find, by an answer and evidence generator of a question answering system executing on the data processing system, one or more answers based on the syllabus of topics for each question in the question paper;   identify, by the answer and evidence generator, evidence for the one or more answers in the syllabus for each question in the question paper;   identify, by a concept identifier of the question answering system, a set of concepts in the syllabus corresponding to the evidence for each question in the question paper to form a plurality of sets of concepts; and   determine a value for a comprehensiveness metric for the question paper with respect to the syllabus of topics based on the plurality of sets of concepts.   
     
     
         8 . The computer program product of  claim 7 , wherein finding the one or more answers in the syllabus of topics for each question in the question paper comprises using the question answering system to find one or more answers having a highest confidence score for each question in the question paper. 
     
     
         9 . The computer program product of  claim 7 , wherein determining the set of concepts for a given question comprises determining all evidence concepts in the evidence of the one or more answers and all support concepts that support the evidence concepts, wherein the set of concepts comprises the evidence concepts and the support concepts. 
     
     
         10 . (canceled) 
     
     
         11 . (canceled) 
     
     
         12 . The computer program product of  claim 7 , wherein the comprehensiveness metric comprises a difficulty metric, wherein determining the value for the comprehensiveness metric further comprises:
 mapping the sets of concepts to topics in the syllabus;   for each given question in the question paper, building a tree of topics contributing to an answer to the given question, each tree of topics comprising a root node representing a central topic, at least one child node representing a topic having concepts that help in understanding concepts of the central topic, and at least one leaf node representing a topic having fundamental concepts;   building a forest of the trees corresponding to the questions of the question paper; and   determining value of the difficulty metric for the question paper to be equal t depth of the forest.   
     
     
         13 . The computer program product of  claim 7 , wherein the computer readable program is stored in a computer readable storage medium in a data processing system and wherein the computer readable program was downloaded over a network from a remote data processing system. 
     
     
         14 . The computer program product of  claim 7 , wherein the computer readable program is stored in a computer readable storage medium in a server data processing system and wherein the computer readable program is downloaded over a network to a remote data processing system for use in a computer readable storage medium with the remote system. 
     
     
         15 . An apparatus comprising:
 a processor; and   a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to:   find, by an answer and evidence generator of a question answering system executing on the data processing system, one or more answers based on the syllabus of topics for each question in the question paper;   identify, by the answer and evidence generator, evidence for the one or more answers in the syllabus for each question in the question paper;   identify, by a concept identifier of the question answering system, a set of concepts in the syllabus corresponding to the evidence for each question in the question paper to form a plurality of sets of concepts; and   determine a value for a comprehensiveness metric for the question paper with respect to the syllabus of topics based on the plurality of sets of concepts.   
     
     
         16 . The apparatus of  claim 15 , wherein finding the one or more answers in the syllabus of topics for each question in the question paper comprises using the question answering system to find one or more answers having a highest confidence score for each question in the question paper. 
     
     
         17 . The apparatus of  claim 15 , wherein determining the set of concepts for a given question comprises determining all evidence concepts in the evidence of the one or more answers and all support concepts that support the evidence concepts, wherein the set of concepts comprises the evidence concepts and the support concepts. 
     
     
         18 . (canceled) 
     
     
         19 . (canceled) 
     
     
         20 . The apparatus of  claim 15 , wherein the comprehensiveness metric comprises a difficulty metric, wherein determining the value for the comprehensiveness metric further comprises:
 mapping the sets of concepts to topics in the syllabus;   for each given question in the question paper, building a tree of topics contributing to an answer to the given question, each tree of topics comprising a root node representing a central topic, at least one child node representing a topic having concepts that help in understanding concepts of the central topic, and at least one leaf node representing a topic having fundamental concepts;   building a forest of the trees corresponding to the questions of the question paper; and   determining value of the difficulty metric for the question paper to be equal to a depth of the forest.   
     
     
         21 . The method of  claim 6 , further comprising performing tree pruning on the trees. 
     
     
         22 . The computer program product of  claim 12 , further comprising performing tree pruning on the trees. 
     
     
         23 . The apparatus of  claim 20 , further comprising performing tree pruning on the trees