System and method for evaluating the performance and usage of a question answering cognitive computing tool

Abstract

A system and method for evaluating the performance and usage of a cognitive computing tool which answers questions from users. A log file for these interactions includes the questions, the answers and a confidence rating assigned by the tool to each answer. Questions and answers are analyzed to determine validity, accuracy, and categories by subject matter experts or text analytics tools, and the results are added to the log file. Comments and sentiments from users may be analyzed and added to the log file. Additional data about the users, such as identities, demographics, and locations, may be added. Data from the log file may be presented in a dashboard display as metrics, such as trends and comparisons, describing the usage and performance of the cognitive computing tool. Answers may be displayed as they were presented to the users. Selectable filters may be provided to control the data displayed.

We claim: 
     
       1. A computer-implemented method for providing a cognitive computing tool configured to provide at least one answer to each question of a plurality of questions, the computer-implemented method comprising:
 accessing a log file including the plurality of questions, and a confidence rating assigned by the cognitive computing tool to the at least one answer to each question; 
 determining a validity of each question, by an expert in a subject matter of each question, and adding the validity to the log file to create an augmented log file; 
 determining an accuracy of the at least one answer to each question, and adding the accuracy to the augmented log file; 
 determining a category of each question using a text analytics tool, and adding the category of each question to the augmented log file; and 
 formatting by an electronic processor and visually communicating on an electronic display a representation of at least some data contained in the augmented log file as one or more metrics describing performance of the cognitive computing tool. 
 
     
     
       2. The computer-implemented method as set forth in  claim 1 , wherein determining the accuracy of the at least one answer to each question is performed by an expert in the subject matter of each question. 
     
     
       3. The computer-implemented method as set forth in  claim 1 , further comprising assigning a numerical accuracy rating to the accuracy of the at least one answer to each question. 
     
     
       4. The computer-implemented method as set forth in  claim 1 , further including analyzing one or more comments by one or more users, and adding an analysis of the one or more comments to the augmented log file. 
     
     
       5. The computer-implemented method as set forth in  claim 1 , further including analyzing one or more sentiments of one or more users and adding an analysis of the one or more sentiments to the augmented log file. 
     
     
       6. The computer-implemented method as set forth in  claim 1 , wherein the log file further includes an alias for each user of one or more users, a timestamp associated with each question of the plurality of questions, an alternative answer to each question of the plurality of questions provided by the cognitive computing tool, one or more feedbacks provided by the one or more users, and one or more free text comments provided by the one or more users. 
     
     
       7. The computer-implemented method as set forth in  claim 1 , wherein the one or more metrics include a trend over time regarding the accuracy of the at least one answer for each question provided by the cognitive computing tool. 
     
     
       8. The computer-implemented method as set forth in  claim 1 , wherein the one or more metrics includes one or more comparisons of the at least one answer to each question of the plurality of questions. 
     
     
       9. The computer-implemented method as set forth in  claim 8 , wherein the one or more comparisons include comparing a numerical accuracy rating assigned to the accuracy of each answer to a confidence rating assigned to each answer. 
     
     
       10. The computer-implemented method as set forth in  claim 1 , further including adding additional data associated with each user of one or more users to the augmented log file. 
     
     
       11. The computer-implemented method as set forth in  claim 10 , wherein adding the additional data includes adding demographic data about the one or more users to the augmented log file. 
     
     
       12. The computer-implemented method as set forth in  claim 10 , wherein adding the additional data includes adding an identity of each user of the one or more users to the augmented log file. 
     
     
       13. The computer-implemented method as set forth in  claim 10 , wherein adding the additional data includes adding a location of each user of the one or more users to the augmented log file. 
     
     
       14. The computer-implemented method as set forth in  claim 1 , further including visually communicating the at least one answer in a format with which it was communicated to a user. 
     
     
       15. The computer-implemented method as set forth in  claim 1 , further including providing one or more electronic filters which are selectable to cause the electronic processor to electronically filter the at least some of the data visually communicated on the electronic display. 
     
     
       16. A system for evaluating performance of a cognitive computing tool, the system comprising:
 an electronic memory containing an augmented log file for the cognitive computing tool, wherein the augmented log file includes:
 a plurality of questions, and a confidence rating assigned by the cognitive computing tool to at least one answer to each question, 
 a validity of each question of the plurality of questions as determined by an expert in a subject matter of each question, and 
 an accuracy of each answer of the at least one answer to each question; 
 
 an electronic processor configured to determine a category of each question of the plurality of questions using a text analytics tool, and to add the category of each question to the augmented log file; and 
 an electronic display configured to communicate a representation of at least some data contained in the augmented log file as one or more metrics describing the performance of the cognitive computing tool. 
 
     
     
       17. The system as set forth in  claim 16 , wherein the accuracy of each answer is determined by an expert in a subject matter of each answer. 
     
     
       18. The system as set forth in  claim 16 , wherein the one or more metrics include a trend over time regarding the accuracy of the at least one answer for each question. 
     
     
       19. A non-transitory computer readable storage medium with a computer program stored thereon for evaluating performance of a cognitive computing tool, wherein the computer program is configured to instruct a system to perform actions comprising:
 accessing a log file stored in an electronic memory for the cognitive computing tool, wherein the log file includes a plurality of questions, and a confidence rating assigned by the cognitive computing tool to at least one answer to each question; 
 determining a validity of each question, and adding the validity to the log file to create an augmented log file; 
 determining an accuracy of each answer of the at least one answer to each question, and adding the accuracy to the augmented log file; 
 using a text analytics tool to determine a category of each question, and adding the category of each question to the augmented log file stored; and 
 formatting by an electronic processor and visually communicating on an electronic display a representation of at least some data contained in the augmented log file as one or more metrics describing the performance of the cognitive computing tool including a trend over time regarding the accuracy of the at least one answer to each question provided by the cognitive computing tool. 
 
     
     
       20. The non-transitory computer readable storage medium set forth in  claim 19 , wherein the computer program is further configured to instruct a system to assign a numerical accuracy rating to the accuracy of each answer to each question.