Process for controlling continuously learning models

Abstract

The subject disclosure relates generally to a method of controlling advancement of a continuous learning model as new versions of the model are generated through the continuous learning process. an update component that updates an original algorithm to a new algorithm based on expert analysis of an outcome of the original algorithm; and an execution component that runs both new and original algorithms together against unlabeled data, wherein outcome differences between the new and original algorithms are raised for expert review; and a comparison component that compares new and original model algorithm performances by evaluating counts corresponding to when each of the respective new or original algorithms are correct and when their outputs are similar to determine if a new model should replace an original model.

What is claimed is: 
     
         1 . A system, comprising:
 a processor that executes computer-executable components stored in a non-transitory computer-readable memory, the computer-executable components comprising:
 an update component that updates an original algorithm to a new algorithm based on expert analysis of an outcome of the original algorithm; and 
 an execution component that runs both new and original algorithms together against unlabeled data wherein outcome differences between the new and original algorithms are raised for expert review; and 
 a comparison component that compares new and original model algorithm performances by evaluating counts corresponding to when each of the respective new or original algorithms are correct and when their outputs are similar to determine if a new model should replace an original model. 
   
     
     
         2 . The system of  claim 1 , wherein improved performance of an event detection algorithm is verified by offering an expert to annotate events, wherein outputs of a trained algorithm and the original algorithm differ, and wherein a decision of taking a modified algorithm into use is based on accuracy results associated with the annotated events. 
     
     
         3 . The system of  claim 1 , wherein the expert review can be from at least one of a human or system, with additional information data in use which collected with invasive methods or data collected later in time. 
     
     
         4 . The system of  claim 1 , wherein events are raised for re-annotation in an on-line process for data already collected but without verified annotations. 
     
     
         5 . The system of  claim 1 , wherein within a comparison, n (a number of) annotations are raised for re-annotation and a trained model needs to outperform an original model with events in scope. 
     
     
         6 . The system of  claim 1  wherein methods to update an algorithm include at least one of: training methods utilizing existing databases, or methods utilizing new sample(s) only. 
     
     
         7 . The system of  claim 1 , wherein the annotations are weighted based on their estimated criticality or importance in a model selection/determination/decision process. 
     
     
         8 . The system of  claim 1 , wherein acceptance criteria for either model is pre-defined with limits developed by an expert in continuous learning models. 
     
     
         9 . The system of  claim 8 , wherein if the acceptance criteria is not satisfied due to low sample sizes below a defined threshold, the comparison is continued by feeding new data samples to the models. 
     
     
         10 . A computer-implemented method, comprising:
 updating, utilizing a processor operatively coupled to a memory, an original algorithm to a new algorithm based on expert analysis of an outcome of the original algorithm; and   executing, utilizing a processor, running both new and original algorithms together against unlabeled data, and selected samples, wherein outcome differences between the new and original algorithms are raised for expert review; and   comparing, utilizing a processor, algorithm performances by evaluating counts wherein each of the respective algorithms are correct and when their outputs are similar to determine if a new model should replace an original model.   
     
     
         11 . The method of  claim 10 , further comprising analyzing improved performance of event detection algorithms that are verified by offering an expert to annotate events, wherein outputs of a trained algorithm and the original algorithm differ, and wherein the determination of taking a modified algorithm into use is based on accuracy results associated with the annotations. 
     
     
         12 . The method of  claim 10 , further comprising the expert analysis being performed by a human or system, with additional information data in use collected with invasive methods or data collected later in time. 
     
     
         13 . The method of  claim 10 , further comprising events that are raised for re-annotation in an on-line process for data already collected but without verified annotations. 
     
     
         14 . The method of  claim 10 , further comprising a comparison in which n (a number of) annotations are raised for re-annotation and a trained model needs to outperform an original model with the events in scope. 
     
     
         15 . The method of  claim 10 , further comprising updating an algorithm utilizing existing databases or new sample(s). 
     
     
         16 . The method of  claim 10 , further comprising annotations that are weighted based on their estimated criticality or importance in a model selection/determination/decision process. 
     
     
         17 . The method of  claim 10 , further comprising utilizing acceptance criteria for either model that is based on pre-defined limits developed by an expert in continuous learning models. 
     
     
         18 . The method of  claim 17 , further comprising utilizing acceptance criteria that if not satisfied due to low sample sizes below a defined threshold, the comparison is continued by feeding new data samples to the models. 
     
     
         19 . A computer program product for facilitating the control of a continuous learning model, the computer program product comprising a non-transitory computer-readable memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
 update an original algorithm to a new algorithm based on expert analysis of the outcome of the original algorithm; and   execute both new and original version algorithms together against un-labeled data, and selected samples, where the algorithm outcome differences are raised for expert review; and   compare the new and original model algorithm performances by evaluating the counts where each of the new or original algorithms are correct and when their outputs are similar to determine if a new model should replace an original model.   
     
     
         20 . The computer program product of  claim 19 , wherein the program instructions are executable by the processor to cause the processor to analyze improved performance of event detection algorithm that is verified by offering an expert to annotate the events, where the outputs of the trained algorithm and the original algorithm differ, and where the decision of taking the modified algorithm into use is based on accuracy results with these annotations.