Dynamic outlier bias reduction system and method

Abstract

A system and method is described herein for data filtering to reduce functional, and trend line outlier bias. Outliers are removed from the data set through an objective statistical method. Bias is determined based on absolute, relative error, or both. Error values are computed from the data, model coefficients, or trend line calculations. Outlier data records are removed when the error values are greater than or equal to the user-supplied criteria. For optimization methods or other iterative calculations, the removed data are re-applied each iteration to the model computing new results. Using model values for the complete dataset, new error values are computed and the outlier bias reduction procedure is re-applied. Overall error is minimized for model coefficients and outlier removed data in an iterative fashion until user defined error improvement limits are reached. The filtered data may be used for validation, outlier bias reduction and data quality operations.

We claim: 
     
         1 . A computer implemented method comprising the steps of:
 electronically receiving, by a specially programmed computing system, at least one error threshold criteria and a data set;   performing, by the specially programmed computing system, a first iteration of outlier bias reduction using a model that comprises at least one coefficient, wherein performing the first iteration of outlier bias reduction comprises the steps of:
 determining a set of predicted values by applying the model to the data set; 
 comparing the set of predicted values to the data set to produce at least one set of error values; 
 removing one or more data values from the data set as data outliers to form an outlier filtered data set, wherein the data outliers are determined from the at least one set of error values and the at least one error threshold criteria; and 
 constructing an updated model comprising at least one updated coefficient using the outlier filtered data set; and 
   performing, by the specially programmed computing system, a second iteration of outlier bias reduction when at least one termination criteria is not satisfied, wherein performing the second iteration of outlier bias reduction comprises determining a set of second predicted values by applying the updated model to the data set.   
     
     
         2 . The computer-implemented method of  claim 1 , performing, by the specially programmed computing system, the first iteration of outlier bias reduction using the model that comprises the at least one coefficient, wherein performing the first iteration of outlier bias reduction further comprises the steps of:
 determining at least one set of first improvement error values for the data set;   determining at least one set of second improvement error values for the outlier filtered data set; and   comparing the at least one set of first improvement error values with the at least one set of second improvement error values.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the at least one termination criteria is not satisfied is determined from the comparison of the at least one set of first improvement error values with the at least one set of second improvement error values. 
     
     
         4 . The computer-implemented method of  claim 3 , wherein the at least one termination criteria is not satisfied is determined from whether the at least one termination criteria has an improvement value that does not exceed the difference of the at least one set of first improvement error values and the at least one set of second improvement error values. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the at least one set of first improvement values are standard error values. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the at least one set of first improvement values are coefficient of determination values. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein the at least one termination criteria is a specified number of iterations. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the at least one termination criteria is a convergence criteria. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein the at least one set of error values comprises a set of relative error values and a set of absolute error values. 
     
     
         10 . The computer-implemented method of  claim 9 , wherein the one or more data values from the data set are data outliers are determined from whether the relative error values of the set of relative error values and the absolute error values of the set of error values for the one or more data values from the data set exceed the at least one error threshold criteria. 
     
     
         11 . The computer-implemented method of  claim 1 , wherein performing, by the specially programmed computing system, the second iteration of outlier bias reduction when at least one termination criteria is not satisfied further comprises the steps of:
 comparing the set of second predicted values to the data set to produce at least one set of second error values;   removing one or more data values from the data set as data outliers to form a second outlier filtered data set, wherein the data outliers are determined from the at least one set of second error values and the at least one error threshold criteria; and   constructing a second iteration updated model using the second outlier filtered data set.   
     
     
         12 . The computer-implemented method of  claim 1 , wherein performing, by the specially programmed computing system, the second iteration of outlier bias reduction when the at least one termination criteria is not satisfied further comprises the steps of recombining the outlier filtered data set with the data outliers to produce the data set. 
     
     
         13 . A system specialized for reducing outlier bias comprising:
 a server, comprising a processor and a non-transient storage subsystem;   a database stored by the non-transient storage subsystem comprising a data set associated with at least one target variable;   a computer program stored by the non-transient storage subsystem comprising instructions, when executed by the processor, cause the system specialized for reducing outlier bias:
 receive at least one error threshold criteria and the data set via the database; 
 perform a first iteration of outlier bias reduction for the data set that comprises:
 determining a set of predicted values by applying a model comprising at least one coefficient to the data set; 
 comparing the set of predicted values to the data set to produce at least one set of error values; 
 removing a plurality of data outliers from the data set determined from the at least one set of error values and the at least one error threshold criteria to generate an outlier filtered data set; and 
 constructing an updated model comprising at least one updated coefficient from the outlier filtered data set; and 
 
 perform a second iteration of outlier bias reduction for the data set based upon a determination that at least one termination criteria is not satisfied, wherein performing the second iteration of outlier bias reduction comprises determining a set of second predicted values by applying the updated model to the data set. 
   
     
     
         14 . The system of  claim 13 , wherein the instructions, when executed by the processor, cause the system specialized for reducing outlier bias to perform the second iteration of outlier bias reduction for the data set that further comprises recombining the outlier filtered data set with the data outliers to produce the data set. 
     
     
         15 . The system of  claim 13 , wherein the instructions, when executed by the processor, cause the system specialized for reducing outlier bias to perform the first iteration of outlier bias reduction for the data set that further comprises:
 obtaining at least one set of first improvement error values for the data set;   obtaining at least one set of second improvement error values for the outlier filtered data set; and   determining a plurality of difference between the at least one set of first improvement error values with the at least one set of second improvement error values.   
     
     
         16 . The system of  claim 15 , wherein the instructions, when executed by the processor, cause the system specialized for reducing outlier bias to perform the first iteration of outlier bias reduction for the data set that further comprises determining whether an improvement value exceeds the differences between the at least one set of first improvement error values with the at least one set of second improvement error values. 
     
     
         17 . The system of  claim 13 , wherein the instructions, when executed by the processor, cause the system specialized for reducing outlier bias to perform the second iteration of outlier bias reduction for the data set that further comprises:
 comparing the set of second predicted values to the data set to produce at least one set of second error values;   removing a plurality of second data outliers from the data set determined from the at least one set of second error values and the at least one error threshold criteria to generate a second outlier filtered data set; and   constructing second iteration updated model comprising at least one second updated coefficient from the second outlier filtered data set.   
     
     
         18 . An apparatus specialized for reducing outlier bias comprising:
 a processor coupled to a non-transitory computer readable medium, wherein the non-transitory computer readable medium comprises instructions when executed by the processor causes the apparatus specialized for reducing outlier bias to:
 receive at least one error threshold criteria and a data set via one or more data inputs, wherein the data set is associated with at least one target variable; 
 execute a first iteration of outlier bias reduction for the data set by implementing at least the following:
 determine a set of predicted values by applying a model comprising at least one coefficient to the data set; 
 compare the set of predicted values to the data set to produce at least one set of error values; 
 remove a plurality of data outliers from the data set determined from the at least one set of error values and the at least one error threshold criteria to generate an outlier filtered data set; and 
 construct an updated model comprising at least one updated coefficient from the outlier filtered data set; and 
 
 execute a second iteration of outlier bias reduction for the data set based upon a determination that at least one termination criteria is not satisfied by implementing at least the following: determine a set of second predicted values by applying the updated model to the data set. 
   
     
     
         19 . The apparatus of  claim 18 , wherein the instructions, when executed by the processor, cause the apparatus specialized for reducing outlier bias to execute a second iteration of outlier bias reduction for the data set based upon a determination that at least one termination criteria is not satisfied by further implementing at least the following:
 recombine the outlier filtered data set with the data outliers to produce the data set;   compare the set of second predicted values to the data set to produce at least one set of second error values;   remove a plurality of second data outliers from the data set determined from the at least one set of second error values and the at least one error threshold criteria to generate a second outlier filtered data set; and   construct a second iteration updated model comprising at least one second updated coefficient from the second outlier filtered data set.   
     
     
         20 . The apparatus of  claim 18 , wherein the instructions, when executed by the processor, cause the apparatus specialized for reducing outlier bias to execute the first iteration of outlier bias reduction for the data set by further implementing at least the following:
 obtain at least one set of first improvement error values for the data set;   obtain at least one set of second improvement error values for the outlier filtered data set;   determine a plurality of difference between the at least one set of first improvement error values with the at least one set of second improvement error values; and   determine whether an improvement value exceeds the differences between the at least one set of first improvement error values with the at least one set of second improvement error values.