Dynamic outlier bias reduction system and method

Abstract

A system and method is described herein for data filtering to reduce functional, and trend line outlier bias. Outliers are removed from the data set through an objective statistical method. Bias is determined based on absolute, relative error, or both. Error values are computed from the data, model coefficients, or trend line calculations. Outlier data records are removed when the error values are greater than or equal to the user-supplied criteria. For optimization methods or other iterative calculations, the removed data are re-applied each iteration to the model computing new results. Using model values for the complete dataset, new error values are computed and the outlier bias reduction procedure is re-applied. Overall error is minimized for model coefficients and outlier removed data in an iterative fashion until user defined error improvement limits are reached. The filtered data may be used for validation, outlier bias reduction and data quality operations.

We claim: 
     
         1 . A system specialized for assessing the viability of a data set for developing a model for a facility, comprising:
 an input unit for inputting one or more data sets to be processed, wherein the input unit comprises a measuring device configured to:
 measure one or more target variables for a facility; and 
 provide a corresponding data set for each of the target variables; 
   a computing unit coupled to the input unit and for processing the one or more data sets, wherein the computing unit comprises a processor and a non-transient storage subsystem; and   an output unit coupled to the computing unit and for outputting one or more of the processed data sets received from the computing unit,   a computer program stored by the non-transient storage subsystem comprising instructions, when executed by the processor, cause the system specialized for assessing the viability of the corresponding data set for developing a model to perform at least the following:
 generate a random data set from the corresponding data set; 
 obtain a set of bias criteria values used to determine one or more outliers; 
 perform dynamic outlier bias reduction on the corresponding data set for one or more bias criteria values of the set of bias criteria values to generate one or more outlier bias reduced target data sets; 
 perform dynamic outlier bias reduction on the random data set for the one or more bias criteria values of the set of bias criteria values to generate one or more outlier bias reduced random data sets; 
 calculate a set of target error values for the one or more outlier bias reduced target data sets and a set of random error values for the one or more outlier bias reduced random data sets; 
 calculate a set of target correlation coefficients for the one or more outlier bias reduced target data sets and a set of random correlation coefficients for the outlier bias reduced random data set; 
 construct a first bias criteria curve for the corresponding data set and a second bias criteria curve for the random data set from the one or more bias criteria values, the set of target error values, the set of random error values, the set of target correlation coefficients, and the set of random correlation coefficients; and 
 compare the first bias criteria curve and the second bias criteria curve for determining viability of the corresponding data set used to develop the model. 
   
     
     
         2 . The system of  claim 1 , wherein the output unit is configured display a plot for the first bias criteria curve and the second bias criteria curve. 
     
     
         3 . The system of  claim 1 , wherein the measuring device comprises a sensor configured to detect a compound corresponding to one of the target variables and quantify the compound corresponding to the one of the target variables. 
     
     
         4 . The system of  claim 1 , wherein the compound is a greenhouse chemical gas compound, and wherein the sensor is further configured to detect and quantify the compound corresponding to the one of the target variables continuously. 
     
     
         5 . The system of  claim 1 , wherein the instructions, when executed by the processor, cause the system specialized for assessing the viability of the corresponding data set for developing the model to translate the comparison of the first bias criteria curve and the second bias criteria curve to an automated advice message that indicates the viability of the corresponding data set used to develop the model. 
     
     
         6 . The system of  claim 1 , wherein the instructions, when executed by the processor, cause the system specialized for assessing the viability of the corresponding data set for developing the model to perform dynamic outlier bias reduction on the corresponding data set for the one or more bias criteria values of the set of bias criteria values to generate the one or more outlier bias reduced target data sets by performing at least the following:
 for each of the one or more bias criteria values:
 generate a plurality of model predicted values for the corresponding data set by applying the model to the corresponding data set; 
 compute a plurality of error values determined from the corresponding data set and the model predicted values; 
 compare the error values with the corresponding bias criteria value; 
 remove outliers within the corresponding data set to form the corresponding outlier bias reduced target data set determined from the comparison of the error values with the corresponding bias criteria value; and 
 optimize the model to from an updated model determined from the corresponding outlier bias reduced target data set. 
   
     
     
         7 . The system of  claim 6 , wherein the instructions, when executed by the processor, cause the system specialized for assessing the viability of the corresponding data set for developing the model to perform dynamic outlier bias reduction on the corresponding data set for the one or more bias criteria values of the set of bias criteria values to generate the one or more outlier bias reduced target data sets by performing at least the following:
 for each of the one or more bias criteria values:
 compare the error values with a predefined termination criteria to determine termination of optimizing the model; and 
 generate a plurality of second model predicted values for the corresponding data set by applying the updated model to the corresponding data set when the comparison of the error values and the predefined termination criteria do not represent termination of optimizing the model. 
   
     
     
         8 . The system of  claim 1 , wherein the instructions, when executed by the processor, cause the system specialized for assessing the viability of the corresponding data set for developing the model to compare the first bias criteria curve and the second bias criteria curve for determining viability of the corresponding data set used to develop the model by performing at least the following:
 determine a first bias criteria value on the first bias criteria curve that corresponds to a first target error value of the set of target error values;   determine a second bias criteria value on the second bias criteria curve that corresponds to a first random error value of the set of random error values; and   compare the first bias criteria value with the second bias criteria value, wherein the first target error value and the first random error value are the same.   
     
     
         9 . The system of  claim 1 , wherein the instructions, when executed by the processor, cause the system specialized for assessing the viability of the corresponding data set for developing the model to determine the influence of the dynamic outlier bias reduction for each bias criteria value by performing at least the following: comparing a number of iterations to optimize the model for each of the bias criteria values and comparing the differences in the set of target correlation coefficients. 
     
     
         10 . The system of  claim 1 , wherein the random data set comprises all random data values based on the corresponding data set, and wherein the instructions, when executed by the processor, cause the system specialized for assessing the viability of the corresponding data set for developing the model to perform dynamic outlier bias reduction on the random data set for the one or more bias criteria values of the set of bias criteria values to generate the one or more outlier bias reduced random data sets by performing at least the following:
 for each of the bias criteria values:
 generate a plurality of model predicted values for the random data set by applying the model to the random data set; 
 compute a plurality of error values using the random data set and the model predicted values; 
 compare the error values with the corresponding bias criteria value; 
 remove outliers within the random data set to form the corresponding outlier bias reduced random data set determined from the comparison of the error values with the corresponding bias criteria value; and 
 optimize, by the specially programmed computing system, the model for form an updated model based on the corresponding outlier bias reduced random data set. 
   
     
     
         11 . The system of  claim 1 , wherein at least one of the set of target error value is a standard error, and wherein at least one of the set of target correlation value is a coefficient of determination value. 
     
     
         12 . The system of  claim 1 , wherein the random data set comprises a plurality of random data values generated within a range of a plurality of predicted values of the model. 
     
     
         13 . A system for specialized for assessing the viability of a target data set for developing a mode for a financial instrument, comprising:
 an input unit configured to receive a target data set corresponding to a financial instrument, wherein the target data set comprises a plurality of data values for at least one target variable corresponding to the financial instrument;   a computing unit coupled to the input unit, wherein the computing unit comprises a processor and a non-transient storage subsystem,   a computer program stored by the non-transient storage subsystem comprising instructions, when executed by the processor, cause the system specialized for assessing the viability of the target data set for developing a model to perform at least the following:
 generate a random data set based on the target data set; 
 receive a plurality of bias criteria values used to determine one or more outliers; 
 produce a plurality of outlier bias reduced target data sets that are associated with the bias criteria values by applying a mathematical model and a dynamic outlier bias reduction to the target data set; 
 produce a plurality of outlier bias reduced random data sets that are associated with the bias criteria values by applying the mathematical model and the dynamic outlier bias reduction to the random data set; 
 calculate at least one target error value for each of the outlier bias reduced target data sets and at least one random error value for each of the outlier bias reduced random data sets; 
 calculate at least one target correlation value for each of the outlier bias reduced target data sets and at least one random correlation value for each of the outlier bias reduced random data sets; 
 construct a first bias criteria curve for the target data set on a graph based on the at least one target error value and the at least one target correlation value for each of the outlier bias reduced target data sets; 
 construct a second bias criteria curve for the random data set on the graph based on the at least one random error value and the at least one random correlation value for each of the outlier bias reduced random data sets; and 
 compare the first bias criteria curve and the second bias criteria curve to determine viability of the target data set used for the mathematical model. 
   
     
     
         14 . The system of  13 , wherein the financial instrument is a common stock, and wherein the target variable is the price of the common stock, and wherein the target variable for the financial instrument represents at least one of: dividends, earnings, cash flow, earnings per share, price-to-earnings ratio, and growth rate. 
     
     
         15 . The system of  claim 13 , wherein the output unit is configured display a plot for the first bias criteria curve and the second bias criteria curve. 
     
     
         16 . The system of  claim 13 , wherein the instructions, when executed by the processor, cause the system specialized for assessing the viability of the target data set for developing the model to produce a plurality of outlier bias reduced target data sets that are associated with the bias criteria values by applying a mathematical model and a dynamic outlier bias reduction to the target data set by performing at least the following:
 for each of the one or more bias criteria values:
 generate a plurality of model predicted values for the target data set by applying the mathematical model to the target data set; 
 compute a plurality of error values determined from the target data set and the model predicted values; 
 compare the error values with the corresponding bias criteria value; 
 remove outliers within the target data set to form the corresponding outlier bias reduced target data set determined from the comparison of the error values with the corresponding bias criteria value; and 
 optimize the mathematical model to from an updated mathematical model determined from the corresponding outlier bias reduced target data set. 
   
     
     
         17 . The system of  claim 13 , wherein the instructions, when executed by the processor, cause specialized for assessing the viability of the target data set for developing the model to compare the first bias criteria curve and the second bias criteria curve for determining viability of the target data set used to develop the model by performing at least the following:
 determine a first bias criteria value on the first bias criteria curve that corresponds to the at least one target error value;   determine a second bias criteria value on the second bias criteria curve that corresponds to the at least one random error values, and   compare the first bias criteria value with the second bias criteria value, wherein the at least one target error value and the at least one random error value are the same.   
     
     
         18 . A system for reducing outlier bias in target variables measured for a facility, comprising:
 an input unit for inputting one or more data sets to be processed, wherein the input unit comprises a measuring device configured to:
 measure one or more target variables for the facility; and 
 provide a corresponding data set for each of the target variables; 
   a computing unit coupled to the input unit and for processing the one or more data sets, wherein the computing unit comprises a processor and a non-transient storage subsystem;   an output unit coupled to the computing unit and for outputting one or more of the processed data sets received from the computing unit; and   a computer program stored by the non-transient storage subsystem comprising instructions, when executed by the processor, cause the system specialized for reducing outlier bias in target variables measured for the facility to perform at least the following:
 receive at least one error threshold criteria and the corresponding data set via the database; 
 perform a first iteration of outlier bias reduction for the corresponding data set that comprises:
 determining a set of predicted values by applying a model comprising at least one coefficient to the data set; 
 comparing the set of predicted values to the data set to produce at least one set of error values; 
 removing a plurality of data outliers from the data set determined from the at least one set of error values and the at least one error threshold criteria to generate an outlier filtered data set; and 
 constructing an updated model comprising at least one updated coefficient from the outlier filtered data set; and 
 
 perform a second iteration of outlier bias reduction for the data set based upon a determination that at least one termination criteria is not satisfied, 
 wherein performing the second iteration of outlier bias reduction comprises determining a set of second predicted values by applying the updated model to the data set. 
   
     
     
         19 . The system of  claim 18 , wherein the instructions, when executed by the processor, cause the system specialized for reducing outlier bias to perform the second iteration of outlier bias reduction for the corresponding data set that further comprises recombining the outlier filtered data set with the data outliers to produce the data set. 
     
     
         20 . The system of  claim 18 , wherein the instructions, when executed by the processor, cause the system specialized for reducing outlier bias to perform the second iteration of outlier bias reduction for the corresponding data set that further comprises:
 comparing the set of second predicted values to the corresponding data set to produce at least one set of second error values;   removing a plurality of second data outliers from the corresponding data set determined from the at least one set of second error values and the at least one error threshold criteria to generate a second outlier filtered data set; and   constructing second iteration updated model comprising at least one second updated coefficient from the second outlier filtered data set.