FIELD OF THE INVENTION

The present invention relates to the analysis of data where outlier elements are removed (or filtered) from the analysis development. The analysis may be related to the computation of simple statistics or more complex operations involving mathematical models that use data in their development. The purpose of outlier data filtering may be to perform data quality and data validation operations, or to compute representative standards, statistics, data groups that have applications in subsequent analyses, regression analysis, time series analysis or qualified data for mathematical models development.

BACKGROUND

Removing outlier data in standards or data-driven model development is an important part of the pre-analysis work to ensure a representative and fair analysis is developed from the underlying data. For example, developing equitable benchmarking of greenhouse gas standards for carbon dioxide (CO2), ozone (O3), water vapor (H2O), hydrofluorocarbons (HFCs), perfluorocarbons (PFCs), chlorofluorocarbons (CFCs), sulfur hexafluoride (SF6), methane (CH4), nitrous oxide (N2O), carbon monoxide (CO), nitrogen oxides (NOx), and non-methane volatile organic compounds (NMVOCs) emissions requires that collected industrial data used in the standards development exhibit certain properties. Extremely good or bad performance by a few of the industrial sites should not bias the standards computed for other sites. It may be judged unfair or unrepresentative to include such performance results in the standard calculations. In the past, the performance outliers were removed via a semi-quantitative process requiring subjective input. The present system and method is a data-driven approach that performs this task as an integral part of the model development, and not at the pre-analysis or pre-model development stage.

The removal of bias can be a subjective process wherein justification is documented in some form to substantiate data changes. However, any form of outlier removal is a form of data censoring that carries the potential for changing calculation results. Such data filtering may or may not reduce bias or error in the calculation and in the spirit of full analysis disclosure, strict data removal guidelines and documentation to remove outliers needs to be included with the analysis results. Therefore, there is a need in the art to provide a new system and method for objectively removing outlier data bias using a dynamic statistical process useful for the purposes of data quality operations, data validation, statistic calculations or mathematical model development, etc. The outlier bias removal system and method can also be used to group data into representative categories where the data is applied to the development of mathematical models customized to each group. In a preferred embodiment, coefficients are defined as multiplicative and additive factors in mathematical models and also other numerical parameters that are nonlinear in nature. For example, in the mathematical model, f(x,y,z)=a*x+b*yc+d*sin(ez)+f, a, b, c, d, e, and f are all defined as coefficients. The values of these terms may be fixed or part of the development of the mathematical model.

BRIEF SUMMARY

A preferred embodiment includes a computer implemented method for reducing outlier bias comprising the steps of: selecting a bias criteria; providing a data set; providing a set of model coefficients; selecting a set of target values; (1) generating a set of predicted values for the complete data set; (2) generating an error set for the dataset; (3) generating a set of error threshold values based on the error set and the bias criteria; (4) generating, by a processor, a censored data set based on the error set and the set of error threshold values; (5) generating, by the processor, a set of new model coefficients; and (6) using the set of new model coefficients, repeating steps (1)-(5), unless a censoring performance termination criteria is satisfied. In a preferred embodiment, the set of predicted values may be generated based on the data set and the set of model coefficients. In a preferred embodiment, the error set may comprise a set of absolute errors and a set of relative errors, generated based on the set of predicted values and the set of target values. In another embodiment, the error set may comprise values calculated as the difference between the set of predicted values and the set of target values. In another embodiment, the step of generating the set of new coefficients may further comprise the step of minimizing the set of errors between the set of predicted values and the set of actual values, which can be accomplished using a linear, or a non-linear optimization model. In a preferred embodiment, the censoring performance termination criteria may be based on a standard error and a coefficient of determination.

Another embodiment includes a computer implemented method for reducing outlier bias comprising the steps of: selecting an error criteria; selecting a data set; selecting a set of actual values; selecting an initial set of model coefficients; generating a set of model predicted values based on the complete data set and the initial set of model coefficients; (1) generating a set of errors based on the model predicted values and the set of actual values for the complete dataset; (2) generating a set of error threshold values based on the complete set of errors and the error criteria for the complete data set; (3) generating an outlier removed data set, wherein the filtering is based on the complete data set and the set of error threshold values; (4) generating a set of new coefficients based on the filtered data set and the set of previous coefficients, wherein the generation of the set of new coefficients is performed by the computer processor; (5) generating a set of outlier bias reduced model predicted values based on the filtered data set and the set of new model coefficients, wherein the generation of the set of outlier bias reduced model predicted values is performed by a computer processor; (6) generating a set of model performance values based on the model predicted values and the set of actual values; repeating steps (1)-(6), while substituting the set of new coefficients for the set of coefficients from the previous iteration, unless: a performance termination criteria is satisfied; and storing the set of model predicted values in a computer data medium.

Another embodiment includes a computer implemented method for reducing outlier bias comprising the steps of: selecting a target variable for a facility; selecting a set of actual values of the target variable; identifying a plurality of variables for the facility that are related to the target variable; obtaining a data set for the facility, the data set comprising values for the plurality of variables; selecting a bias criteria; selecting a set of model coefficients; (1) generating a set of predicted values based on the complete data set and the set of model coefficients; (2) generating a set of censoring model performance values based on the set of predicted values and the set of actual values; (3) generating an error set based on the set of predicted values and the set of actual values for the target variable; (4) generating a set of error threshold values based on the error set and the bias criteria; (5) generating, by a processor, a censored data set based on the data set and the set of error thresholds; (6) generating, by the processor, a set of new model coefficients based on the censored data set and the set of model coefficients; (7) generating, by the processor, a set of new predicted values based on the data set and the set of new model coefficients; (8) generating a set of new censoring model performance values based on the set of new predicted values and the set of actual values; using the set of new coefficients, repeating steps (1)-(8) unless a censoring performance termination criteria is satisfied; and storing the set of new model predicted values in a computer data medium.

Another embodiment includes a computer implemented method for reducing outlier bias comprising the steps of: determining a target variable for a facility, wherein the target variable is a metric for an industrial facility related to its production, financial performance, or emissions; identifying a plurality of variables for the facility, wherein the plurality of variables comprises: a plurality of direct variables for the facility that influence the target variable; and a set of transformed variables for the facility, each transformed variable is a function of at least one direct facility variable that influences the target variable; selecting an error criteria comprising: an absolute error, and a relative error; obtaining a data set for the facility, wherein the data set comprises values for the plurality of variables; selecting a set of actual values of the target variable; selecting an initial set of model coefficients; generating a set of model predicted values based on the complete data set and the initial set of model coefficients; generating a complete set of errors based on the set of model predicted values and the set of actual values, wherein the relative error is calculated using the formula: Relative Errorm((Predicted Valuem−Actual Valuem)/Actual Valuem)2wherein ‘m’ is a reference number, and wherein the absolute error is calculated using the formula: Absolute Errorm=(Predicted Valuem−Actual Valuem)2; generating a set of model performance values based on the set of model predicted values and the set of actual values, wherein the set of overall model performance values comprises of: a first standard error, and a first coefficient of determination; (1) generating a set of errors based on the model predicted values and the set of actual values for the complete dataset; (2) generating a set of error threshold values based on the complete set of errors and the error criteria for the complete data set; (3) generating an outlier removed data set by removing data with error values greater than or equal to the error threshold values, wherein the filtering is based on the complete data set and the set of error threshold values; (4) generating a set of outlier bias reduced model predicted values based on the outlier removed data set and the set of model coefficients by minimizing the error between the set of predicted values and the set of actual values using at least one of: a linear optimization model, and a nonlinear optimization model, wherein the generation of the new model predicted values is performed by a computer processor; (5) generating a set of new coefficients based on the outlier removed data set and the previous set of coefficients, wherein the generation of the set of new coefficients is performed by the computer processor; (6) generating a set of overall model performance values based on the set of new predicted model values and the set of actual values, wherein the set of model performance values comprise: a second standard error, and a second coefficient of determination; repeating steps (1)-(6), while substituting the set of new coefficients for the set of coefficients from the previous iteration, unless: a performance termination criteria is satisfied, wherein the performance termination criteria comprises: a standard error termination value and a coefficient of determination termination value, and wherein satisfying the performance termination criteria comprises: the standard error termination value is greater than the difference between the first and second standard error, and the coefficient of determination termination value is greater than the difference between the first and second coefficient of determination; and storing the set of new model predicted values in a computer data medium.

Another embodiment includes a computer implemented method for reducing outlier bias comprising the steps of: selecting an error criteria; selecting a data set; selecting a set of actual values; selecting an initial set of model predicted values; determining a set of errors based on the set of model predicted values and the set of actual values; (1) determining a set of error threshold values based on the complete set of errors and the error criteria; (2) generating an outlier removed data set, wherein the filtering is based on the data set and the set of error threshold values; (3) generating a set of outlier bias reduced model predicted values based on the outlier removed data set and the previous model predicted values, wherein the generation of the set of outlier bias reduced model predicted values is performed by a computer processor; (4) determining a set of errors based on the set of new model predicted values and the set of actual values; repeating steps (1)-(4), while substituting the set of new model predicted values for the set of model predicted values from the previous iteration, unless: a performance termination criteria is satisfied; and storing the set of outlier bias reduced model predicted values in a computer data medium.

Another embodiment includes a computer implemented method for reducing outlier bias comprising the steps of: determining a target variable for a facility; identifying a plurality of variables for the facility, wherein the plurality of variables comprises: a plurality of direct variables for the facility that influence the target variable; and a set of transformed variables for the facility, each transformed variable being a function of at least one direct facility variable that influences the target variable; selecting an error criteria comprising: an absolute error, and a relative error; obtaining a data set, wherein the data set comprises values for the plurality of variables, and selecting a set of actual values of the target variable; selecting an initial set of model coefficients; generating a set of model predicted values by applying a set of model coefficients to the data set; determining a set of performance values based on the set of model predicted values and the set of actual values, wherein the set of performance values comprises: a first standard error, and a first coefficient of determination; (1) generating a set of errors based on the set of model predicted values and the set of actual values for the complete dataset, wherein the relative error is calculated using the formula: Relative Errorm=((Predicted Valuem−Actual Valuem)/Actual Valuem)2, wherein ‘in’ is a reference number, and wherein the absolute error is calculated using the formula: Absolute Errorm=(Predicted Valuem−Actual Valuem)2) (2) generating a set of error threshold values based on the complete set of errors and the error criteria for the complete data set; (3) generating an outlier removed data set by removing data with error values greater than or equal to the set of error threshold values, wherein the filtering is based on the data set and the set of error threshold values; (4) generating a set of new coefficients based on the outlier removed data set and the set of previous coefficients (5) generating a set of outlier bias reduced model predicted values based on the outlier removed data set and the set of new model coefficient by minimizing the error between the set of predicted values and the set of actual values using at least one of: a linear optimization model, and a nonlinear optimization model, wherein the generation of the model predicted values is performed by a computer processor; (6) generating a set of updated performance values based on the set of outlier bias reduced model predicted values and the set of actual values, wherein the set of updated performance values comprises: a second standard error, and a second coefficient of determination; repeating steps (1)-(6), while substituting the set of new coefficients for the set of coefficients from the previous iteration, unless: a performance termination criteria is satisfied, wherein the performance termination criteria comprises: a standard error termination value, and a coefficient of determination termination value, and wherein satisfying the performance termination criteria comprises the standard error termination value is greater than the difference between the first and second standard error, and the coefficient of determination termination value is greater than the difference between the first and second coefficient of determination; and storing the set of outlier bias reduction factors in a computer data medium.

Another embodiment includes a computer implemented method for assessing the viability of a data set as used in developing a model comprising the steps of: providing a target data set comprising a plurality of data values; generating a random target data set based on the target dataset; selecting a set of bias criteria values; generating, by a processor, an outlier bias reduced target data set based on the data set and each of the selected bias criteria values; generating, by the processor, an outlier bias reduced random data set based on the random data set and each of the selected bias criteria values; calculating a set of error values for the outlier bias reduced data set and the outlier bias reduced random data set; calculating a set of correlation coefficients for the outlier bias reduced data set and the outlier bias reduced random data set; generating bias criteria curves for the data set and the random data set based on the selected bias criteria values and the corresponding error value and correlation coefficient; and comparing the bias criteria curve for the data set to the bias criteria curve for the random data set. The outlier bias reduced target data set and the outlier bias reduced random target data set are generated using the Dynamic Outlier Bias Removal methodology. The random target data set can comprise of randomized data values developed from values within the range of the plurality of data values. Also, the set of error values can comprise a set of standard errors, and wherein the set of correlation coefficients comprises a set of coefficient of determination values. Another embodiment can further comprise the step of generating automated advice regarding the viability of the target data set to support the developed model, and vice versa, based on comparing the bias criteria curve for the target data set to the bias criteria curve for the random target data set. Advice can be generated based on parameters selected by analysts, such as a correlation coefficient threshold and/or an error threshold. Yet another embodiment further comprises the steps of: providing an actual data set comprising a plurality of actual data values corresponding to the model predicted values; generating a random actual data set based on the actual data set; generating, by a processor, an outlier bias reduced actual data set based on the actual data set and each of the selected bias criteria values; generating, by the processor, an outlier bias reduced random actual data set based on the random actual data set and each of the selected bias criteria values; generating, for each selected bias criteria, a random data plot based on the outlier bias reduced random target data set and the outlier bias reduced random actual data; generating, for each selected bias criteria, a realistic data plot based on the outlier bias reduced target data set and the outlier bias reduced actual target data set; and comparing the random data plot with the realistic data plot corresponding to each of the selected bias criteria.

A preferred embodiment includes a system comprising: a server, comprising: a processor, and a storage subsystem; a database stored by the storage subsystem comprising: a data set; and a computer program stored by the storage subsystem comprising instructions that, when executed, cause the processor to: select a bias criteria; provide a set of model coefficients; select a set of target values; (1) generate a set of predicted values for the data set; (2) generate an error set for the dataset; (3) generate a set of error threshold values based on the error set and the bias criteria; (4) generate a censored data set based on the error set and the set of error threshold values; (5) generate a set of new model coefficients; and (6) using the set of new model coefficients, repeat steps (1)-(5), unless a censoring performance termination criteria is satisfied. In a preferred embodiment, the set of predicted values may be generated based on the data set and the set of model coefficients. In a preferred embodiment, the error set may comprise a set of absolute errors and a set of relative errors, generated based on the set of predicted values and the set of target values. In another embodiment, the error set may comprise values calculated as the difference between the set of predicted values and the set of target values. In another embodiment, the step of generating the set of new coefficients may further comprise the step of minimizing the set of errors between the set of predicted values and the set of actual values, which can be accomplished using a linear, or a non-linear optimization model. In a preferred embodiment, the censoring performance termination criteria may be based on a standard error and a coefficient of determination.

Another embodiment of the present invention includes a system comprising: a server, comprising: a processor, and a storage subsystem; a database stored by the storage subsystem comprising: a data set; and a computer program stored by the storage subsystem comprising instructions that, when executed, cause the processor to: select an error criteria; select a set of actual values; select an initial set of coefficients; generate a complete set of model predicted values from the data set and the initial set of coefficients; (1) generate a set of errors based on the model predicted values and the set of actual values for the complete dataset; (2) generate a set of error threshold values based on the complete set of errors and the error criteria for the complete data set; (3) generate an outlier removed data set, wherein the filtering is based on the complete data set and the set of error threshold values; (4) generate a set of outlier bias reduced model predicted values based on the outlier removed data set and the set of coefficients, wherein the generation of the set of outlier bias reduced model predicted values is performed by a computer processor; (5) generate a set of new coefficients based on the outlier removed data set and the set of previous coefficients, wherein the generation of the set of new coefficients is performed by the computer processor; (6) generate a set of model performance values based on the outlier bias reduced model predicted values and the set of actual values; repeat steps (1)-(6), while substituting the set of new coefficients for the set of coefficients from the previous iteration, unless: a performance termination criteria is satisfied; and store the set of overall outlier bias reduction model predicted values in a computer data medium.

Yet another embodiment includes a system comprising: a server, comprising: a processor, and a storage subsystem; a database stored by the storage subsystem comprising: a target variable for a facility; a set of actual values of the target variable; a plurality of variables for the facility that are related to the target variable; a data set for the facility, the data set comprising values for the plurality of variables; and a computer program stored by the storage subsystem comprising instructions that, when executed, cause the processor to: select a bias criteria; select a set of model coefficients; (1) generate a set of predicted values based on the data set and the set of model coefficients; (2) generate a set of censoring model performance values based on the set of predicted values and the set of actual values; (3) generate an error set based on the set of predicted values and the set of actual values for the target variable; (4) generate a set of error threshold values based on the error set and the bias criteria; (5) generate a censored data set based on the data set and the set of error thresholds; (6) generate a set of new model coefficients based on the censored data set and the set of model coefficients; (7) generate a set of new predicted values based on the data set and the set of new model coefficients; (8) generate a set of new censoring model performance values based on the set of new predicted values and the set of actual values; using the set of new coefficients, repeat steps (1)-(8) unless a censoring performance termination criteria is satisfied; and storing the set of new model predicted values in the storage subsystem.

Another embodiment includes a system comprising: a server, comprising: a processor, and a storage subsystem; a database stored by the storage subsystem comprising: a data set for a facility; and a computer program stored by the storage subsystem comprising instructions that, when executed, cause the processor to: determine a target variable; identify a plurality of variables, wherein the plurality of variables comprises: a plurality of direct variables for the facility that influence the target variable; and a set of transformed variables for the facility, each transformed variables being a function of at least one direct variable that influences the target variable; select an error criteria comprising: an absolute error, and a relative error; select a set of actual values of the target variable; select an initial set of coefficients; generate a set of model predicted values based on the data set and the initial set of coefficients; determine a set of errors based on the set of model predicted values and the set of actual values, wherein the relative error is calculated using the formula: Relative Errorm=((Predicted Valuem−Actual Valuem)/Actual Valuem)2, wherein ‘m’ is a reference number, and wherein the absolute error is calculated using the formula: Absolute Errorm=(Predicted Valuem−Actual Valuem)2; determine a set of performance values based on the set of model predicted values and the set of actual values; wherein the set of performance values comprises: a first standard error, and a first coefficient of determination; (1) generate a set of errors based on the model predicted values and the set of actual values; (2) generating a set of error threshold values based on the complete set of errors and the error criteria for the complete data set; (3) generate an outlier removed data set by filtering data with error values outside the set of error threshold values, wherein the filtering is based on the data set and the set of error threshold values; (4) generate a set of new model predicted values based on the outlier removed data set and the set of coefficients by minimizing an error between the set of model predicted values and the set of actual values using at least one of: a linear optimization model, and a nonlinear optimization model, wherein the generation of the outlier bias reduced model predicted values is performed by a computer processor; (5) generate a set of new coefficients based on the outlier removed data set and the set of previous coefficients, wherein the generation of the set of new coefficients is performed by the computer processor; (6) generate a set of performance values based on the set of new model predicted values and the set of actual values; wherein the set of model performance values comprises: a second standard error, and a second coefficient of determination; repeat steps (1)-(6), while substituting the set of new coefficients for the set of coefficients from the previous iteration, unless: a performance termination criteria is satisfied, wherein the performance termination criteria comprises: a standard error, and a coefficient of determination, and wherein satisfying the performance termination criteria comprises: the standard error termination value is greater than the difference between the first and second standard error, and the coefficient of determination termination value is greater than the difference between the first and second coefficient of determination; and store the set of new model predicted values in a computer data medium.

Another embodiment of the present invention includes a system comprising: a server, comprising: a processor, and a storage subsystem; a database stored by the storage subsystem comprising: a data set, a computer program stored by the storage subsystem comprising instructions that, when executed, cause the processor to: select an error criteria; select a data set; select a set of actual values; select an initial set of model predicted values; determine a set of errors based on the set of model predicted values and the set of actual values; (1) determine a set of error threshold values based on the complete set of errors and the error criteria; (2) generate an outlier removed data set, wherein the filtering is based on the data set and the set of error threshold values; (3) generate a set of outlier bias reduced model predicted values based on the outlier removed data set and the complete set of model predicted values, wherein the generation of the set of outlier bias reduced model predicted values is performed by a computer processor; (4) determine a set of errors based on the set of outlier bias reduction model predicted values and the corresponding set of actual values; repeat steps (1)-(4), while substituting the set of outlier bias reduction model predicted values for the set of model predicted values unless: a performance termination criteria is satisfied; and store the set of outlier bias reduction factors in a computer data medium.

Another embodiment of the present invention includes a system comprising: a server, comprising: a processor, and a storage subsystem; a database stored by the storage subsystem comprising: a data set, a computer program stored by the storage subsystem comprising instructions that, when executed, cause the processor to: determine a target variable; identify a plurality of variables for the facility, wherein the plurality of variables comprises: a plurality of direct variables for the facility that influence the target variable; and a set of transformed variables for the facility, each transformed variable is a function of at least one primary facility variable that influences the target variable; select an error criteria comprising: an absolute error, and a relative error; obtain a data set, wherein the data set comprises values for the plurality of variables, and select a set of actual values of the target variable; select an initial set of coefficients; generate a set of model predicted values by applying the set of model coefficients to the data set; determine a set of performance values based on the set of model predicted values and the set of actual values, wherein the set of performance values comprises: a first standard error, and a first coefficient of determination; (1) determine a set of errors based on the set of model predicted values and the set of actual values, wherein the relative error is calculated using the formula: Relative Errork=((Predicted Valuek−Actual Valuek)/Actual Valuek)2, wherein ‘k’ is a reference number, and wherein the absolute error is calculated using the formula: Absolute Errork=(Predicted Valuek−Actual Valuek)2; (2) determine a set of error threshold values based on the set of errors and the error criteria for the complete data set; (3) generate an outlier removed data set by removing data with error values greater than or equal to the error threshold values, wherein the filtering is based on the data set and the set of error threshold values; (4) generate a set of new coefficients based on the outlier removed dataset and the set of previous coefficients; (5) generate a set of outlier bias reduced model values based on the outlier removed data set and the set of coefficients and minimizing an error between the set of predicted values and the set of actual values using at least one of: a linear optimization model, and a nonlinear optimization model; (5) determine a set of updated performance values based on the set of outlier bias reduced model predicted values and the set of actual values, wherein the set of updated performance values comprises: a second standard error, and a second coefficient of determination; repeat steps (1)-(5), while substituting the set of new coefficients for the set of coefficients from the previous iteration, unless: a performance termination criteria is satisfied, wherein the performance termination criteria comprises: a standard error termination value, and a coefficient of determination termination value, and wherein satisfying the performance termination criteria comprises the standard error termination value is greater than the difference between the first and second standard error, and the coefficient of determination termination value is greater than the difference between the first and second coefficient of determination; and storing the set of outlier bias reduction factors in a computer data medium.

Yet another embodiment includes a system for assessing the viability of a data set as used in developing a model comprising: a server, comprising: a processor, and a storage subsystem; a database stored by the storage subsystem comprising: a target data set comprising a plurality of model predicted values; a computer program stored by the storage subsystem comprising instructions that, when executed, cause the processor to: generate a random target data set; select a set of bias criteria values; generate outlier bias reduced data sets based on the target data set and each of the selected bias criteria values; generate an outlier bias reduced random target data set based on the random target data set and each of the selected bias criteria values; calculate a set of error values for the outlier bias reduced target data set and the outlier bias reduced random target data set; calculate a set of correlation coefficients for the outlier bias reduced target data set and the outlier bias reduced random target data set; generate bias criteria curves for the target data set and the random target data set based on the corresponding error value and correlation coefficient for each selected bias criteria; and compare the bias criteria curve for the target data set to the bias criteria curve for the random target data set. The processor generates the outlier bias reduced target data set and the outlier bias reduced random target data set using the Dynamic Outlier Bias Removal methodology. The random target data set can comprise of randomized data values developed from values within the range of the plurality of data values. Also, the set of error values can comprise a set of standard errors, and the set of correlation coefficients comprises a set of coefficient of determination values. In another embodiment, the program further comprises instructions that, when executed, cause the processor to generate automated advice based on comparing the bias criteria curve for the target data set to the bias criteria curve for the random target data set. Advice can be generated based on parameters selected by analysts, such as a correlation coefficient threshold and/or an error threshold. In yet another embodiment, the system's database further comprises an actual data set comprising a plurality of actual data values corresponding to the model predicted values, and the program further comprises instructions that, when executed, cause the processor to: generate a random actual data set based on the actual data set; generate an outlier bias reduced actual data set based on the actual data set and each of the selected bias criteria values; generate an outlier bias reduced random actual data set based on the random actual data set and each of the selected bias criteria values; generate, for each selected bias criteria, a random data plot based on the outlier bias reduced random target data set and the outlier bias reduced random actual data; generate, for each selected bias criteria, a realistic data plot based on the outlier bias reduced target data set and the outlier bias reduced actual target data set; and compare the random data plot with the realistic data plot corresponding to each of the selected bias criteria.

