1. Field of the Invention
Embodiments of the present invention generally relate to machine learning techniques and computational chemistry. More particularly, embodiments of the invention relate to techniques for estimating the accuracy of a molecular properties model, and for estimating the accuracy of predictions generated by a molecular properties model.
2. Description of the Related Art
Many industries use machine learning techniques to construct software applications that provide a predictive model of relevant phenomena. For example, machine learning applications have been developed to detect fraudulent credit card transactions, predict creditworthiness, or recognize words spoken by an individual. Machine learning techniques have also been applied to create predictive models of chemical and biological systems. Generally, machine learning techniques are used to construct a software application that improves its ability to perform a task as it attribute or quantity from known information (e.g., whether a particular molecule will bind to a protein receptor, based on an evaluation of other molecules known to, or to not, bind to the protein) or to classify an object as belonging to a particular group or class. A machine learning application may improve its performance on the selected task as the number of training examples used to train the model is increased. Each training example may include an example of an object (e.g., a molecule, compound, or substituent group thereof), along with a value for the otherwise unknown classification of the object.
During “training” a selected machine learning algorithm processes thousands, if not millions or billions, of potential models (also referred to as hypotheses). By evaluating how well different possible potential models perform against the training data a trained model is selected. For example, a classification learning algorithm may be configured to process a set of training examples that includes both an object and a classification for the object. In one embodiment, the hypothesis that correctly classifies the greatest number of training examples may be selected by a machine learning algorithm as the molecular properties model. Further, various machine learning algorithms may be configured to tweak or otherwise modify the selected model by also considering minor variations to a promising hypothesis. For example, genetic algorithms may be used to “mate,” and “mutate” hypotheses identified as interesting. The final “learned model” may then be used to predict the classification for other objects supplied to the model.
A molecular properties model, however, is of limited usefulness without an estimation of how well it performs. Thus, the accuracy of the model must be estimated. Often, the accuracy of a molecular properties model is calculated using statistical techniques; thus, the accuracy estimate is a random variable, and does not reflect a direct measurement of the actual accuracy for a specific molecular properties model. Thus, simply estimating that a model is 80% accurate is useful only if one has a minimal confidence in the accuracy of the estimate. It is not, however, currently the practice to expend effort estimating or bounding the statistical confidence or higher moments of estimates of model accuracy generated using statistical techniques. In practice, this has led to many molecular properties models with a very high estimated accuracy that, in fact, perform very poorly (i.e., the predictions or classifications prove to be erroneous). Accordingly, there is a need for improved techniques for generating molecular properties models and for estimating and bounding the accuracy and performance of these models or the predictions made using these models.