SYSTEM AND METHOD FOR EVALUATING AND DEPLOYING UNSUPERVISED OR SEMI-SUPERVISED MACHINE LEARNING MODELS
FIG1 is a schematic diagram illustrating an exemplary networked system embodying the invention
FIG1 is a block diagram illustrating an exemplary networked system 100 including an evaluation system 102, which is configured to implement a method of evaluating and deploying one or more unsupervised or semi-supervised machine learning models, in accordance with an embodiment of the invention. The evaluation system 102 may comprise a computer system having a conventional architecture. In particular, the evaluation system 102, as illustrated, comprises a processor 104. The processor 104 is operably associated with a non-volatile memory/storage device 106, e.g. via one or more data/address busses 108 as shown. The non-volatile storage 106 may be a hard disk drive, and/or may include a solid-state non-volatile memory, such as ROM, flash memory, solid-state drive (SSD), or the like. The processor 104 is also interfaced to volatile storage 110, such as RAM, which contains program instructions and transient data relating to the operation of the evaluation system 102. Continuing the discussion of FIG1, the networked system 100 also includes a monitored system 118. The monitored system 118 may be, for example, a database system, a transaction processing system, an information system, a web service provider system, an industrial system, a security system, a robotics system, or any other system for which it is desirable to perform monitoring to detect unusual, novel, outlying, or anomalous behaviours. The monitored system 118 may take any one of a number of forms, included centralised and distributed architectures. While a variety of terms are used for the type of monitoring to which embodiments of the present invention are directed (e.g. outlier detection, change detection, or novelty detection), in the present specification the term 'anomaly detection' is consistently employed, with the understanding that other terminology is commonplace in the art.
schematic diagram
122
processor
128
device
124
data/address busses
116
Internet
130
communications interface
100
networked system
1
about AP is thus
126
volatile storage
132
executable image
104
processor
112
communications interface
102
evaluation system
108
data/address busses
106
device
110
volatile storage
114
instructions
120
monitoring system
['1. A method of evaluating and deploying machine learning models for anomaly detection of a monitored system (118), comprising: providing a plurality of candidate machine learning algorithms (204) configured for anomaly detection of the monitored system; providing training data (206) including a pool of negative data samples (302) representing normal activity of the monitored system and one or more smaller pools of positive data samples (304), wherein each pool of positive data samples represents one type of anomalous activity of the monitored system; for each type of anomalous activity, generating (412) a benchmarking dataset (306) which comprises a first number of samples drawn from the pool of negative samples, and a second number of samples drawn from the pool of positive samples corresponding with the type of anomalous activity, wherein the first number of samples is substantially larger than the second number of samples; for each combination of a candidate machine learning algorithm selected from the plurality of candidate machine learning algorithms with a type of anomalous activity: drawing (414) a plurality of training and cross-validation sets (314, 316) from the benchmarking dataset corresponding with the type of anomalous activity, wherein each training set comprises only negative data samples, and each cross-validation set comprises a mix of negative and positive data samples; for each one of the plurality of training and cross-validation sets, training (416) a machine-learning model based on the candidate algorithm using the training set, and validating (418) the machine-learning model using the cross-validation set with average precision as a performance metric; computing a mean average precision value for the candidate machine learning algorithm across the average precision performance  metrics obtained using the plurality of training and cross-validation sets, for each candidate machine learning algorithm, computing (608) a ranking value based upon at least the mean average precision values computed for the candidate machine learning algorithm for each type of anomalous activity; selecting (610) a machine learning algorithm from the candidate machine learning algorithms based upon the computed ranking values; and deploying (612) a machine learning model based on the selected algorithm to a monitoring system (120), whereby the monitoring system executes the deployed machine learning model to detect anomalies of the monitored system.', '7. A computing system for evaluating machine learning models for anomaly detection of a monitored system (118), comprising: a processor (104); at least one memory device (110) accessible by the processor; and at least one data store (106) accessible by the processor and containing a plurality of candidate machine learning algorithms (204) configured for anomaly detection of the monitored system and training data (206) including a pool of negative data samples (302) representing normal activity of the monitored system and one or more smaller pools of positive data samples (304), wherein each pool of positive data samples represents one type of anomalous activity of the monitored system, wherein the memory device contains a body of program instructions (114) which, when executed by the processor, cause the computing system to implement a method comprising steps of: for each type of anomalous activity, generating (412) a benchmarking dataset (306) which comprises a first number of samples drawn from the pool of negative samples, and a second number of samples drawn from the pool of positive samples corresponding with the type of anomalous activity, wherein the first number of samples is substantially larger than the second number of samples; for each combination of a candidate machine learning algorithm selected from the plurality of candidate machine learning algorithms with a type of anomalous activity: drawing (414) a plurality of training and cross-validation sets (314, 316) from the benchmarking dataset corresponding with the type of anomalous activity, wherein each training set comprises only negative data samples, and each cross-validation set comprises a mix of negative and positive data samples; for each one of the plurality of training and cross-validation sets, training (416) a machine-learning model based on the  candidate algorithm using the training set, and validating (418) the machine-learning model using the cross-validation set with average precision as a performance metric; computing a mean average precision value for the candidate machine learning algorithm across the average precision performance metrics obtained using the plurality of training and cross-validation sets, and for each candidate machine learning algorithm, computing (608) a ranking value based upon at least the mean average precision values computed for the candidate machine learning algorithm for each type of anomalous activity.']