Semi supervised EM Weak Supervision in Anomaly Detection CS229 Project Report
Anomaly Detection from an unlabeled high dimensional dataset is a challenge in an unsupervised setup. Building models with a k dimensional dataset(out of n dimensional original feature set) under the assumption that the low kdimensional subspace captures the variability in the data might affect the accuracy of model learning in the later phase. We analyzed NSL KDD20% dataset[1] an improved version of UCI archive KDDCUP'99 network anomaly data using unsupervised and semi supervised Expectation Maximization(EM) algorithms and observed how the choice of hyperparameters influences accuracy of the algorithm. In real world, intrusion detection and identification of security vulnerabilities within a network is an unsupervised problem. Since the ground truth(normal or attack) is available in this known dataset, the accuracy of different learning methods is compared against the ground truth. Apart from EM algorithm, we demonstrated weak supervision technique to label data with a few labelling functions by using snorkel[2] Our results suggest that by using only a few labeled data Semi Supervised EM(SSEM) uncovers distribution correctly, the log likelihood converges much faster and the mixture output maintains a stable data assignments. The results from Weak Supervision strongly suggest that a completely unlabeled dataset can be labeled with high accuracy using a sufficient number of labelling functions with accurate domain knowledge.