My company provides managed services to a lot of its clients. Our customers typically uses following monitoring tools to monitor their servers/webapps:OpsViewNagiosPingdomCustom shell scriptsWhenever any issue is found, an alert mail comes to our Ops team so that they act upon rectifying the issue.As we manage thousands of servers, our Ops teams' inbox is flooded with email alerts all the time. Even a single issue which has a cascading effect, can trigger 20-30 emails.Now, what I want to do is to implement a system which will be able to extract important features out of an alert email - like server IP address, type of problem, severity of problem etc. and also classify the emails into proper category, like CPU-Load-Customer1-Server2, MySQL-Replication-Customer2-DBServer3 etc. We will then have a pre-defined set of debugging steps for each category, in order to help the Ops team to rectify the problem faster. Also, the feature extractor will provide input data to the team for a problem.So far I have been able to train NaiveBayesClassifier with supervised learning techniques i.e. labeled training data(cluster data), and able to classify new unseen emails into its proper cluster/category. As the emails are based on certain templates, the accuracy of the classifier is very high. But we also get alert emails from custom scripts, which may not follow the templates. So, instead of doing supervised learning, I want to try out unsupervised learning for the same. I am looking into KMeans clustering. But again the problem is, we won't know the number of clusters beforehand. So, which algorithm will be best for this use case? Right now I am using Python's TextBlob library for classification.Also, for feature extraction out of an alert email, I am looking into NLTK (http://www.nltk.org/book/ch07.html) library. I tried it out, but it seems to work on proper English paragraphs/texts well, however, for alert emails, it extracted a lot of unnecessary features. Is there already any existing solution for the same? If not, what will be the best way to implement the same? Which library, which algorithm?PS: I am not a Data Scientist.Sample emails: PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.10.0.100  Host Group Hierarchy: Opsview > Customer1  - BIG C > Customer1_PROD  State: CRITICAL  Date & Time: Sat Oct 4 07:02:06 UTC 2014    Additional Information:     CRITICAL - load average: 41.46, 40.69, 37.91RECOVERY: OK - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.1.1.100  Host Group Hierarchy: Opsview > Customer1  - BIG C > Customer1_PROD  State: OK  Date & Time: Sat Oct 4 07:52:05 UTC 2014    Additional Information:     OK - load average: 0.36, 0.23, 4.83PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.100.10.10  Host Group Hierarchy: Opsview > Customer1  - BIG C > Customer1_PROD  State: CRITICAL  Date & Time: Sat Oct 4 09:29:05 UTC 2014    Additional Information:     CRITICAL - load average: 29.59, 26.50, 18.49Classifier code:(format of csv - email, <disk/cpu/memory/mysql>) from textblob import TextBlobfrom textblob.classifiers import NaiveBayesClassifierimport csvtrain = []with open('cpu.txt', 'r') as csvfile:    reader = csv.reader(csvfile, delimiter=',', quotechar='')    for row in reader:        tup = unicode(row[0], ISO-8859-1), row[1]        train.append(tup)// this can be done in a loop, but for the time being let it bewith open('memory.txt', 'r') as csvfile:    reader = csv.reader(csvfile, delimiter=',', quotechar='')    for row in reader:        tup = unicode(row[0], ISO-8859-1), row[1]        train.append(tup)with open('disk.txt', 'r') as csvfile:    reader = csv.reader(csvfile, delimiter=',', quotechar='')    for row in reader:        tup = unicode(row[0], ISO-8859-1), row[1]        train.append(tup)with open('mysql.txt', 'r') as csvfile:    reader = csv.reader(csvfile, delimiter=',', quotechar='')    for row in reader:        tup = unicode(row[0], ISO-8859-1), row[1]        train.append(tup)cl = NaiveBayesClassifier(train)cl.classify(email)Feature extractor code taken from: https://gist.github.com/shlomibabluki/5539628Please let me know if any more information is required here.Thanks in advance.
How to extract features and classify alert emails coming from monitoring tools into proper category?
machine learning;classification;clustering;feature extraction