June 5 , 2013 by yhat Random forest is a highly versatile machine learning method with numerous applications ranging from marketing to healthcare and insurance . It can be used to model the impact of marketing on customer acquisition , retention , and churn or to predict disease risk and susceptibility in patients . Random forest is a capable of regression and classification . It can handle a large number of features , and it ' s helpful for estimating which or your variables are important in the underlying data being modeled . This is a post about random forests using Python . What is a Random Forest ? Random forest is solid choice for nearly any prediction problem ( even non - linear ones ) . It ' s a relatively new machine learning strategy ( it came out of Bell Labs in the 90 s ) and it can be used for just about anything . It belongs to a larger class of machine learning algorithms called ensemble methods . Ensemble LearningEnsemble learning involves the combination of several models to solve a single prediction problem . It works by generating multiple classifiers / models which learn and make predictions independently . Those predictions are then combined into a single ( mega ) prediction that should be as good or better than the prediction made by any one classifer . Random forest is a brand of ensemble learning , as it relies on an ensemble of decision trees . More on ensemble learning in Python here : Scikit - Learn docs . Randomized Decision TreesSo we know that random forest is an aggregation of other models , but what types of models is it aggregating ? As you might have guessed from its name , random forest aggregates Clasification ( or Regression ) Trees . A decision tree is composed of a series of decisions that can be used to classify an observation in a dataset . Random ForestThe algorithm to induce a random forest will create a bunch of random decision trees automatically . Since the trees are generated at random , most won ' t be all that meaningful to learning your classification / regression problem ( maybe 99 . 9 % of trees ) . If an observation has a length of 45 , blue eyes , and 2 legs , it ' s going to be classified as red . Arboreal VotingSo what good are 10000 ( probably ) bad models ? Well it turns out that they really aren ' t that helpful . But what is helpful are the few really good decision trees that you also generated along with the bad ones . When you make a prediction , the new observation gets pushed down each decision tree and assigned a predicted value / label . once each of the trees in the forest have reported its predicted value / label , the predictions are tallied up and the mode vote of all trees is returned as the final prediction . Simply , the 99 . 9 % of trees that are irrelevant make predictions that are all over the map and cancel each another out . The predictions of the minority of trees that are good top that noise and yield a good prediction . Why you should I use it ? It ' s EasyRandom forest is the Leatherman of learning methods . You can throw pretty much anything at it and it ' ll do a serviceble job . It does a particularly good job of estimating inferred transformations , and , as a result , doesn ' t require much tuning like SVM ( i . e . it ' s good for folks with tight deadlines ) . An Example TransformationRandom forest is capable of learning without carefully crafted data transformations . Take the the f ( x ) = log ( x ) function for example . Create some fake data and add a little noise . import numpy as npx = np . random . uniform ( 1 , 100 , 1000 ) y = np . log ( x ) + np . random . normal ( 0 , . 3 , 1000 ) full gist here If we try and build a basic linear model to predict y using x we wind up with a straight line that sort of bisects the log ( x ) function . Whereas if we use a random forest , it does a much better job of approximating the log ( x ) curve and we get something that looks much more like the true function . You could argue that the random forest overfits the log ( x ) function a little bit . Either way , I think this does a nice job of illustrating how the random forest isn ' t bound by linear constraints . UsesVariable SelectionOne of the best use cases for random forest is feature selection . one of the byproducs of trying lots of decision tree variations is that you can examine which variables are working best / worst in each tree . When a certain tree uses a one variable and another doesn ' t , you can compare the value lost or gained from the inclusion / exclusion of that variable . The good random forest implementations are going to do that for you , so all you need to do is know which method or variable to look at . In the following examples , we ' re trying to figure out which variables are most important for classifying a wine as being red or white . ClassificationRandom forest is also great for classification . It can be used to make predictions for categories with multiple possible values and it can be calibrated to output probabilities as well . one thing you do need to watch out for is overfitting . Random forest can be prone to overfitting , especially when working with relatively small datasets . You should be suspicious if your model is making " too good " of predictions on our test set . One way to overfitting is to only use really relevant features in your model . While this isn ' t always cut and dry , using a feature selection technique ( like the one mentioned previously ) can make it a lot easier . --> RegressionYep . It does regression too . I ' ve found that random forest -- unlike other algorithms -- does really well learning on categorical variables or a mixture of categorical and real variables . Categorical variables with high cardinality ( # of possible values ) can be tricky , so having something like this in your back pocket can come in quite useful . A Short Python ExampleScikit - Learn is a great way to get started with random forest . The scikit - learn API is extremely consistent across algorithms , so you horse race and switch between models very easily . A lot of times I start with something simple and then move to random forest . One of the best features of the random forest implementation in scikit - learn is the n _ jobs parameter . This will automatically paralellize fitting your random forest based on the number of cores you want to use . Here ' s a great presentation by scikit - learn contributor Olivier Grisel where he talks about training a random forest on a 20 node EC 2 cluster . 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 from sklearn . datasets import load _ irisfrom sklearn . ensemble import R a n domForestClassifierimport pandas as pdimport numpy as np iris = load _ iris ( ) df = pd . DataFrame ( iris . data , columns = iris . feature _ names ) df [ ' is _ train '] = np . random . uniform ( 0 , 1 , len ( df ) ) <= . 75 df [ ' species '] = pd . Factor ( iris . target , iris . target _ names ) df . head ( ) train , test = df [ df [ ' is _ train ']== True ] , df [ df [ ' is _ train ']== False ] features = df . columns [ : 4 ] clf = RandomForestClassifier ( n _ jobs = 2 ) y , _ = pd . factorize ( train [ ' species ']) clf . fit ( train [ features ] , y ) preds = iris . target _ names [ clf . predict ( test [ features ] ) ] pd . crosstab ( test [ ' species '], preds , rownames =[' actual '], colnames =[' preds ']) view raw rf _ iris . py This Gist brought to you by GitHub . Looks pretty good ! Final ThoughtsRandom forests are remarkabley easy to use given how advanced they are . As with any modeling , be wary of overfitting . If you ' re interested in getting started with random forest in R , check out the randomForest package . Berkley ResourcesKaggle blogpostAndy Mueller ' s blog ( scikit - learn contributor ) Random Forest GuideOlivier Grisel ' s website