<jupyter_start><jupyter_text>Random Forests- 'Random' - each tree gets some unique training data randomly - so helps against overfitting !- 'Forest' because it is a collection - an ensemble - of decision trees!- It's like cross-validation meets decision tree.**Aside: Awesome data visualisation website** http://www.r2d3.us/visual-intro-to-machine-learning-part-1/| concept | description ||:-------:|:-----------:|| decision tree | a classification model constructing questions|| root node | the first question a tree asks || leaf node | the result of the last question || depth | number of consecutive questions || CART (Classification & Regression Tree)| name of the tree construction algorithm || GINI | a metric for the purity of nodes || ensemble | a model consisting of many sub-models || Random Forest | an ensemble of randomized decision trees || Gradient Boosting | a deluxe tree-based ensemble model |![image.png](attachment:image.png) Let's try out a Random Forest Classifier in Scikit-LearnTrain and fine-tune a Random Forest on the Titanic dataset.- experiment with the hyperparameters `max_depth` and `n_estimators`- compare your model to the Decision Tree model  **1. Read Data in Pandas**<jupyter_code>import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
df = pd.read_csv('train.csv', index_col=0)
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 891 entries, 1 to 891
Data columns (total 11 columns):
Survived    891 non-null int64
Pclass      891 non-null int64
Name        891 non-null object
Sex         891 non-null object
Age         714 non-null float64
SibSp       891 non-null int64
Parch       891 non-null int64
Ticket      891 non-null object
Fare        891 non-null float64
Cabin       204 non-null object
Embarked    889 non-null object
dtypes: float64(2), int64(4), object(5)
memory usage: 83.5+ KB<jupyter_text>**2. Split data - Select columns - into X and y** **Note:** Now we're gonna purposefuly not choose 'Age' and 'cabin' so that we don't have missing values<jupyter_code>X = df[['Pclass', 'Sex', 'Fare']]
y = df['Survived']
len(X) == len(y)<jupyter_output><empty_output><jupyter_text>**3. Split X data into training and testing sets**<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, 
                                                    random_state=42)
X_train.shape, X_test.shape, y_train.shape, y_test.shape<jupyter_output><empty_output><jupyter_text>**4. Feature Engineering**<jupyter_code># pd.get_dummies(X_train['Sex']).head()
# Four options for encoding strings to numbers:
# 1. LabelEncoder
# 2. pd.factorize()
# 3. pd.get_dummies()
# 4. .map() - Probably the best option here!
    
# d = {'female': 0, 'male': 1}

# X_train = X_train.copy() # gets rid of warning!!
# X_train['Sex'] = X_train['Sex'].map(d)
# X_train.head()<jupyter_output><empty_output><jupyter_text>We now have X_train data that has different columns compared to the original dataframe, therefore we need to be able to apply these changes to the X_test dataframe too - We need to write a function!<jupyter_code>def feature_engineer(dataframe):
    
    d = {'female': 0, 'male': 1}

    dataframe = dataframe.copy() # just to supress pandas warning!!
    dataframe['Sex'] = dataframe['Sex'].map(d)
    
    return dataframe
X_train = feature_engineer(X_train)
X_train.head()<jupyter_output><empty_output><jupyter_text>**5. "Build" a Random Forest model**`m = RandomForestClassifier()`<jupyter_code>m = RandomForestClassifier(n_estimators=3, max_depth=2, criterion='gini' )<jupyter_output><empty_output><jupyter_text>In the brackets, we can now introduce the hyperparameters:- number of trees i.e. n_estimators- number of levels/depth - how many questions deep you want it to go- criterion - RandomForest is using the gini index to split the data as well as possible The GINI indexThe GINI index is used to evaluate which partition results in the best separation of the classes K:\begin{align}G = \sum_{k=1}^{K}{(p_k (1 - p_k))}\end{align}Where p indicates the proportion of points belonging to class K.**Which is the best possible GINI score, which the worst?**  **6. Train the model**<jupyter_code>m.fit(X_train, y_train)
m.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Now we check the test data (remember we wrote a function earlier!)<jupyter_code>X_test = feature_engineer(X_test)
m.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Now we want to optimise both our train and test scores: change the hyperparameter values (will see how to optimise these tomorrow)<jupyter_code>m = RandomForestClassifier(n_estimators=50000, max_depth=60000, 
                           criterion='gini')
m.fit(X_train, y_train)
m.score(X_train, y_train)
m.score(X_test, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1080582325199918, 'fraction_numerical': 0.02050440844781628, 'mean_word_length': 4.639306358381503, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '169897', 'license': 'MIT', 'n_tokens_mistral': 1636, 'n_tokens_neox': 1524, 'n_words': 511, 'path': '07_RandomForests.ipynb', 'repo_name': 'maximcondon/Project_Titanic', 'size': '4877'}