<jupyter_start><jupyter_text>Let's look at OOB estimate closely.<jupyter_code># OOB error matrix
err <- credit_model$err.rate

head(err)

# Look at final OOB error rate (last row in err matrix)
(oob_err <- err[nrow(err), "OOB"])
# Let's plot the model trained above
plot(credit_model)

# Adding a legend is a must :D
legend(x = "right", 
       legend = colnames(err),
       fill = 1:ncol(err))<jupyter_output><empty_output><jupyter_text>OOB is not discussed here because it was discussed in the notebook on bagged trees. Let's evaluate model performance on a test set<jupyter_code># Let's test the model
class_prediction <- predict(object = credit_model,
                            newdata = credit_test,
                            type = "class") # return classification labels
                            
# The confusion matrix for the test set
cm <- confusionMatrix(data = class_prediction,       
                      reference = credit_test$default)
print(cm)
# Let's compare test set accuracy to OOB accuracy
paste0("Test Accuracy: ", cm$overall[1])
paste0("OOB Accuracy: ", 1 - oob_err)<jupyter_output><empty_output><jupyter_text>Test accuracy and OOB accuracy are close enough. OOB gives a generic idea about the performance of the model.<jupyter_code># Let's check the model accuracy with another metrics: classification error
(ce(actual = credit_test$default, 
   predicted = class_prediction))
# Another famous metric is AUC/ROC. Let's calculate AUC of the predicted model.

# For that to happen we need to generate predictions on the test set
pred <- predict(object = credit_model,
            newdata = credit_test,
            type = "prob")

                
# And now compute the AUC
auc(actual = ifelse(credit_test$default == "yes", 1, 0), 
    predicted = pred[,"yes"])<jupyter_output><empty_output><jupyter_text>Like any machine learning algo, to achieve the best model tune the hyperparameters of the model.Let's do that. Some of the key hyperparameters we are going to play with are:1. ntrees - number of trees. Default is 500.2. mtry - number of variables choosen as candidates to train at each split. Default is sqrt(total input features)3. sampsize - number of samples to train on. Default to 63.2%There are two other important parameters but we won't cover them here.1. nodesize - minimum size of the terminal nodes, helps in controlling the complexity.Lesser the nodesize -> more complex the tree.2. maxnodes - maximum number of terminal nodes, helps in controlling the complexity and inavoiding overfitting.  The most important of the hyperparameters is mtry. Let's fix that first.<jupyter_code># Let's start the tuning process    
res <- tuneRF(x = subset(credit_train, select = -default),
              y = credit_train$default,
              ntreeTry = 500)
               
# Look at results
print(res)
# Find the mtry value that minimizes OOB Error
mtry_opt <- res[,"mtry"][which.min(res[,"OOBError"])]
print(mtry_opt)<jupyter_output>4.OOB 
    4<jupyter_text>Therefore the best mtry value is 1. Previously, we trained mtry using the tune function provided by randomForest. However,let's tune it using a grid of all possible combinations.<jupyter_code># Setting seed for reproducible train and test partitions
set.seed(123)

# List of possible values for mtry, nodesize and sampsize
mtry <- seq(4, ncol(credit_train) * 0.8, 2)
nodesize <- seq(3, 8, 2)
sampsize <- nrow(credit_train) * c(0.7, 0.8)

# Create a data frame containing all combinations 
hyper_grid <- expand.grid(mtry = mtry, nodesize = nodesize, sampsize = sampsize)

# Create an empty vector to store OOB error values and model
oob_err <- c()
model_vec <- list()

# Loop over the rows of hyper_grid to train the grid of models
for (i in 1:nrow(hyper_grid)) {

    model <- randomForest(formula = default ~ ., 
                          data = credit_train,
                          mtry = hyper_grid$mtry[i],
                          nodesize = hyper_grid$nodesize[i],
                          sampsize = hyper_grid$sampsize[i])
                          
    # Store OOB error for the model                      
    oob_err[i] <- model$err.rate[nrow(model$err.rate), "OOB"]
    model_vec[[i]] <- model
}

# Identify optimal set of hyperparmeters based on OOB error
(opt_i <- which.min(oob_err))
print(hyper_grid[opt_i,])
model_vec[[9]] #best model<jupyter_output><empty_output><jupyter_text>VOILA! Mission accomplished. Let's check out the results of our final model on thetest dataset.<jupyter_code>class_prediction <- predict(object = model_vec[[9]],
                            newdata = credit_test,
                            type = "class") # return classification labels
                            
# The confusion matrix for the test set
cm <- confusionMatrix(data = class_prediction,       
                      reference = credit_test$default)
print(cm)<jupyter_output>Confusion Matrix and Statistics

          Reference
Prediction  no yes
       no  161  47
       yes  14  28
                                          
               Accuracy : 0.756           
                 95% CI : (0.6979, 0.8079)
    No Information Rate : 0.7             
    P-Value [Acc > NIR] : 0.02945         
                                          
                  Kappa : 0.3355          
 Mcnemar's Test P-Value : 4.182e-05       
                                          
            Sensitivity : 0.9200          
            Specificity : 0.3733          
         Pos Pred Value : 0.7740          
         Neg Pred Value : 0.6667          
             Prevalence : 0.7000          
         Detection Rate : 0.6440          
   Detection Prevalence : 0.8320          
      Balanced Accuracy : 0.6467          
                                          
       'Positive' Class : no
{'fraction_non_alphanumeric': 0.08, 'fraction_numerical': 0.021551724137931036, 'mean_word_length': 2.299772468714448, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '143993', 'license': 'MIT', 'n_tokens_mistral': 1756, 'n_tokens_neox': 1592, 'n_words': 569, 'path': 'Machine_Learning/R/Random Forest in R.ipynb', 'repo_name': 'pallavbakshi/datascience', 'size': '5800'}