<jupyter_start><jupyter_text>This notebook covers a R-based approach to Gradient Boosting Machine. We'll start with implementing a simple GBM  model and then evalaute the model using AUC. Then we will tune our model's hyper-parameters for better performance. Another important feature of GBM is early stopping. We shall look into that as well.<jupyter_code># reading data file ~ credit.csv 
# taken from https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29

# credit.csv is composed of qualitative and quantitative variables.
# For this exercise we will focus on all of the features

creditsub <- read.csv(file = 'data/credit.csv', 
                      header = T)

# Let's take a look at the dataframe
head(creditsub)
# Let's split the data into train and test

# Setting seed for reproducible train and test partitions
set.seed(123)

smp_size <- floor(0.75 * nrow(creditsub))

train_ind <- sample(seq_len(nrow(creditsub)), size = smp_size)

credit_train <- creditsub[train_ind, ]

credit_test <- creditsub[-train_ind, ]
# Convert "yes" to 1, "no" to 0
credit_train$default <- ifelse(credit_train$default == "yes", 1, 0)

# Let's tart with a large 10000-tree GBM model, we will early stop it later
set.seed(123)
credit_model <- gbm(formula = default ~ ., 
                    distribution = 'bernoulli', 
                    data = credit_train,
                    n.trees = 10000)
                    
# Print the model object                    
print(credit_model)<jupyter_output>gbm(formula = default ~ ., distribution = "bernoulli", data = credit_train, 
    n.trees = 10000)
A gradient boosted model with bernoulli loss function.
10000 iterations were performed.
There were 16 predictors of which 16 had non-zero influence.<jupyter_text>It seems that all the predictors are contributing to the final outupt, which means there weren't any noise varaibles.<jupyter_code># Let's look at a more appealing description of the model
summary(credit_model)<jupyter_output><empty_output><jupyter_text>Checking balance seems to be the major predictor/influencer. Let's test our model on test set<jupyter_code>credit_test$default <- ifelse(credit_test$default == "yes", 1, 0)

# Generating predictions on the test set
preds1 <- predict(object = credit_model, 
                  newdata = credit_test,
                  n.trees = 10000)

# Generating predictions on the test set. But this time let's scale the response.
# Scaling gives us a repsonse between 0 and 1
preds2 <- predict(object = credit_model, 
                  newdata = credit_test,
                  n.trees = 10000,
                  type = "response")
data.frame(head(preds1), head(preds2), head(credit_test$default))
# Compare the range of the two sets of predictions
range(preds1)
range(preds2)
# Let's check out the AUC of the test set
auc(actual = credit_test$default, predicted = preds2)<jupyter_output><empty_output><jupyter_text>Note: AUC is same for preds1 and preds2 since it is based on ranks not scale. However, something like RMSE will be affected by rescaling. Here the AUC came out to be high which is a good sign about the model's accuracy.<jupyter_code># Let's further optimize the model by tuning the hyperparameteres.
# Here we will be focusing on early stopping in order to optimize
# the number of trees in our model. 

# Since we already have a model trained,let's just tune that
ntree_opt_oob <- gbm.perf(object = credit_model, 
                          method = 'OOB', 
                          oobag.curve = TRUE)

# For optimizing the ntrees using Cross-validation method we need to recreate our model
# with cv.folds.
credit_model_cv <- gbm(formula = default ~ ., 
                       distribution = "bernoulli", 
                       data = credit_train,
                       n.trees = 10000,
                       cv.folds = 2 # added for optimization
                      )

# Optimal ntree estimate based on CV
ntree_opt_cv <- gbm.perf(object = credit_model_cv,
                        method = 'cv')
# Compare the estimates                         
print(paste0("Optimal n.trees (OOB Estimate): ", ntree_opt_oob))                         
print(paste0("Optimal n.trees (CV Estimate): ", ntree_opt_cv))<jupyter_output>[1] "Optimal n.trees (OOB Estimate): 2967"
[1] "Optimal n.trees (CV Estimate): 7953"<jupyter_text>The number of trees vary significantly. Let's test both of the ntree opt parameters to find out the suitable number of trees for this model.<jupyter_code># Predictions on the test set using ntree_opt_oob number of trees
preds1 <- predict(object = credit_model, 
                  newdata = credit_test,
                  n.trees = ntree_opt_oob)
                  
# Predictions on the test set using ntree_opt_cv number of trees
preds2 <- predict(object = credit_model_cv, 
                  newdata = credit_test,
                  n.trees = ntree_opt_cv)   

# Let's generate the test set AUCs using the two sets of preditions & compare
auc1 <- auc(actual = credit_test$default, predicted = preds1)  #OOB
auc2 <- auc(actual = credit_test$default, predicted = preds2)  #CV 

# Comparing AUC 
print(paste0("Test set AUC (OOB): ", auc1))                         
print(paste0("Test set AUC (CV): ", auc2))<jupyter_output>[1] "Test set AUC (OOB): 0.791390476190476"
[1] "Test set AUC (CV): 0.806628571428571"
{'fraction_non_alphanumeric': 0.082803738317757, 'fraction_numerical': 0.02355140186915888, 'mean_word_length': 2.9490774907749078, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '52483', 'license': 'MIT', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1527, 'n_words': 561, 'path': 'Machine_Learning/R/Boosting in R.ipynb', 'repo_name': 'pallavbakshi/datascience', 'size': '5350'}