<jupyter_start><jupyter_text>Section B. Prediction Uncertainty with Bootstrapping  Question 3 [Bootstrapping, 25 Marks]  1. Bootstrapping for KNN regression   Load libraries<jupyter_code># libraries
library(ggplot2)<jupyter_output><empty_output><jupyter_text>Knn function & Error function<jupyter_code># KNN function (distance = manhattan)
knn <- function(train.data, train.label, test.data, K=K, distance = 'manhattan'){
    ## count number of train samples
    train.len <- nrow(train.data)
    
    ## count number of test samples
    test.len <- nrow(test.data)

    ## calculate distances between samples
    dist <- as.matrix(dist(rbind(test.data, train.data), method= distance))[1:test.len, (test.len+1):(test.len+train.len)]
    
    ## for each test sample...
    for (i in 1:test.len){
        ### ...find its K nearest neighbours from training sampels...
        nn <- as.data.frame(sort(dist[i,], index.return = TRUE))[1:K,2]

        ###... and calculate the predicted values according to average.
        test.label[i]<- sum(train.label[nn])/K
    }
    
    ## return the predictions as output
    return (test.label)
}

# error function: root mean square error
RMSE <- function(predicted, target) {
    se <- 0
    for (i in 1:length(predicted)) {
        se <- se + (predicted[i]-target[i])^2
    }
    return (sqrt(se/length(predicted)))
}<jupyter_output><empty_output><jupyter_text>Boot function<jupyter_code># define a function that generates sample indixes based on bootstrap technique
boot <- function (original.size, sample.size = N, times = L){
    indx <- matrix(nrow=times, ncol=sample.size)
    for (t in 1:times){
        indx[t, ] <- sample(x=original.size, size=sample.size, replace = TRUE)
    }
    return(indx)
}<jupyter_output><empty_output><jupyter_text>2. Apply bootstrapping with times = 30, size = 60 and change K=1:20  Import data<jupyter_code># read train and test data
train_1b <- read.csv("Task1B_train.csv")
test_1b <- read.csv("Task1B_test.csv")
# create train and test subsets
train.data <- train_1b[,-5]
train.label <- train_1b[,5]
test.data <- test_1b[,-5]
test.label <- test_1b[,5]
# initialize the parameters
K <- 20           # Maximum K for KNN 
L <- 30          # number of datasets
N <- 60          # size of datasets

# generate bootstrap indices:
boot.indx <- boot(nrow(train.data), N, L)

# a dataframe to track the RMSE in each case
RMSE_boot <- data.frame('K'=1:K, 'L'=1:L, 'test'=rep(0,L*K))

## apply bootstrapping for every k values:
for (k in 1: K){
    
    ### for every dataset sizes:
    for (l in 1:L){
        
        #### calculate iteration index i
        i <- (k-1)*L+l
        
        #### save sample indices that were selected by bootstrap
        indx <- boot.indx[l,]
        
        #### save the value of k and l
        RMSE_boot[i,'K'] <- k
        RMSE_boot[i,'L'] <- l
        
        #### calculate and record the test RMSE
        RMSE_boot[i,'test'] <-  RMSE(knn(train.data[indx, ], train.label[indx], test.data, K=k), test.label)
    } 
}
# modified the code from acitivity 1.2 and draw boxplot of RMSE of bootstrapped knn
ggplot(data=RMSE_boot, aes(factor(K),test)) + geom_boxplot(outlier.shape = NA)  + 
    scale_color_discrete(guide = guide_legend(title = NULL)) + 
    ggtitle('RMSE vs. K (Box Plot)') + theme_minimal()
# ignore the warnings (because of ignoring outliers)
options(warn=-1)<jupyter_output><empty_output><jupyter_text>3. Discuss how test error and its uncertainty behave as K increases The boxplot shows:- test error increase as k increases- uncertainty increase as k increases- test error is minimum at k = 2The boxplot indicates a positive correlation between the k value and test error. Generally, we expect the plot to form a parabola shape with high test errors on both low k values and high k values. Normally, a small k value leads to overfitting and increase the test error. In this case, as the dataset is not large enough, the test error is relatively small when k value is small. In terms of uncertainty, the size boxplot increases when k increases, indicates the variance of error increases and thus leads to a larger uncertainty. There is one thing to notice that at k = 1, the uncertainty and test error is higher than k = 2 & 3. This matches with our expectation because at k = 1, the model is highly overfitted, which should have a high test error and high variance.If a optimal k value need to be selected based on the plot, k = 2 should be the optimal k value where the test error is the smallest and the uncertainty is small too.  4. Bootstrapping for KNN regression with K=5,times = 50 and change sizes = 5, 10, 15,..., 75<jupyter_code># initialize the parameters
K <- 5           # Maximum K for KNN 
L <- 50          # number of datasets
N <- c(5,10,15,20,25,30,35,40,45,50,55,60,65,70,75)  # size of datasets

# a dataframe to track the RMSE in each size
RMSE_size <- data.frame('size'= rep(0, L*K*15), 'error'=rep(0,L*K*15))

for (n in N){
    # generate bootstrap indices:
    boot.indx <- boot(nrow(train.data), n, L)

    # a dataframe to track the RMSE in each case
    RMSE_case <- data.frame('size'= rep(0, L*K), 'error'=rep(0,L*K))
    
    ## apply bootstrapping for every k values:

    for (k in 1: K){

        ### for every dataset sizes:
        for (l in 1:L){

            #### calculate iteration index i
            i <- (k-1)*L+l

            #### save sample indices that were selected by bootstrap
            indx <- boot.indx[l,]

            #### save the value of k and l
            RMSE_case[i,'size'] = n 

            #### calculate and record the test RMSE
            RMSE_case[i,'error'] <-  RMSE(knn(train.data[indx, ], train.label[indx], test.data, K=k), test.label)
        }
        # update to RMSE VS SIZE table
        RMSE_size <- rbind(RMSE_size, RMSE_case)
    }
}
# modified the code from acitivity 1.2 and draw boxplot of RMSE of bootstrapped knn
ggplot(data=RMSE_size, aes(factor(size),error)) + geom_boxplot(outlier.shape = NA)  + 
    scale_color_discrete(guide = guide_legend(title = NULL)) + 
    ggtitle('RMSE vs. size (Box Plot)') + theme_minimal()

# ignore the warnings (because of ignoring outliers)
options(warn=-1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11709977542508823, 'fraction_numerical': 0.018126403593198587, 'mean_word_length': 3.2128378378378377, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '443561', 'license': 'MIT', 'n_tokens_mistral': 2133, 'n_tokens_neox': 1955, 'n_words': 691, 'path': 'Machine Learning study notes/Linear Models for Classification and regression/src/31436285_assessment_1_q3.ipynb', 'repo_name': 'alanwzy/my-projects', 'size': '6234'}