<jupyter_start><jupyter_text>Problem StatmentYou have been provided a dataset that consists of echo-location clicks of two types of whales, namely, Gervais and Cuviers. In this assignment, your task is to classify the different types whales using Gradient Boosting with the help of the XGBoost library. You are expected to fill in functions that would complete this task. We use XGBoost here instead of GradientBoostedTrees in Spark because XGBoost running on a single machine is much faster than Spark running on 10 machines.The data files were preprocessed on PySpark (10 nodes) cluster. The code for the same can be found in Data_Processing_Whales.ipynb. The preprocessed data is a numpy array with `4175` rows (for the 10mb file) with following columns (zero-indexed):* Col 0-9: projections on first 10 eigen vectors* Col 10: rmse* Col 11: peak2peak* Col 12: label (`0 if row.species==u'Gervais' else 1`)You can also refer to XGBoost_Whales.ipynb under for more details on the XGBoost Analysis before you attempt this assignment.Both Data_Processing_Whales.ipynb and XGBoost_Whales.ipynb can be found under XGBoost directory that was uploaded in edX as a part of "Notebooks for weeks 7 & 8".   XGBoost - TheoryA brief overview of gradient boosting in XGBoost can be found here:* http://xgboost.readthedocs.io/en/latest/model.html* http://xgboost.readthedocs.io/en/latest/python/python_intro.html Use the XGBoost API for training and predicting scores: * http://xgboost.readthedocs.io/en/latest/python/python_api.html Main API* `xgboost.train` is the learning API that trains the Gradient Boosting Model,   * The main parameters are:      * **plst** – XGBoost parameter list      * **dtrain** – Data to be trained      * **num_round** – Number of iterations of boosting. (default: 100)      * **evallist** – List of items to be evaluated during training      * **verbose_eval** - This can be used to control how much information the train function prints. You might want to set to **False** to avoid printing logs.* `bst.predict` is the API that makes score predictions   * The main parameters are:      * **dtest** – Test Data      * **dtrain** – Data to be trained      * **ntree_limit** – Limit number of trees in the prediction (Use: ntree_limit=bst.best_ntree_limit)      * **output_margin** - Whether to output the raw untransformed margin value (Use: output_margin=True)  Notebook Setup  Importing Required Libraries<jupyter_code>%matplotlib inline
import numpy as np
import xgboost as xgb
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import pickle
import random<jupyter_output><empty_output><jupyter_text>Loading Data<jupyter_code>with open('Data/X_train.pkl', 'rb') as f:
    X_train = pickle.load(f)

with open('Data/X_test.pkl', 'rb') as f:
    X_test = pickle.load(f)

with open('Data/y_train.pkl', 'rb') as f:
    y_train = pickle.load(f)

with open('Data/y_test.pkl', 'rb') as f:
    y_test = pickle.load(f)<jupyter_output><empty_output><jupyter_text>Setting Parameters for XG Boost* Maximum Depth of the Tree = 3 _(maximum depth of each decision trees)_* Step size shrinkage used in update to prevents overfitting = 0.3 _(how to weigh trees in subsequent iterations)_* Evaluation Criterion= Maximize Loglikelihood according to the logistic regression _(logitboost)_* Maximum Number of Iterations = 1000 _(total number trees for boosting)_* Early Stop if score on Validation does not improve for 5 iterations[Full description of options](https://xgboost.readthedocs.io/en/latest//parameter.html)<jupyter_code>#You can change this cell if you wish to, but you aren't expected to
def xgboost_plst():
    param = {}
    param['max_depth']= 3   # depth of tree
    param['eta'] = 0.3      # shrinkage parameter
    param['silent'] = 1     # not silent
    param['objective'] = 'binary:logistic'
    param['nthread'] = 7 # Number of threads used
    param['eval_metric'] = 'logloss'

    plst = param.items()
    return plst<jupyter_output><empty_output><jupyter_text>Exercises  Computing the score rangesThe function calc_stats takes the xgboost margin scores as input and returns two numpy arrays min_scr, max_scr which are calculated as follows:1. **min_scr**: mean - (3 $\times$ std)2. **max_scr**: mean + (3 $\times$ std)Here the input margin scores, represents the processed XGBoost margin scores obtained from the bootstrap_pred function. Each row represents the various scores for a specific example in an iteration and your calc_stats function is supposed to compute the **min_scr** and **max_scr** as defined for each example. So in the example below, we take a scenario where we have 3 examples which have 4 values each (From 4 bootstrap iterations).**Example Input**``` python[[-0.22 -0.19 -0.17 -0.13][-0.1 -0.05 0.02 0.10][0.03 0.11 0.12 0.15]]```Output: min_scr (numpy array), max_scr (numpy array)**Example Output**``` python(array([-0.28 -0.23 -0.03]), array([-0.08  0.22  0.24]))```**Note**: Ensure you round the values in the numpy arrays to two decimal places<jupyter_code>def calc_stats(margin_scores):
    min_scr = []
    max_scr = []
    
    for scores in margin_scores:
        std = np.std(scores)
        mean = np.mean(scores)
        min_scr.append(round(mean - 3 * std, 2))
        max_scr.append(round(mean + 3 * std, 2))
    
    return (np.array(min_scr), np.array(max_scr))
margin_score = np.array([[-0.22, -0.19, -0.17, -0.13], [-0.1, -0.05, 0.02, 0.10], [0.03, 0.11, 0.12, 0.15]])
min_score, max_score = calc_stats(margin_score)
assert type(min_score) == np.ndarray, 'Incorrect Return type'
assert type(max_score) == np.ndarray, 'Incorrect Return type'
assert (min_score == np.array([-0.28, -0.23, -0.03])).all(), "Incorrect return value"
assert (max_score == np.array([-0.08,  0.22,  0.24])).all(), "Incorrect return value"
# Hidden Tests Here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests Here
#
# AUTOGRADER TEST - DO NOT REMOVE
#<jupyter_output><empty_output><jupyter_text>Calculating predictions Based on the ranges for each of the examples, i.e, (min_scr, max_scr), we can predict whether it's a Gervais or a Cuvier. Since all our scores will be between -1 and +1, we use 0 as the margin line. All examples which are on the left side of the margin, can be classified as Cuvier's and all which are on the right side can be classified as Gervais. However, since we take margin scores from a set of bootstraps for each example, we use the minimum and maximum score arrays to predict to determine the classification.The function predict takes the minimum score array and maximum score array and returns predictions as follows:1. If respective minimum score and maximum score values are less than 0, predict -1 (**Cuvier's**)2. If respective minimum score value is less than 0 and maximum score value is greater than 0, predict 0 (**Unsure**)3. If respective minimum score and maximum score values are greater than 0, predict 1 (**Gervais**)**Example Input**``` pythonmin_scr (numpy array) = [-0.78 -0.68 -0.6 -0.53 -0.47 -0.42 -0.32 -0.21 -0.07 0.22]max_scr (numpy array) = [-0.49 -0.39 -0.33 -0.25 -0.2 -0.11 -0.04 0.1 0.3 0.51]```Output: pred (numpy array of predictions)**Example Output**``` python[-1 -1 -1 -1 -1 -1 -1  0  0  1]```<jupyter_code>def predict(min_scr, max_scr):
    #
    # YOUR CODE HERE
    #
    assert len(min_scr) == len(max_scr), "Invalid parameter length."
    
    ret = []
    for i in range(len(min_scr)):
        minScr = min_scr[i]
        maxScr = max_scr[i]
        
        if (minScr < 0.0 and maxScr < 0.0):
            ret.append(-1)
        elif (minScr > 0.0 and maxScr > 0.0):
            ret.append(1)
        else:
            ret.append(0)
    
    return np.array(ret)
max_s = np.array([-0.49, -0.39, -0.33, -0.25, -0.2, -0.11, -0.04, 0.1, 0.3, 0.51])
min_s = np.array([-0.78, -0.68, -0.6, -0.53, -0.47, -0.42, -0.32, -0.21, -0.07, 0.22])
pred = predict(min_s, max_s)
true_pred = np.array([-1, -1, -1, -1, -1, -1, -1, 0, 0, 1])

assert type(pred) == np.ndarray, 'Incorrect return type'
assert (pred == true_pred).all(), 'Incorrect return value'<jupyter_output><empty_output><jupyter_text>Calculating scores The function bootstrap_pred takes as input:1. **Training set**1. **Test set**1. **n_bootstrap** Number of bootstrap samples that run XGBoost and trains one part of the sample set.1. **minR, maxR** two numbers such that $0 < minR < maxR < 1$ that define the fractions of the `n_bootstrap` scores that define the range.1. **bootstrap_size** - Number of bootstrap samples on which you will run XGBoost.1. **num_round** - Number of iterations for running xgboost1. **plst** - Parameter ListThe output should be a confidence interval for each example in the test set. Together with a prediction that is `Gervais / Cuviers / Unsure`. The prediction `unsure` is to be output if the confidence interval contains the point 0.After generating the confidence intervals, the function predict can be used to make predictions. **Procedure**Repeat the given procedure for n_bootstrap number of iterations:For **n_bootstrap** iterations:* Sample **boostrap_size** indices from the training set **with replacmennt*** Create train and test data matrices (dtrain, dtest) using xgb.DMatrix(X_sample, label=y_sample)* Initialise the evallist parameter [(dtrain, 'train'), (dtest, 'eval')]* Train the model using the XGBoost train API and make score predictions using bst.predict object returned by XGB train API. Ensure you set **output_margin=True** to get raw untransformed output scores.* Normalize them by dividing them with the normalizing factor as max(scores) - min(scores) and round these values to a precision of two decimal placesThen: * For each individual example, remove scores below the minRth percentile and greater than maxRth percentile (sort for each example if necessary)* Call the calc_stats function to compute min_scr and max_scr with the filtered margin scores as parameter* Return the min_scr and max_scr computed by the **calc_stat** function using the margin scores.      **Note**: You can experiment by changing **n_bootstraps**, but it takes about 200 iterations to get consistent values.<jupyter_code>from sklearn.utils import resample

def removeOutliers(resultRow, minIndex, maxIndex):
    return np.sort(resultRow)[minIndex : maxIndex]

def bootstrap_pred(X_train, X_test, y_train, y_test, n_bootstrap, minR, maxR, bootstrap_size, \
                   num_round=100, plst=xgboost_plst()):
    #
    # YOUR CODE HERE
    #
    predictionMatrix = []
    dTest = xgb.DMatrix(X_test, label=y_test)
    for i in range (n_bootstrap):
        # Sampling dataset with replacement to create data to train model.
        X_trainSample, y_trainSample = resample(X_train, y_train, n_samples = bootstrap_size)          
        dTrain = xgb.DMatrix(X_trainSample, label=y_trainSample)
        evallist = [(dTrain, 'train'), (dTest, 'eval')]
        
        # Creating gradient boost model and getting its test results.
        bst = xgb.train(plst, dTrain, num_round, evallist, verbose_eval=False)
        y_pred = bst.predict(dTest, ntree_limit=bst.best_ntree_limit, output_margin=True)
                
        # Normalizing results.
        yMin = np.min(y_pred)
        yMax = np.max(y_pred)
        normalizer = (yMax - yMin)
        normalizedPred = [y/normalizer for y in y_pred]
        normalizedPred = np.round(normalizedPred, 2)             
        predictionMatrix.append(normalizedPred)

    # After running XGBoost n_bootstrap times, each row represents a single test prediction set,
    # where each column is the result of a single test element prediction.
    # We will transpose it so each row will represent prediction set over the same element.
    resultsForElements = np.array(predictionMatrix).transpose()

    # Removing outliers.
    ret = []
    minIndex = round(minR * n_bootstrap)
    maxIndex = round(maxR * n_bootstrap)    
    for row in resultsForElements:
        ret.append(np.array(removeOutliers(row, minIndex, maxIndex)))
    
    # Finally we return for each test element, the minimum and maximum score found by
    # the n_bootstrap XGBoost executions based on the boostrap aggregation.
    return calc_stats(ret)
def process(X_train, X_test, y_train, y_test, n_bootstrap=200):
    min_scr, max_scr = bootstrap_pred(X_train, X_test, y_train, y_test, n_bootstrap=n_bootstrap, \
                                            minR=0.1, maxR=0.9, bootstrap_size=len(X_train))
    pred = predict(min_scr, max_scr)
    return min_scr, max_scr, pred<jupyter_output><empty_output><jupyter_text>TestsHow we test the function:1. We have computed the average mid-point of the range of values and verify that this midpoint is present in the range computed by your function2. We check that the length of the interval(max_scr-min_scr) is not more than twice the average length of the interval<jupyter_code># Loading data.
sample_indices = np.load('Data/vis_indices.npy')
X_test_samp = X_test[sample_indices]
y_test_samp = np.array(y_test[sample_indices], dtype=int)
midpt = np.load('Data/vis_midpt.npy')
avg_length = np.load('Data/vis_avg_length.npy')

# Test indexes.
print(sample_indices)

# Data element example.
print(X_test_samp[0])

# Expected results for all tests.
print(y_test_samp)

# Average mid-point of the range of values.
print(midpt)

# Measuring boosting algorithm.
min_scr, max_scr, pred = process(X_train, X_test_samp, y_train, y_test_samp)
length = max_scr - min_scr
print(length[0:10])
print(np.mean(length))
assert sum(min_scr <= midpt) >= (0.7 * len(sample_indices)), "Incorrect range (mean - 3*std) to (mean + 3*std)"
assert sum(max_scr >= midpt) >= (0.7 * len(sample_indices)), "Incorrect range (mean - 3*std) to (mean + 3*std)"
assert sum(length < 2*avg_length) >= (0.7 * len(sample_indices)), "Incorrect length of range (mean - 3*std) to (mean + 3*std)"
# Hidden Test Here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests Here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests Here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests Here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests Here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests here
#
# AUTOGRADER TEST - DO NOT REMOVE
#

# Hidden Tests here
#
# AUTOGRADER TEST - DO NOT REMOVE
#<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10234703007777549, 'fraction_numerical': 0.024571546562048317, 'mean_word_length': 3.9354619565217392, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '160291', 'license': 'MIT', 'n_tokens_mistral': 4801, 'n_tokens_neox': 4540, 'n_words': 1705, 'path': 'BigDataWithSpark/BaggingAndGradientBoost/XGBoost_Whales.ipynb', 'repo_name': 'wfaria/dataScience', 'size': '14529'}