<jupyter_start><jupyter_text>XgboostClassifier_RandomizedSearchCV**--------------------------------------------------------------------------------------------------------------------------****--------------------------------------------------------------------------------------------------------------------------****--------------------------------------------------------------------------------------------------------------------------****---------------------------------------------------****STRUCTURE***In this notebook, the use of the 'xgboost' algorithm for classification is demonstrated. The scope of this case study is to predict whether an asteroid can be hazardous or not based on its characteristics/properties(NASA Asteroid dataset). In the first part of this project a feature selection takes place so as to identify the dataset features that are to be used as inputs to the model and to drop all the uneccesary columns. In addition a mini exploratory analysis is provided that includes a) a heatmap of the correlations between the dataset features, b) a plot of the correlation between the output (dependent) parameter and each input (independent) feature and c) a countplot to observe whether the classes of the output feature are balanced or not.**In the second part of this project there is use of the 'RandomizedSearchCV' and 'XGBClassifier' function. 'RandomizedSearchCV' function takes as input a set of values for each of the selected 'xgboost' model hyperparameters and does a randomized search (fit & score approach) so as to determine an optimal value for each of them. In contrast to 'GridSearchCV', randomized search combines the hyperparameters values randomly (whereas in 'GridSearchCV' all possible combinations are tested) based on the parameters distribution. Tuning the values of hyperparameters (learning rate,max depth,min_child_weight,gamma,etc.) is important as it can improve the model training performance and avoid/minimize overfitting (by making the model less sensitive to noise and by contolling the complexity of its structure). The optimal hyperparameter values determined by the random search  are then used by the 'xgboost' classifier for fitting (training set) and prediction (test set). 'Xgboost' is an ensemble supervised machine learning algorithm (based on decision trees) that tries to reduce the residuals (difference between target and predicted values) of the previous tree each time a new tree is added (similar to 'Gradient Boosting') and it is also capable of reducing the computational complexity and time of the training compared to other machine learning models (data storage in CPU cache).***The Dataset (.csv file format) for this project has been obtained from Kaggle:**"*NASA: Asteroids Classification*" -- File: "nasa.csv" -- Source:https://www.kaggle.com/shrutimehta/nasa-asteroids-classification  Part A<jupyter_code># Importing the Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
# Importing the dataset
data=pd.read_csv('nasa.csv')
# Dataset First 5 records
data.head()
# Dataset columns
data.columns
# Dataset Info
data.info()
# In this classification example, the model has to learn to classify whether an asteroid is hazardous or not based on its
# characteristics/properties (without knowning its name or reference id). For this reason, the 'Neo Reference ID'  and 'Name'
# columns are to be dropped from the dataset.
data=data.drop(['Neo Reference ID','Name'],axis=1)
# Date features are also dropped from the dataset
data=data.drop(['Close Approach Date','Orbit Determination Date'],axis=1)
# This column has only one class ('Earth'), therefore it is not going to contribute during the training, thus it will be dropped
data['Orbiting Body'].value_counts()
# This column has also only one class('J2000'),therefore it is also going to be dropped
data['Equinox'].value_counts()
# Dropping 'Orbiting Body' and 'Equinox' columns from the dataset
data=data.drop(['Orbiting Body','Equinox'],axis=1)
# Replacing the white space in column names with underscore
data.columns=data.columns.str.replace(' ','_')
data.head()
# Heatmap displaying the correlations between the dataset features. 
plt.figure(figsize=(14,10))
sns.heatmap(data.corr(),cmap='plasma')
# Bar plot showing the correlations between the 'Hazardous' feature and the other numerical variables. 
# Correlation varies between -1 and 1, where a) values close to 1 indicate positive correlation, b) values close to -1
# indicate negative correlation between two variables.
plt.figure(figsize=(14,5))
plt.rcParams['font.size']=12
data.corr()['Hazardous'].sort_values().drop('Hazardous').plot(kind='bar')
plt.title("Correlations of the numerical variables to 'Hazardous' feature")
# Due to the boolean nature of the dependent(output) parameter (Hazardous:True/False), the countplot is 
# employed to observe whether or not the data samples are balanced with respect to the 'Hazardous' feature entries.
# As expected, the majority of the Asteroids are not Hazardous (83.89%). Therefore,the classification should not be based
# on accuracy as the model would be 83.89% accurate, but on other relevant metrics such as precision,recall and F1-Score
plt.figure(figsize=(10,6))
total = len(data['Hazardous'])
count_entries=sns.countplot(x='Hazardous',data=data)
plt.xlabel('Hazardous',fontweight='bold')
plt.ylabel('Count',fontweight='bold')
for p in count_entries.patches:
    count_entries.annotate('{:.2f}%'.format(100*p.get_height()/total), (p.get_x() + p.get_width() / 2., p.get_height()), 
                               ha = 'center', va = 'center', xytext = (0, -20), textcoords = 'offset points')<jupyter_output><empty_output><jupyter_text>Part B<jupyter_code># XGBOOST model independent(input) & dependent(output) features
X=data.drop('Hazardous',axis=1).values
y=data['Hazardous'].values
# Data are to be used for training and for validation purposes, thus the necessary libraries and functions are imported
from sklearn.model_selection import train_test_split
# Data are split for training and testing,test_size =0.25 which means 25% of the dataset will be used for validation purposes
# to assess the model performance
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.25, random_state=0)

# As it can be observed, the percentage of 'Hazardous ==True' for the y_train samples is the same with the y_test samples,
# (i.e.for both train and test sets)
print(round(len(y_train[y_train==True])/len(y_train),2))
print(round(len(y_test[y_test==True])/len(y_test),2))
# Optimizing the hyperparameters by selecting different values for each hyperparameter. Then by applying Random Search, the 
# model will try to determine the optimal values for all parameters
opt_parameters={
 "learning_rate"    : [0.001,0.005,0.01,0.05,0.10, 0.15, 0.20,0.25,0.3,0.35],#Learning_Rate:the step size at every iteration
 "max_depth"        : [ 2,3,4,5,6,7,8,9,10,11,12,13,14],#max_depth: max tree depth (very high values can lead to overfitting)
 "min_child_weight" : [ 1,2,3,4,5,6],# min_child_weight:minimum samples number in each node
 "gamma"            : [ 0,0.05, 0.1,0.15, 0.2,0.25,0.3,0.35,0.4,0.45,0.5],#gamma:higher gamma value --> higher regularization
'colsample_bylevel' : [ 0.3,0.4, 0.5,0.6,0.7],#colsample_bytree:columns subsample ratio for each level
'colsample_bynode'  : [ 0.3,0.4, 0.5,0.6,0.7],#colsample_bynode:columns subsample ratio for each split
'colsample_bytree'  : [ 0.3,0.4, 0.5,0.6,0.7]}#colsample_bytree:feature percentage for building each tree
## Hyperparameter optimization using RandomizedSearchCV
from sklearn.model_selection import RandomizedSearchCV
# Importing xgboost classifier fit the model to the training set
from xgboost import XGBClassifier
xgboost_clf=XGBClassifier()
# Use of RandomizedSearchCV function that takes as inputs the classifier, the parameters that are to be optimized, the
#  number of iterations, the score metric and the cross validation number
from sklearn.metrics import make_scorer,average_precision_score
scoring = make_scorer(average_precision_score)
rand_search=RandomizedSearchCV(xgboost_clf,param_distributions=opt_parameters,n_iter=7,scoring=scoring,
                                 cv=10,verbose=3)
# RandomizedSearchCV to fit the model to the training set
rand_search.fit(X_train, y_train)
# Best estimator returns optimal values for all the hyperparameters of the RandomizedSearchCV()
rand_search.best_estimator_
# Best params returns optimal values for the previously defined opt_parameters(dict)  
rand_search.best_params_
# Best score obtained from the RandomizedSearchCV()
rand_search.best_score_
# RandomizedSearchCV cross validation results
rand_search.cv_results_
# Mean test score for each split
rand_search.cv_results_['mean_test_score']
# Mean test score of all splits
rand_search.cv_results_['mean_test_score'].mean()
# Fitting the model to the training set by using the RandomizedSearchCV best params

xgboost_clf=XGBClassifier(objective='binary:logistic',missing=None,booster='gbtree',n_estimators=100,scale_pos_weight=1,
                          min_child_weight=1,max_depth=8,learning_rate=0.1,gamma=0.35,
                          colsample_bylevel=0.7,colsample_bynode=0.3,colsample_bytree=0.7,seed=0)
xgboost_clf.fit(X_train,y_train,verbose=True,early_stopping_rounds=10,eval_metric='aucpr',eval_set=[(X_test,y_test)])
# XGBClassifier Predictions on the test set
ypred=xgboost_clf.predict(X_test)
# Importing the metrics
from sklearn.metrics import classification_report,plot_confusion_matrix

# Classification report. As it can be observed, F1 scores are very high,meaning that the xgboost has provided very accurate
# classifications (i.e. whether an asteroid is hazardous or not)
print(classification_report(y_test,ypred))
# Confusion Matrix
fig, axs = plt.subplots(figsize=(8,8))
plot_confusion_matrix(xgboost_clf,X_test,y_test,values_format='d',
                      display_labels=['Not Hazardous','Hazardous'],cmap='plasma',ax=axs);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1267909715407262, 'fraction_numerical': 0.018743866535819432, 'mean_word_length': 4.24768280123584, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '534958', 'license': 'MIT', 'n_tokens_mistral': 2997, 'n_tokens_neox': 2848, 'n_words': 1028, 'path': 'XgboostClassifier_RandomizedSearchCV.ipynb', 'repo_name': 'tourloukisg/XGBOOST_Classification_HyperparameterTuning', 'size': '10190'}