<jupyter_start><jupyter_text>XGBoostXGBoost improves the gradient boosting method even further.> **XGBoost** (*extreme gradient boosting*) regularises data better than normal gradient boosted Trees.It was developed by <NAME> in C++ but now has interfaces for Python, R, Julia.XGBoost's objective function is the sum of loss function evaluated over all the predictions and a regularisation function for all predictors ($j$ trees). In the formula $f_j$ means a prediction coming from the $j^th$ tree.$$obj(\theta) = \sum_{i}^{n} l(y_i - \hat{y_i}) +  \sum_{j=1}^{j} \Omega (f_j)$$Loss function depends on the task being performed (classification, regression, etc.) and a regularization term is described by the following equation:$$\Omega(f) = \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T}w_j^2$$First part ($\gamma T$) is responsible for controlling the overall number of created leaves, and the second term ($\frac{1}{2} \lambda \sum_{j=1}^{T}w_j^2$) watches over the scores. **Mathematics Involved**Unlike the other tree-building algorithms, XGBoost doesn’t use entropy or Gini indices. Instead, it utilises gradient (the error term) and hessian for creating the trees. Hessian for a Regression problem is the *number of residuals* and for a classification problem. Mathematically, Hessian is a second order derivative of the loss at the current estimate given as:where **L** is the loss function. - Initialise the tree with only one leaf.- compute the similarity using the formula$$Similarity= \frac {Gradient^2}{ hessian +\lambda}$$Where $\lambda $ is the regularisation term.- Now for splitting data into a tree form, calculate$$Gain=  left similarity+right similarity-similarity for root$$ - For tree pruning, the parameter $ \gamma$ is used. The algorithm starts from the lowest level of the tree and then starts pruning based on the value of $\gamma$. If $Gain- \gamma < 0$, remove that branch. Else, keep the branch  - Learning is done using the equation$$New Value= old Value+ \eta * prediction$$where $\eta$ is the learning rate  Python Implementation **Problem Statement**:The Pima Indians Diabetes Dataset involves predicting the onset of diabetes within 5 years in Pima Indians given medical details.It is a binary (2-class) classification problem. The number of observations for each class is not balanced. There are 768 observations with 8 input variables and 1 output variable. Missing values are believed to be encoded with zero values. The variable names are as follows:1.	Number of times pregnant.2.	Plasma glucose concentration 2 hours in an oral glucose tolerance test.3.	Diastolic blood pressure (mm Hg).4.	Triceps skinfold thickness (mm).5.	2-Hour serum insulin (mu U/ml).6.	Body mass index (weight in kg/(height in m)^2).7.	Diabetes pedigree function.8.	Age (years).9.	Is Diabetic (0 or 1).<jupyter_code>import pandas as pd
import numpy as np
import xgboost as xgb
import pickle
from sklearn import datasets
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score
# reading the features and the labels
data= pd.read_csv('pima-indians-diabetes.csv')
data.head()
data.columns
cols = ['Plasma glucose concentration',
       'Diastolic blood pressure (mm Hg)', 'Triceps skinfold thickness (mm)',
       '2-Hour serum insulin (mu U/ml)',
       'Body mass index (weight in kg/(height in m)^2)',
       'Diabetes pedigree function', 'Age']
# as mentioned in the data description, the missing values have been replaced by zeroes. So, we are replacing zeroes with nan
for col in cols:
    data[col]=data[col].replace(0, np.nan)
# checking for missing values
data.isna().sum()
# imputing the missing values
data['Plasma glucose concentration']=data['Plasma glucose concentration'].fillna(data['Plasma glucose concentration'].mode()[0])
data['Diastolic blood pressure (mm Hg)']=data['Diastolic blood pressure (mm Hg)'].fillna(data['Diastolic blood pressure (mm Hg)'].mode()[0])
data['Triceps skinfold thickness (mm)']=data['Triceps skinfold thickness (mm)'].fillna(data['Triceps skinfold thickness (mm)'].mean())
data['2-Hour serum insulin (mu U/ml)']=data['2-Hour serum insulin (mu U/ml)'].fillna(data['2-Hour serum insulin (mu U/ml)'].mean())
data['Body mass index (weight in kg/(height in m)^2)']=data['Body mass index (weight in kg/(height in m)^2)'].fillna(data['Body mass index (weight in kg/(height in m)^2)'].mean())

# checking for missing values after imputation
data.isna().sum()
#Separating the feature and the Label columns 
x=data.drop(labels='Is Diabetic', axis=1)
y= data['Is Diabetic']
x.head()
# as the datapoints differ a lot in magnitude, we'll scale them
from sklearn.preprocessing import StandardScaler
scaler=StandardScaler()
scaled_data=scaler.fit_transform(x)
from sklearn.model_selection import train_test_split
train_x,test_x,train_y,test_y=train_test_split(scaled_data,y,test_size=0.3,random_state=42)
# fit model no training data
model = XGBClassifier(objective='binary:logistic')
model.fit(train_x, train_y)
# cheking training accuracy
y_pred = model.predict(train_x)
predictions = [round(value) for value in y_pred]
accuracy = accuracy_score(train_y,predictions)
accuracy
# cheking initial test accuracy
y_pred = model.predict(test_x)
predictions = [round(value) for value in y_pred]
accuracy = accuracy_score(test_y,predictions)
accuracy
test_x[0]
# Now to increae the accuracy of the model, we'll do hyperparameter tuning using grid search

from sklearn.model_selection import GridSearchCV
param_grid={
   
    ' learning_rate':[1,0.5,0.1,0.01,0.001],
    'max_depth': [3,5,10,20],
    'n_estimators':[10,50,100,200]
    
}
grid= GridSearchCV(XGBClassifier(objective='binary:logistic'),param_grid, verbose=3)
grid.fit(train_x,train_y)
# To  find the parameters givingmaximum accuracy
grid.best_params_
# Create new model using the same parameters
new_model=XGBClassifier(learning_rate= 1, max_depth= 5, n_estimators= 50)
new_model.fit(train_x, train_y)
y_pred_new = new_model.predict(test_x)
predictions_new = [round(value) for value in y_pred_new]
accuracy_new = accuracy_score(test_y,predictions_new)
accuracy_new
# As we have increased the accuracy of the model, we'll save this model
filename = 'xgboost_model.pickle'
pickle.dump(new_model, open(filename, 'wb'))

loaded_model = pickle.load(open(filename, 'rb'))
# we'll save the scaler object as well for prediction
filename_scaler = 'scaler_model.pickle'
pickle.dump(scaler, open(filename_scaler, 'wb'))

scaler_model = pickle.load(open(filename_scaler, 'rb'))
# Trying a random prediction
d=scaler_model.transform([[6,148,72,35,80,33.6,0.627,50]])
pred=loaded_model.predict(d)
print('This data belongs to class :',pred[0])<jupyter_output>This data belongs to class : 1<jupyter_text>**The main advantages:**- out of the box feature of appropriate bias-variance trade-off,- great computation speed as it utilises parallel computing and cache optimization,- uses hardware optimization,- works well even if the features are correlated- robust even if there is noise for classification problem- the facility of early stopping- the package is evolving, i.e., new features are being added.  Cloud DeploymentOnce the training is completed, we need to expose the trained model as an API for the user to consume it. For prediction, the saved model is loaded first and then the predictions are done using it. If the web app works fine, the same app is deployed to the cloud platform.The flow for that can be shown as:We'll deploy this model to the Google Cloud Platform. **Pre-requisites for Cloud Deployment:*** Basic knowledge of flask framework.* Any Python IDE installed(we are using PyCharm).* A Google Cloud Platform account.* Basic understanding of HTML. **The Flask App**As we’ll expose the created model as a web API to be consumed by the client/client APIs, we’d do it using the flask framework. * Create the project structure, as shown below: The contents of **main.py** are:<jupyter_code># importing the necessary dependencies
from flask import Flask, render_template, request,jsonify
from flask_cors import CORS,cross_origin
import pickle

app = Flask(__name__) # initializing a flask app


@app.route('/predict',methods=['POST','GET']) # route to show the predictions in a web UI
@cross_origin()
def index():
    if request.method == 'POST':
        try:
            #  reading the inputs given by the user
            number_of_times_pregnant=float(request.json['number_of_times_pregnant'])
            plasma_glucose_concentration = float(request.json['plasma_glucose_concentration'])
            diastolic_blood_pressure = float(request.json['diastolic_blood_pressure'])
            triceps_skinfold_thickness  = float(request.json['triceps_skinfold_thickness'])
            serum_insulin  = float(request.json['serum_insulin'])
            body_mass_index  = float(request.json['body_mass_index'])
            diabetes_pedigree_function = float(request.json['diabetes_pedigree_function'])
            age = float(request.json['age'])

            # Loading the saved models into memory
            filename_scaler = 'scaler_model.pickle'
            filename = 'xgboost_model.pickle'
            scaler_model = pickle.load(open(filename_scaler, 'rb'))
            loaded_model = pickle.load(open(filename, 'rb'))

            # predictions using the loaded model file
            scaled_data=scaler_model.transform([[number_of_times_pregnant,plasma_glucose_concentration,diastolic_blood_pressure,triceps_skinfold_thickness,serum_insulin,body_mass_index,diabetes_pedigree_function, age]])
            prediction=loaded_model.predict(scaled_data)
            print('prediction is', prediction[0])
            if prediction[0]==1:
                result= 'The Patient is Diabetic'
            else:
                result = 'The Patient is not Diabetic'
            # showing the prediction results in a UI
            return jsonify(result)
        except Exception as e:
            print('The Exception message is: ',e)
            return jsonify('error: Something is wrong')
    # return render_template('results.html')



if __name__ == "__main__":
    #app.run(host='127.0.0.1', port=8001, debug=True)
   app.run(debug=True) # running the app<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09914764377388068, 'fraction_numerical': 0.011364749681591065, 'mean_word_length': 4.7967064168086315, 'pattern_counts': {'":': 1, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '136254', 'license': 'Apache-2.0', 'n_tokens_mistral': 3099, 'n_tokens_neox': 2919, 'n_words': 1080, 'path': 'Boosting/XG-Boost in detail..ipynb', 'repo_name': 'alan-ai-learner/Boosting-Algorithms-in-ML', 'size': '10212'}