<jupyter_start><jupyter_text>Gradient boosting produces an ensemble of decision trees that, on their own, are weak decision models.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt


%matplotlib inline
# Mock data
x = np.arange(0, 60)
y = list(map(lambda x: x / 2 + (x // 10) % 2 * 20 * x / 5 + np.random.random() * 10, x))
x = pd.DataFrame({'x': x})

# Plot mock data
plt.figure(figsize=(10, 5))
plt.scatter(x, y)

# Using linear regression to fit in this first
from sklearn.linear_model import LinearRegression
lr= LinearRegression(normalize=True)
lr.fit(x,y)
# Plot  data
plt.figure(figsize=(10, 5))
plt.scatter(x, y)
plt.plot(x,lr.predict(x),'r--')
r= lr.score(x,y) # r2 score
#adjusted r2 """1-((1-r2)*(n-1)/n-p-1)"""
adr= 1-(((1-r)*(len(x)-1))/(len(x)-1-1)) 

print(r,adr)


from sklearn.metrics import mean_squared_error, mean_absolute_error
mse= mean_squared_error(y,lr.predict(x))
mae= mean_absolute_error(y,lr.predict(x))
rmse= np.sqrt(mse)
print(mse,mae,rmse)
# Graph of residuals
plt.figure(figsize=(10, 5))
plt.title("Residuals $(y - \hat{y})$ of Linear Regression model")
plt.scatter(x, y - lr.predict(x), color='brown')
plt.show()<jupyter_output><empty_output><jupyter_text>It is also clear from this plot that there is a pattern in the residual errors, these are not random errors. We could fit model to the error terms from the output of the first model<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor
params = {
    'n_estimators': 1, 
    'max_depth': 1,
    'learning_rate': 1,
    'criterion': 'mse'
}

gb= GradientBoostingRegressor(**params)<jupyter_output><empty_output><jupyter_text>learning_rate : float, optional (default=0.1)    learning rate shrinks the contribution of each tree by `learning_rate`.    There is a trade-off between learning_rate and n_estimators.n_estimators : int (default=100)    The number of boosting stages to perform. Gradient boosting    is fairly robust to over-fitting so a large number usually    results in better performance.max_depth : integer, optional (default=3)    maximum depth of the individual regression estimators. The maximum    depth limits the number of nodes in the tree. Tune this parameter    for best performance; the best value depends on the interaction    of the input variables.criterion : string, optional (default="friedman_mse")    The function to measure the quality of a split. Supported criteria    are "friedman_mse" for the mean squared error with improvement    score by Friedman, "mse" for mean squared error, and "mae" for    the mean absolute error. The default value of "friedman_mse" is    generally the best as it can provide a better approximation in    some cases.    min_samples_split : int, float, optional (default=2)    The minimum number of samples required to split an internal node:min_samples_leaf : int, float, optional (default=1)    The minimum number of samples required to be at a leaf node:<jupyter_code>gb.fit(x,y)
yp= gb.predict(x)
plt.figure(figsize=(10, 5))
plt.title('Gradient Boosting model (1 estimators, Single tree split)')
plt.scatter(x, y)
plt.plot(x, yp, color='r')
plt.show()<jupyter_output><empty_output><jupyter_text>We see that the depth 1 decision tree is split at x = 50, where:If x < 50, y = 56If x >= 50, y = 250This isn’t the best model, but Gradient Boosting models aren’t meant to have just 1 estimator and a single tree split. So where do we go from here, let’s look again at the residuals from this model:<jupyter_code>plt.figure(figsize=(10, 5))
plt.title('Residuals of Gradient boosting model (1 estimator, Single tree split)')
plt.scatter(x, y - yp, color='brown')
plt.show()<jupyter_output><empty_output><jupyter_text>With one estimator, the residuals between 30-40 are very high. So what if we had 2 estimators and we fed the residuals from this first tree into the next tree,<jupyter_code>params = {
    'n_estimators': 2, 
    'max_depth': 1,
    'learning_rate': 1,
    'criterion': 'mse'
}

gb= GradientBoostingRegressor(**params)
gb.fit(x,y)
yp= gb.predict(x)

plt.figure(figsize=(10, 5))
plt.title('Gradient Boosting model (2 estimators, Single tree split)')
plt.scatter(x, y)
plt.plot(x, yp, color='r')
plt.show()
params = {
    'n_estimators': 10, 
    'max_depth': 2,
    'learning_rate': 0.3,
    'criterion': 'mse'
}

gb= GradientBoostingRegressor(**params)
gb.fit(x,y)
yp= gb.predict(x)

plt.figure(figsize=(10, 5))
plt.title('Gradient Boosting model (2 estimators, Single tree split)')
plt.scatter(x, y)
plt.plot(x, yp, color='r')
plt.show()
params = {
    'n_estimators': 20, 
    'max_depth': 5,
    'learning_rate': 1,
    'criterion': 'mse',
    
}

gb= GradientBoostingRegressor(**params)
gb.fit(x,y)
yp= gb.predict(x)

plt.figure(figsize=(10, 5))
plt.title('Gradient Boosting model (2 estimators, Single tree split)')
plt.scatter(x, y)
plt.plot(x, yp, color='r')
plt.show()
r= gb.score(x,y) # r2 score
#adjusted r2 """1-((1-r2)*(n-1)/n-p-1)"""
adr= 1-(((1-r)*(len(x)-1))/(len(x)-1-1)) 

print(r,adr)


from sklearn.metrics import mean_squared_error, mean_absolute_error
mse= mean_squared_error(y,yp)
mae= mean_absolute_error(y,yp)
rmse= np.sqrt(mse)
print(mse,mae,rmse)

plt.figure(figsize=(10, 5))
plt.title('Residuals of Gradient boosting model (1 estimator, Single tree split)')
plt.scatter(x, y - yp, color='brown')
plt.show()<jupyter_output>0.9999999999999893 0.9999999999999891
8.372871019558281e-11 1.670615757857578e-06 9.150339348657119e-06<jupyter_text>So if you will see above, error line is straight, r2 and adj r2 is also 1, model is overfiiting according to me.s now i will increase the min sample leaf and splt which can effect it.<jupyter_code>params = {
    'n_estimators': 20, 
    'max_depth': 5,
    'learning_rate': 1,
    'criterion': 'mse',
    'min_samples_split': 8,
    'min_samples_leaf':5
}

gb= GradientBoostingRegressor(**params)
gb.fit(x,y)
yp= gb.predict(x)

plt.figure(figsize=(10, 5))
plt.title('Gradient Boosting model (2 estimators, Single tree split)')
plt.scatter(x, y)
plt.plot(x, yp, color='r')
plt.show()
r= gb.score(x,y) # r2 score
#adjusted r2 """1-((1-r2)*(n-1)/n-p-1)"""
adr= 1-(((1-r)*(len(x)-1))/(len(x)-1-1)) 

print(r,adr)


from sklearn.metrics import mean_squared_error, mean_absolute_error
mse= mean_squared_error(y,yp)
mae= mean_absolute_error(y,yp)
rmse= np.sqrt(mse)
print(mse,mae,rmse)

plt.figure(figsize=(10, 5))
plt.title('Residuals of Gradient boosting model (1 estimator, Single tree split)')
plt.scatter(x, y - yp, color='brown')
plt.show()<jupyter_output>0.9988734942941616 0.998854071781992
8.80788524057839 1.5339203375462722 2.9678081542745294<jupyter_text>So Parameter Tuning is the most important things.<jupyter_code>f, ax = plt.subplots(2, 2, figsize=(15, 10))

for idx, n_estimators in enumerate([5, 10, 20, 50]):
    
    params = {
    'n_estimators': n_estimators, 
    'max_depth': 1,
    'learning_rate': 1,
    'criterion': 'mse',
#     'min_samples_split': 8,
#     'min_samples_leaf':5
                        }

    gradient_boosting_regressor = GradientBoostingRegressor(**params)

    gradient_boosting_regressor.fit(x, y)
    subplot = ax[idx // 2][idx % 2]
    subplot.set_title('Gradient Boosting model ({} estimators, tree split)'.format(n_estimators))
    subplot.scatter(x, y)
    subplot.plot(x, gradient_boosting_regressor.predict(x), color='r')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13533631073216945, 'fraction_numerical': 0.044390309920151576, 'mean_word_length': 4.897845171588188, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '379205', 'license': 'MIT', 'n_tokens_mistral': 3015, 'n_tokens_neox': 2668, 'n_words': 756, 'path': 'Gradient Bossting.ipynb', 'repo_name': 'pankush9096/Boosting-Algorithms', 'size': '7389'}