<jupyter_start><jupyter_text>Introduction[Video](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch1_1.master.m3u8)**Supervised Learning**- Relies on labelled data- Problems can be either classification or regression.- Classification problems can forther be divided into binary classification problems and multi-class classification problems- Eg. Classification problem of identifying if an image contains face of a specific person or not where target would be either 1 or 0- AUC (Area Under Receiver Operating Characteristics) curve is one of the most versatile and commonly used evaluation metric to judge the quality of a **binary** classification model.**AUC (Area Under Receiver Operating Characteristics) curve**- It is simply the probability that a randomly chosen positive data point will have a higher rank than a randomly chosen negative data point for your learning problem.> Higher AUC => More sensitive, better performing model------**Accuracy Score**- For **multi-class** classification problems, it is more common to use **Accuracy score** (higher is better) and to look at the overall **confusion matrix** to evaluate the quality of the model.**Confusion matrix**  ![Confusion matrix](https://www.researchgate.net/profile/Enoch_Opanin_Gyamfi5/publication/319183193/figure/fig4/AS:668388829437953@1536367723057/Modified-Confusion-Matrix-Table-for-Accuracy-Prediction-of-24.ppm)> **Accuracy = $\frac{t_p + t_n}{t_p + t_n + f_p + f_n} $**- All features should be either numerical or categorical- Numeric features should be scaled (commonly scaled on Z-score)- Categorical features should be encoded (most commonly used is one-hot)  ContentPart 1:  Classification With XGBoost  Part 2:  Regression witb XGBoost  Part 3:  Fine-tuning XGBoost model  Part 4:  Using XGBoost in pipelines  Part-1: Classification with XGBoost<jupyter_code>import pandas as pd
pd.read_csv("https://assets.datacamp.com/production/repositories/943/datasets/4dbcaee889ef06fb0763e4a8652a4c1f268359b2/ames_housing_trimmed_processed.csv")<jupyter_output><empty_output><jupyter_text>XGBoost[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb2/hls-ch1_3.master.m3u8)- Optimized gradient boosting- XGBoost is very popular mainly because of its speed and performance- The core XGBoost algorithm is parallelizable. It can use up all the CPU cores and can also run on GPUs across networks of computers making it feasible to train models on very large datasets- XGBoost consistently outperforms all single-algorithm methods. It is capable of achieving SOTA in many machine learning tasks```Pythonimport xgboostxg_cl = xgboost.XGBClassifier(objective='binary:logistic', n_estimators=10, seed=123)xg_cl.fit(X_train, y_train)labels = xg_cl.predict(X_test)```  XGBoost on churn data from a ride sharing appThe dataset contains imaginary data from a ride-sharing app with user behaviors over their first month of app usage in a set of imaginary cities as well as whether they used the service 5 months after sign-up.  **Downloading the dataset**<jupyter_code>!wget --no-check-certificate -q "https://docs.google.com/uc?export=download&id=16D-Ab5jj66q05MuxQTh3BRONc4VC5FkB" -O churn_data.csv
import xgboost as xgb
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

churn_data = pd.read_csv('churn_data.csv')
churn_data.head()<jupyter_output><empty_output><jupyter_text>**Keeping the target variable aside from the dataset**<jupyter_code>X, y = churn_data.iloc[:, :-1], churn_data.iloc[:,-1]<jupyter_output><empty_output><jupyter_text>**Splitting the dataset into training and testing**<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
                                                    random_state=123)<jupyter_output><empty_output><jupyter_text>**Creating the XGBoost classifier, fitting the data, predicting on test set and calculating accuracy**<jupyter_code># Creating the classifier
xg_cl = xgb.XGBClassifier(method='binary:logistic', n_estimators=10, seed=123)

# Fitting the training data
xg_cl.fit(X_train, y_train)

# Preciting labels of test set
labels = xg_cl.predict(X_test)

# Accuarcy
accuracy = np.sum(labels == y_test)/y_test.shape[0]
print('Accuracy: {}'.format(accuracy))<jupyter_output>Accuracy: 0.7432<jupyter_text>Well, our model has an accuracy of around 74%. This can be further improved by fine tuning. --- What is a Decision Tree?[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/v3/hls-3679_ch1_2.master.m3u8)- XGBoost is usually used with tree based learners.**Example Decision Tree**![alt text](https://databricks.com/wp-content/uploads/2014/09/decision-tree-example.png)- A single questtion is asked at each decision node and there are only 2 possible choices- At the bottom of the tree there is a single possible decision Decision tree as a base learner Base learnerAn individual learning algorithm in an ensemble Decision Trees- XGBoost is itself an ensemble method, and uses the outputs from many models to make the final predicion- Using decision tree as a base learner, it is composed of a series of binary decisions- Predictions happen at the leaves of the tree- Decision Trees are constructed iteratively, that is, one binary decision at a time, until some stopping criteria is met- During construction, tree is built one split at a time. The way that the split is selected can vary (entropy, gini etc.), that is what feature to split on, where in the features values to split can vary, but it involves choosing a split point that segregates the target values better.- Using this process, each leaf would have a single category in majority, or should be exclusively of one category- Individual decision trees are **low-bias, high-variance** learning algorithms, meaning they tend to **overfit** the training data XGBoost and Decision Trees- XGBoost uses a slighlty different verison of decision trees (CARTs)- Each leaf **always** contains a real-values score, regardless of whether they are used for classification or regression- These real-values scores can later be thresholded to convert to different categories for classification problems  Decision Trees on Breast Cancer dataset<jupyter_code>from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)

print(X.shape) # Numpy array w/o column names info
print(y.shape)
# Splitting the data ito train and test sets as 80%-20%
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
                                                    random_state=123)

# # Instantiate the classifier: dt_clf_4
dt_clf_4 = DecisionTreeClassifier(max_depth=4)

# Fitting the classifier to the training set
dt_clf_4.fit(X_train, y_train)

# Predicting the labels of the test set
y_pred_4 = dt_clf_4.predict(X_test)

# Computing the accuracy of the predictions
accuracy = float(np.sum(y_pred_4==y_test))/y_test.shape[0]
print("accuracy:", accuracy)<jupyter_output>accuracy: 0.9736842105263158<jupyter_text>What is Boosting?[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch1_4.master.m3u8)- Not a specific ML algorithm, but a concept that can be applied to a set of ML models. So it is a "meta-algorithm".- Ensemble "meta-algorithm" that converts many weak learners into a strong learner Weak Learners and Strong Learners- **Weak learner:** ML algorithm that is only slightly better than chance. Eg. a DT whose prediction accuracy is slightly over 50%.- Boosting converts a collection of weak learners into a strong learner- **Strong learner:** Any algorithm that can be tuned to achieve arbitarily good performance for some supervised learning problem How boosting is accomplished- By iteratively learning a set of weak models on subsets of data- Weighting each prediction according to each weak learner's performance.- Then combine all of the weak learners' predictions multiplied by their weights to obtain a single weighted prediction. This prediction turns out to be much better than any of the individual predictions**XGBoost example**![alt text](https://raw.githubusercontent.com/dmlc/web-data/master/xgboost/model/twocart.png)--- Model validation through cross validation- Different API for cross validation from scikit-learn API- **Cross Validation:** A robust method for estimating the expected performance of a ML model on unseen data by generating many non-overlapping train-test splits on the training data, and reporting the average test set accuracy across all data splits- We create something called as a **XGBoost DMatrix**, which is an optimized data structure that gives XGBoost its performance efficiency gains- In the previous exercise, the input data was converted into DMatrix on-the-fly. But when we use the XGBoost CV object, we have to first explicitly convert our data into DMatrix<jupyter_code>import xgboost as xgb
import pandas as pd

churn_data = pd.read_csv('churn_data.csv')
# Create arrays for the features and the target
X, y = churn_data.iloc[:,:-1], churn_data.iloc[:,-1]

# Creating XGBoost DMatrix from X and y
churn_dmatrix = xgb.DMatrix(data=X, label=y)

# Create the parameter dictionary
params = {"objective":"reg:logistic", "max_depth":3}

# Perform cross-validation
cv_results = xgb.cv(dtrain=churn_dmatrix, params=params, 
                  nfold=3, num_boost_round=5, 
                  metrics="error", as_pandas=True, seed=123)
# dtrain is the data to train on as a DMatrix
# params: Parameter dictionary for XGBoost
# nfold: number of folds for cross validation
# num_boost_round: Number of trees we want to build
# metrics: metric you want to compute

# Print cv_results
print(cv_results)

# Print the accuracy
print(((1-cv_results["test-error-mean"]).iloc[-1]))<jupyter_output>train-error-mean  train-error-std  test-error-mean  test-error-std
0           0.28244         0.002134          0.28354        0.001930
1           0.26989         0.001944          0.27150        0.002149
2           0.25842         0.005334          0.26102        0.005348
3           0.25325         0.003673          0.25542        0.002431
4           0.25079         0.003725          0.25158        0.001407
0.7484200000000001<jupyter_text>**NOTE:****`cv_results`** stores the training and test mean and standard deviation of the error per boosting round (tree built) as a DataFrame. From **`cv_results`**, the final round **`'test-error-mean'`** is extracted and converted into an accuracy, where accuracy is **`1-error`**. The final accuracy of around 75% is an improvement from earlier!  Measuring AUCNow that we've used cross-validation to compute average out-of-sample accuracy (after converting from an error), it's very easy to compute any other metric we might be interested in. All we have to do is pass it (or a list of metrics) in as an argument to the metrics parameter of xgb.cv().We'll now compute another common metric used in binary classification - the area under the curve ("auc").<jupyter_code># Perform cross_validation: cv_results
cv_results = xgb.cv(dtrain=churn_dmatrix, params=params, 
                  nfold=3, num_boost_round=5, 
                  metrics="auc", as_pandas=True, seed=123)

# Print cv_results
print(cv_results)

# Print the AUC
print((cv_results["test-auc-mean"]).iloc[-1])<jupyter_output>train-auc-mean  train-auc-std  test-auc-mean  test-auc-std
0        0.768974       0.001915       0.767709      0.001880
1        0.790360       0.006924       0.788853      0.006686
2        0.813151       0.007752       0.811993      0.008464
3        0.821427       0.002913       0.820558      0.004087
4        0.826219       0.001374       0.825163      0.001548
0.8251633333333332<jupyter_text>Fantastic! This AUC valus is quite strong. As we have seen, XGBoost's learning API makes it very easy to compute any metric you may be interested in. ---  When should I use XGBoost[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch1_5.master.m3u8)- Use XGBoost for any supervised machine learning problem that has  - Have large number of training samples ($>1000$ samples and $<100$ features)  - No. of features $<$ Number of training samples  - Yuo have a mixture of categorical and numeric features or when you just have numeric features When to NOT use XGBoost-  Low dataset size. Also when number os samples < No. of features- Image recognition- Computer Vision- NLP problems| Scenario |  Use XGBoost? ||----------|---------------||Visualizing the similarity between stocks by comparing the time series of their historical prices relative to each other| **NO** ||Predicting whether a person will develop cancer using genetic data with millions of genes, 23 examples of genomes of people that didn't develop cancer, 3 genomes of people who wound up getting cancer| **NO**||Clustering documents into topics based on the terms used in them |**NO**||Predicting the likelihood that a given user will click an ad from a very large clickstream log with millions of users and their web interactions|**YES**|---  Part-2:  Regression with XGBoost[SLIDES](https://projector-video-pdf-converter.datacamp.com/3679/chapter2.pdf)  [VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/v3/hls-3679_ch2_1.master.m3u8) Regression Review- Outcome is real-valued Common regression metrics- Root mean square error (RMSE)- Mean absolute error (MAE) Root Mean Square Error (RMSE)- Calculating RMSE  - Find the error (= Actual - Predicted)  - Square the error values  - Mean of sqaured errors  - Square root the mean squared error- Treats positive and negative values equally- Tends to punish large differences between actual and predicted much more than smaller ones Mean Absolute Error (MAE)- Calculating MAE  - Calculate absolute errors (|Actual-Predicted|)  - Take the mean  Objective (loss) functions and base learners[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/v3/hls-3679_ch2_2.master.m3u8) Objective functions and why we use them- Quantifies how far off a prediction is from the actual result- Measures the difference between estimated and true values for some collection of data- GOAL: Find the model that yields the min. value of the loss function across all fo the data points Common loss functions used with XGBoost- **`reg:linear`:** used for regression problems- **`reg:logistic`:** used for binary classification problems, when we want just the decision, not the probabilities- **`binary:logistic`:** When we want the actual probability for binary classification instead of just decision Base learners and why we need them- XGBoost is an ensemble method. We combine the output from several weak learners to give the final prediction- Individual models = base learners- We want that base learners, when combined, create final output prediction that is **non-linear**- Base learners are weak models, that are only slightly better than randomly guessing the outcome- Each base-learner should be good at distinguishing or predicting different parts of the dataset- The slightly-better-than-chance predictions combine together to give a single very good prediction- **Two kinds of base learners can be used**  - **Tree**  - **Linear**   Example of using Tree-based base learner in XGBoost```Pythonimport xgboost as xgbfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_errorimport pandas as pdimport numpy as npboston_data = pd.read_csv("boston_housing.csv")X,y = boston_data.iloc[:,:-1], boston_data.iloc[:,-1]X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123) Creating XGBoost Regressor instancexg_reg = xgb.XGBoostRegressor(objective='reg:linear', n_estimators=10, seed=123) Fit the model on training setxg_reg.fit(X_train, y_train) Predict target values on test setpreds = xg_reg.predict(X_test) Compute RMSErmse = np.sqrt(mean_squared_error(y_test, preds)print("RMSE: {}".format(rmse))```  Example of using Linear base learner in XGBoost```Pythonimport xgboost as xgbfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_errorimport pandas as pdimport numpy as npboston_data = pd.read_csv("boston_housing.csv")X,y = boston_data.iloc[:,:-1], boston_data.iloc[:,-1]X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123) Converting trainign and testing dataset into DMatrix objectsDM_train = xgb.DMatrix(data=X_train, label=y_train)DM_test = xgb.DMatrix(data=X_test, label=y_test) Creating a paramter dictionaryparams = {"booster":"gblinear", "objective":"reg_linear"} Training the modelrg_reg = xgb.train(params=params, dtrain=DM_train, num_boost_round=10) Predicting values for the test setpreds = xgb.predict(DM_test) Compute RMSErmse = np.sqrt(mean_squared_error(y_test, preds)print("RMSE: {}".format(rmse))```  Decision trees as base learnerst's now time to build an XGBoost model to predict house prices - not in Boston, Massachusetts, but in Ames, Iowa!  Here, we'll use decision trees as base learners. By default, XGBoost uses trees as base learners, so we don't have to specify that we want to use trees here with **`booster="gbtree"`**<jupyter_code>import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
import xgboost as xgb
from sklearn.metrics import mean_squared_error

df = pd.read_csv("https://assets.datacamp.com/production/repositories/943/datasets/4dbcaee889ef06fb0763e4a8652a4c1f268359b2/ames_housing_trimmed_processed.csv")
df.head()<jupyter_output><empty_output><jupyter_text>**NOTE** We have both numeric (such as `LotArea`, `GrLivArea`) and categorical (such as `BsmtFullBath`, `Remodeled`, `OverallQual`) columns in our dataset.<jupyter_code># Splitting the dataset into train and test set
y = df['SalePrice']
X = df.drop('SalePrice', axis=1)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
                                                    random_state=123)

# Instantiate the XGBRegressor
xg_reg = xgb.XGBRegressor(n_estimators=10, objectibe='reg_linear', seed=123)
#  Note: We didn't have to specify booster="gbtree" as this is the default.

# Fitting the training set
xg_reg.fit(X_train, y_train)

# Predict values on test set
pred = xg_reg.predict(X_test)

# Computing RMSE
rmse = np.sqrt(mean_squared_error(y_test, pred))
print(">>RMSE: {}".format(rmse))<jupyter_output>[13:53:15] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
>>RMSE: 78847.40175843415<jupyter_text>Next, we'll train an XGBoost model using linear base learners and XGBoost's learning API, and see if it performs better than this or not.  XGBoost with linear base learnersNow that we've used trees as base models in XGBoost, let's use the other kind of base model that can be used with XGBoost - a linear learner. This model, although not as commonly used in XGBoost, allows us to create a regularized linear regression using XGBoost's powerful learning API. However, because it's uncommon, we'll have to use XGBoost's own non-scikit-learn compatible functions to build the model, such as `xgb.train()`.We'll use the same `Ames, Iowa housing prices dataset`<jupyter_code># Creating DMatrices for training and test sets
DM_train = xgb.DMatrix(data=X_train, label=y_train)
DM_test = xgb.DMatrix(data=X_test, label=y_test)

# Creating parameter dictionary
params = {'booster':'gblinear', 'objective':'reg:linear'}
 
# Instantiating XGBoost regressor and fitting the training set
xg_reg = xgb.train(params=params, dtrain=DM_train, num_boost_round=5)

# Predicting housing price values on test set
pred = xg_reg.predict(DM_test)

# Calculating RMSE
rmse = np.sqrt(mean_squared_error(y_test, pred))
print(">>RMSE: {}".format(rmse))<jupyter_output>[14:04:13] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
>>RMSE: 44331.64506083733<jupyter_text>Interesting - it looks like linear base learners performed better than tree-based base learner!------  Evaluating model quality- RMSE and MAEIt's now time to begin evaluating model quality.Here, we will compare the RMSE and MAE of a cross-validated XGBoost model on the Ames housing data.<jupyter_code>import pandas as pd
import numpy as np
import xgboost as xgb

df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/4dbcaee889ef06fb0763e4a8652a4c1f268359b2/ames_housing_trimmed_processed.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Using RMSE as a metric<jupyter_code>y = df['SalePrice']
X = df.drop('SalePrice', axis=1)

# Creating DMatrix
housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Create the parameter dictionary: params
params = {"objective":"reg:linear", "max_depth":4}

# Performing 4-fold corss validation with 5 boosting rounds as "rmse" as the metric
cv_results = xgb.cv(params=params, dtrain=housing_dmatrix, num_boost_round=5,
                    nfold=4, metrics='rmse', as_pandas=True, seed=123)<jupyter_output>[14:13:44] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[14:13:44] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[14:13:44] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[14:13:44] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>Carefully go through the paramters of the call to `xgb.cv()` method...Seeing what do we have in `cv_results`. We get 5 values because we performed 5 boosting rounds (each with 4-fold cross validation). Notice the train and test RMSE is going down with each boosting round.<jupyter_code>cv_results<jupyter_output><empty_output><jupyter_text>Using MAE as a metricNow performing the same evaluation with MAE instead of RMSE.<jupyter_code># Creating the DMatrix
housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Creating the paramete dictionary
params = {'objective':'reg:linear', 'max_depth':4}

# Performing 4-fold cross validation with 5 boosting rounds and 'mae' as metric
cv_results = xgb.cv(params=params, dtrain=housing_dmatrix, num_boost_round=5,
                    nfold=4, metrics='mae', as_pandas=True, seed=123)

cv_results<jupyter_output>[14:21:03] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[14:21:03] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[14:21:03] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[14:21:03] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>---  Regularization and base learners in XGBoost[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch2_3.master.m3u8)- Loss functions in XGBoost not only take into account how close a model's predictions are to the actual values, but also take into account how complex the model is. - **Regularization** is a control on model complexity- We want models that are both accurate as well as simple Regularization parameters in XGBoost- **`gamma`:** minimum loss reduction allowed for a split to occur. Higher values lead to fewer splits- **`alpha`:** Another name for L1 regularization. Puts L1 regularization penalty on leaf weights rather than feature weights (as in the case of linearor logistic regression). Larger value means stronger L1 regularization, causing many leaf weights in the base learner to go to `0`.- **`lambda`:** Another name for L2 regularization. Puts L2 regularization penalty on leaf weights. Causes leaf weights to smoothly decrease, much more smoothly than L1 penalty.  L1 regularization in XGBoost example```Pythonimport xgboost as xgbfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_errorimport pandas as pdimport numpy as npboston_data = pd.read_csv("boston_housing.csv")X,y = boston_data.iloc[:,:-1], boston_data.iloc[:,-1]boston_dmatrix = xgb.DMatrix(data=X, label=y)params = {'objective':'reg:linear', 'max_depth':4} Creating list of L1 (alpha) valuesl1_params = [1,10,100]rmses_l1 = []for reg in l1_params:  params['alpha'] = reg  cv_results = xgb.cv(dtrain=boston_dmatrix, params=params, nfold=4, num_boost_round=10, metrics='rmse', as_pandas=True, seed=123)  rmses_l1.append(cv_results['test-rmse-mean'].tail(1).values)  print(pd.DataFrame(list(zip(l1_params, rmses_l1)), columns=['l1','rmse']))```  Base learners in XGBoost- Linear base learner:  - Sum of linear terms (same as that in linear and logistic models)  - Boosted model is the weighted sum of all linear models (thus it itself is linear)  - Rarely used- Tree base learner  - Uses decision tree as base learner  - Boosted model is the weighted sum of decision trees (non-linear)  - Almost exclusively used in XGBoost    Using regularization in XGBoostHaving seen an example of l1 regularization in the video, we'll now vary the l2 regularization penalty - also known as "lambda" - and see its effect on overall model performance on the Ames housing dataset.<jupyter_code>import pandas as pd
import xgboost as xgb

df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/4dbcaee889ef06fb0763e4a8652a4c1f268359b2/ames_housing_trimmed_processed.csv')
y = df['SalePrice']
X = df.drop('SalePrice', axis=1)

# Creating DMatrix
housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Creating list of values to be used for lambda (L2 regularization strength)
reg_params = [1, 10, 100]

# Create the initial parameter dictionary for varying l2 strength: params
params = {"objective":"reg:linear", "max_depth":3}

# Create an empty list for storing rmses as a function of l2 complexity
rmses_l2 = []

# Iterate over reg_params
for reg in reg_params:

    # Update l2 strength
    params["lambda"] = reg
    
    # Pass this updated param dictionary into cv
    cv_results_rmse = xgb.cv(dtrain=housing_dmatrix, params=params, nfold=2, num_boost_round=5, metrics="rmse", as_pandas=True, seed=123)
    
    # Append best rmse (final round) to rmses_l2
    rmses_l2.append(cv_results_rmse["test-rmse-mean"].tail(1).values[0])

# Look at best rmse per l2 param
print("Best rmse as a function of l2:")
print(pd.DataFrame(list(zip(reg_params, rmses_l2)), columns=["l2", "rmse"]))<jupyter_output>[16:22:14] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[16:22:14] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[16:22:14] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[16:22:14] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[16:22:14] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[16:22:14] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
Best rmse as a function of l2:
    l2          rmse
0    1  52275.355469
1   10  57746.060547
2  100  76624.617188<jupyter_text>Visualizing individual XGBoost treesNow that we've used XGBoost to both build and evaluate regression as well as classification models, we'll see how to visually explore your models. Here, we will visualize individual trees from the fully boosted model that XGBoost creates using the entire housing dataset.XGBoost has a `plot_tree()` function that makes this type of visualization easy. Once we train a model using the XGBoost learning API, we can pass it to the `plot_tree()` function along with the number of trees you want to plot using the `num_trees argument`.<jupyter_code>import matplotlib.pyplot as plt
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 14,6

# Create the DMatrix: housing_dmatrix
housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Create the parameter dictionary: params
params = {"objective":"reg:linear", "max_depth":2}

# Train the model: xg_reg
xg_reg = xgb.train(params=params, dtrain=housing_dmatrix, num_boost_round=10)

# Plot the first tree
xgb.plot_tree(xg_reg, num_trees=0)
plt.show()

# Plot the fifth tree
xgb.plot_tree(xg_reg, num_trees=4)
plt.show()

# Plot the last tree sideways
xgb.plot_tree(xg_reg, num_trees=9, rankdir='LR')
fig.show()<jupyter_output>[17:32:26] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>The plots provide insight into how the model arrived at its final decisions and what splits it made to arrive at those decisions. This allows us to identify which features are the most important in determining house price.----  Visualizing feature importances: What features are most important in my datasetAnother way to visualize your XGBoost models is to examine the importance of each feature column in the original dataset within the model.One simple way of doing this involves counting the number of times each feature is split on across all boosting rounds (trees) in the model, and then visualizing the result as a bar graph, with the features ordered according to how many times they appear. XGBoost has a `plot_importance()` function that allows us to do exactly this.<jupyter_code># Create the DMatrix: housing_dmatrix
housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Create the parameter dictionary: params
params = {'objective':'reg:linear', 'max_depth':4}

# Train the model: xg_reg
xg_reg = xgb.train(dtrain=housing_dmatrix, params=params, num_boost_round=10)

# Plot the feature importances
xgb.plot_importance(booster=xg_reg)
plt.show()<jupyter_output>[17:32:45] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>Brilliant! It looks like `GrLivArea` is the most important feature.------  Part-3: Fine-tuning XGBoost model Why tune your model[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch3_1.master.m3u8) Tuning the number of boosting roundsLet's start with parameter tuning by seeing how the number of boosting rounds (number of trees we build) impacts the out-of-sample performance of your XGBoost model. We'll use `xgb.cv()` inside a for loop and build one model per `num_boost_round parameter`.<jupyter_code>import pandas as pd
import numpy as np
import xgboost as xgb

df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/4dbcaee889ef06fb0763e4a8652a4c1f268359b2/ames_housing_trimmed_processed.csv')
df.head()
y = df['SalePrice']
X = df.drop('SalePrice', axis=1)

# Create the DMatrix
housing_dmatrix = xgb.DMatrix(data=X,label=y)

# Create the parameter dictionary for each tree
params = {"objective":"reg:linear", "max_depth":3}

# Create list of number of boosting rounds
num_rounds = [5, 10, 15]

# Empty list to store final round rmse per XGBoost model
final_rmse_per_round = []

# Iterate over num_rounds and build one model per num_boost_round parameter
for curr_num_rounds in num_rounds:

    # Perform cross-validation
    cv_results = xgb.cv(dtrain=housing_dmatrix, params=params, nfold=3, num_boost_round=curr_num_rounds, metrics="rmse", as_pandas=True, seed=123)
    
    # Append final round RMSE
    final_rmse_per_round.append(cv_results["test-rmse-mean"].tail().values[-1])

# Print the resultant DataFrame
num_rounds_rmses = list(zip(num_rounds, final_rmse_per_round))
print(pd.DataFrame(num_rounds_rmses,columns=["num_boosting_rounds","rmse"]))<jupyter_output>[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of re[...]<jupyter_text>Awesome! As we can see, increasing the number of boosting rounds decreases the RMSE.  Automated boosting round selection using early_stoppingNow, instead of attempting to cherry pick the best possible number of boosting rounds, we can very easily have XGBoost automatically select the number of boosting rounds for us within `xgb.cv()`. This is done using a technique called **early stopping**.Early stopping works by testing the XGBoost model after every boosting round against a hold-out dataset and stopping the creation of additional boosting rounds (thereby finishing training of the model early) if the hold-out metric (`"rmse"` in our case) does not improve for a given number of rounds. Here we will use the `early_stopping_rounds` parameter in `xgb.cv()` with a large possible number of boosting rounds (`50`). Bear in mind that if the holdout metric continuously improves up through when `num_boost_rounds` is reached, then early stopping does not occur.<jupyter_code># Create your housing DMatrix
housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Create the parameter dictionary for each tree
params = {"objective":"reg:linear", "max_depth":4}

# Perform cross-validation with early stopping
cv_results = xgb.cv(nfold=3, metrics='rmse',num_boost_round=50,
                    early_stopping_rounds=10, seed=123, as_pandas=True, 
                    dtrain=housing_dmatrix, params=params)

# Print cv_results
print(cv_results)<jupyter_output>[04:05:59] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:06:00] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:06:00] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
    train-rmse-mean  train-rmse-std  test-rmse-mean  test-rmse-std
0     141871.619792      403.625044   142640.651042     705.559164
1     103057.020833       73.773330   104907.661458     111.116005
2      75975.960937      253.731460    79262.057292     563.761707
3      57420.535156      521.648794    61620.139323    1087.688565
4      44552.957031      544.167830    50437.565104    1846.445052
5      35763.953125      681.798925    43035.660156    2034.472841
6      29861.465495      769.569137    38600.881510    2169.801278
7      25994.675781      756.523981    36071.817708    2109.795546
8      23306.836[...]<jupyter_text>Overview of XGBoost's hyperparameters Tunable parameters in XGBoost- Paramters that can be tuned differ significantly depending on then type of base learner used For tree base learner- **`learning_rate`:** Controls how quickly the model fits the residual error using additional base learners. Low learning rate will require more boosting rounds to achieve the same reduction in residual error as compared to an XGBoost model with a high learning rate- **`gamma`:** Min. loss reduction to create new tree split- **`alpha`:** L1 reg on leaf weights- **`lambda`:** L2 reg on leaf weights- **`max_depth`:** A positive integer value indicating how deeply each tree is allowed to go in any given boosting round- **`subsample`:**% A value between 0 and 1, indicating the fraction of the total training set that can be used for any given boosting round. If the value of low, then the fraction of training data used in any given boosting round would be low, and we may run into underfitting problems. Similarly, a high value can lead to overfitting as well.- **`colsample_by_tree`:** % features used per tree. Value between 0 and 1. Large value means that almost all features will be used to build a tree during a given boosting round. A small value means that the fraction of features that can be selected is very small. Smaller `col_samples_by_tree` can be considered as providing additional regularization to the model. On the contrary using all the columns, may lead to overfitting of a trained model. For linear base learner- **`alpha`:** L1 reg on leaf weights- **`lambda`:** L2 reg on leaf weights- **`lambda_bias`:** L2 reg term to be applied to model's bias  .- In either of types of base learners, the number of estimators (or number of base learners) used is itself a tunable parameter  Tuning etaIt's time to practice tuning other XGBoost hyperparameters in earnest and observing their effect on model performance! We'll begin by tuning the `"eta"`, also known as the learning rate.The learning rate in XGBoost is a parameter that can range between `0` and `1`, with higher values of `"eta"` penalizing feature weights more strongly, causing much stronger regularization.<jupyter_code># Creating DMAtrix for the dataset
housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Creating params dictionary
params = {'objective':'reg:linear', 'max_depth':3}

# Creating a list of learning rate eta values and empty list to store final round rmse per xgboost
eta_vals = [0.001, 0.01, 0.1]
best_rmse = []

for eta_val in eta_vals:
  params['eta'] = eta_val

  # Fitting the data
  cv_results = xgb.cv(params=params, dtrain=housing_dmatrix, num_boost_round=10, 
                      nfold=3, early_stopping_rounds=5, metrics='rmse', 
                      as_pandas=True, seed=123)
  
  # Appending the final rounf rmse to best_rmse list
  best_rmse.append(cv_results['test-rmse-mean'].tail(1).values[-1])

# Print the resultant DataFrame
print(pd.DataFrame(list(zip(eta_vals, best_rmse)), columns=["eta","best_rmse"]))<jupyter_output>[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[04:23:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of re[...]<jupyter_text>Tuning max_depthIn this exercise, our job is to tune `max_depth`, which is the parameter that dictates the maximum depth that each tree in a boosting round can grow to. Smaller values will lead to shallower trees, and larger values to deeper trees.<jupyter_code>housing_dmatrix = xgb.DMatrix(data=X, label=y)

# Defining the parameter dictionary
params = {'objective':'reg:linear'}

# List of values to try out for max_depth
max_depths = [2,5,10,20]
best_rmse = []

for max_depth in max_depths:
  params['max_depth'] = max_depth

  cv_results = xgb.cv(params=params, dtrain=housing_dmatrix, num_boost_round=10,
                      nfold=2, early_stopping_rounds=5, metrics='rmse',
                      as_pandas=True, seed=123)
  
  # Last value of RMSE obtained from cross validation
  best_rmse.append(cv_results['test-rmse-mean'].tail(1).values[-1])

# Print the resultant DataFrame
print(pd.DataFrame(list(zip(max_depths, best_rmse)),columns=["max_depth","best_rmse"]))<jupyter_output>[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:02:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of re[...]<jupyter_text>Tuning `colsample_bytree`Now, it's time to tune `"colsample_bytree"`. In scikit-learn's `RandomForestClassifier` or `RandomForestRegressor`, where it is called `max_features`. In both `xgboost` and `sklearn`, this parameter (although named differently) simply specifies the fraction of features to choose from at every split in a given tree. In xgboost, `colsample_bytree` must be specified as a float between `0` and `1`.<jupyter_code># Create your housing DMatrix
housing_dmatrix = xgb.DMatrix(data=X,label=y)

# Create the parameter dictionary
params={"objective":"reg:linear","max_depth":3}

# Create list of hyperparameter values: colsample_bytree_vals
colsample_bytree_vals = [0.1, 0.5, 0.8, 1]
best_rmse = []

# Systematically vary the hyperparameter value 
for colsample_bytree_val in colsample_bytree_vals:

    params['colsample_bytree'] = colsample_bytree_val
    
    # Perform cross-validation
    cv_results = xgb.cv(dtrain=housing_dmatrix, params=params, nfold=2,
                 num_boost_round=10, early_stopping_rounds=5,
                 metrics="rmse", as_pandas=True, seed=123)
    
    # Append the final round rmse to best_rmse
    best_rmse.append(cv_results["test-rmse-mean"].tail().values[-1])

# Print the resultant DataFrame
print(pd.DataFrame(list(zip(colsample_bytree_vals, best_rmse)), columns=["colsample_bytree","best_rmse"]))<jupyter_output>[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[08:15:56] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of re[...]<jupyter_text>There are several other individual parameters that you can tune, such as **`subsample`**, which dictates the fraction of the training data that is used during any given boosting round. --- Review of grid search and random search[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch3_4.master.m3u8) Gris Search- Method of searching exhaustively over a given set of hyperparameters, once per set of hyperparameters- Number of models = Number of distinct values per hyperparameter multiplied across each hyperparameter```Pythonimport pandas as pdimport numpy as npimport xgboost as xgbfrom sklearn.model_selection import GridSearchCVX, y = housing_data[housing_data.columns.tolist()[:-1]], housing_data[housing_data.columns.tolist()[-1]])housing_dmatrix = xgb.DMatrix(data=X, label=y)gbm_param_grid = {'learning_rate': [0.01, 0.1, 0.5, 0.9], 'nestimators': [200], 'subsample': [0.3, 0.5, 0.9]} 12 different models will be trainedgbm = xgb.XGBoostRegressor()grid_mse = GridSearchCV(estimator=gbm, param_grid=gbm_param_grid, scoring='neg_mean_squared_error', cv=4, verbose=1)grid_mse.fit(X, y)print("Best parameters:", grid_mse.best_params_)print("Lowest RMSE found:", np.sqrt(np.abs(grid_mse.best_score_))```  Random search- Unlike grid search, the number of models required to be iterated over does not grow as we expand the overall hyperparameter space- We set the number of iterations we'd like our random search to continue- During each iteration, randomly draw a value in the range of specified values for each hyperparameter searched over and train/evaluate a model with those hyperparameters- After we have reached maximum number of iterations, select the hyperparameter configuration with the best evaluated score```Pythonimport pandas as pdimport numpy as npimport xgboost as xgbfrom sklearn.model_selection import GridSearchCVX, y = housing_data[housing_data.columns.tolist()[:-1]], housing_data[housing_data.columns.tolist()[-1]])housing_dmatrix = xgb.DMatrix(data=X, label=y)gbm_param_grid = {'learning_rate':np.arange(0.05, 1.05, 0.05), 'nestimators': [200], 'subsample': np.arange(0.05, 1.05, 0.05)} With this parameter space, there would be 400 models if we do a grid searchgbm = xgb.XGBRegressor()randomized_mse = RandomizedSearchCV(estimator=gbm, param_distributions=gbm_param_grid, n_iter=25, scoring='neg_mean_squared_error', cv=4, verbose=1)randomized_mse.fit(X, y)print("Best parameters:", randomized_mse.best_params_)print("Lowest RMSE found:", np.sqrt(np.abs(randomized_mse.best_score_))```  Grid search with XGBoostNow that we've learned how to tune parameters individually with XGBoost, let's take our parameter tuning to the next level by using scikit-learn's GridSearch and RandomizedSearch capabilities with internal cross-validation using the `GridSearchCV` and `RandomizedSearchCV` functions. We will use these to find the best model exhaustively from a collection of possible parameter values across multiple parameters simultaneously.<jupyter_code>from sklearn.model_selection import GridSearchCV 

# Create the parameter grid: gbm_param_grid
gbm_param_grid = {
    'colsample_bytree': [0.3, 0.7],
    'n_estimators': [50],
    'max_depth': [2, 5]
}

# Instantiate the regressor: gbm
gbm = xgb.XGBRegressor()

# Perform grid search: grid_mse
grid_mse = GridSearchCV(param_grid=gbm_param_grid, estimator=gbm, scoring='neg_mean_squared_error', verbose=1, cv=4)


# Fit grid_mse to the data
grid_mse.fit(X, y)

# Print the best parameters and lowest RMSE
print("Best parameters found: ", grid_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(grid_mse.best_score_)))<jupyter_output>Fitting 4 folds for each of 4 candidates, totalling 16 fits
[10:15:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[10:15:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[10:15:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[10:15:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>Random search with XGBoostOften, `GridSearchCV` can be really time consuming, so in practice, we may want to use `RandomizedSearchCV` instead, as we will do in this exercise. The good news is we only have to make a few modifications to our `GridSearchCV` code to do `RandomizedSearchCV`. The key difference is we have to specify a `param_distributions` parameter instead of a `param_grid` parameter.<jupyter_code>from sklearn.model_selection import RandomizedSearchCV

# Create the parameter grid: gbm_param_grid 
gbm_param_grid = {
    'n_estimators': [25],
    'max_depth': np.arange(2, 12)
}

# Instantiate the regressor: gbm
gbm = xgb.XGBRegressor(n_estimators=10)

# Perform random search: grid_mse
randomized_mse = RandomizedSearchCV(param_distributions = gbm_param_grid,
estimator=gbm, scoring='neg_mean_squared_error', n_iter=5, cv=4, verbose=1)


# Fit randomized_mse to the data
randomized_mse.fit(X, y)

# Print the best parameters and lowest RMSE
print("Best parameters found: ", randomized_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(randomized_mse.best_score_)))<jupyter_output>Fitting 4 folds for each of 5 candidates, totalling 20 fits
[10:47:58] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>--- Limits of grid search and random search[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch3_5.master.m3u8) Grid Search- No. of models that train grow significantly with any additional new paramter, unlike in Random Search where we specify an upper limit on the number of iterations search would take place, and so time it takes to run random search won't explode as we add additional parameters or values. Random Search- Parameter space to explore can be massive- The search space size can be massive for Grid Search in certain cases, whereas for Random Search the number of hyperparameters has a significant effect on how long it takes to run. --------  Part-4: Using XGBoost in pipelines[SLIDES](https://projector-video-pdf-converter.datacamp.com/3679/chapter4.pdf) Review of sklearn pipelines[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb2/hls-ch4_1.master.m3u8)- Pipelines take in list of named 2-tuples (name, pipeline_step) as input- Tuples can contain any arbitary scikit-learn compatible estimator or transformer object- Steps are executed in the order they are passed- Pipeline implements fit(), predict() methods- Pipelines can be used as input estimator objects into grid/randomized search and cross_val_score methods Working with unprocessed Ames housing prices dataset<jupyter_code>import pandas as pd

df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/17a7c5c0acd7bfa253827ea53646cf0db7d39649/ames_unprocessed_data.csv')
df.head()<jupyter_output><empty_output><jupyter_text>**Checking if the dataframe has missing values**<jupyter_code>df.isna().sum()<jupyter_output><empty_output><jupyter_text>`LotFrontage` column has several missing values  Encoding categorical columns I: LabelEncoderNow that we've seen what will need to be done to get the housing data ready for XGBoost, let's go through the process step-by-step.First, we will need to fill in missing values - as we saw previously, the column `LotFrontage` has many missing values. Then, we will need to encode any categorical columns in the dataset using one-hot encoding so that they are encoded numerically.The data has five categorical columns: `MSZoning`, `PavedDrive`, `Neighborhood`, `BldgType`, and `HouseStyle`. Scikit-learn has a `LabelEncoder` function that converts the values in each categorical column into integers.<jupyter_code># Import LabelEncoder
from sklearn.preprocessing import LabelEncoder

# Fill missing values with 0
df.LotFrontage = df.LotFrontage.fillna(0)

# Create a boolean mask for categorical columns
categorical_mask = (df.dtypes == object)

# Get list of categorical column names
categorical_columns = df.columns[categorical_mask].tolist()

# Print the head of the categorical columns
print(df[categorical_columns].head())

# Create LabelEncoder object: le
le = LabelEncoder()

# Apply LabelEncoder to categorical columns
df[categorical_columns] = df[categorical_columns].apply(lambda x: le.fit_transform(x))

# Print the head of the LabelEncoded categorical columns
print(df[categorical_columns].head())<jupyter_output>MSZoning Neighborhood BldgType HouseStyle PavedDrive
0       RL      CollgCr     1Fam     2Story          Y
1       RL      Veenker     1Fam     1Story          Y
2       RL      CollgCr     1Fam     2Story          Y
3       RL      Crawfor     1Fam     2Story          Y
4       RL      NoRidge     1Fam     2Story          Y
   MSZoning  Neighborhood  BldgType  HouseStyle  PavedDrive
0         3             5         0           5           2
1         3            24         0           2           2
2         3             5         0           5           2
3         3             6         0           5           2
4         3            15         0           5           2<jupyter_text>Notice how the entries in each categorical column are now encoded numerically. A `BldgTpe` of `1Fam` is encoded as `0`, while a `HouseStyle` of `2Story` is encoded as `5`.  Encoding categorical columns II: OneHotEncoderOkay - so we have your categorical columns encoded numerically. Can we now move onto using pipelines and XGBoost? Not yet! In the categorical columns of this dataset, there is no natural ordering between the entries. As an example: Using `LabelEncoder`, the `CollgCr` `Neighborhood` was encoded as `5`, while the `Veenker` `Neighborhood` was encoded as `24`, and `Crawfor` as `6`. Is Veenker "greater" than Crawfor and CollgCr? No - and allowing the model to assume this natural ordering may result in poor performance.As a result, there is another step needed: We have to apply a one-hot encoding to create binary, or "dummy" variables. We can do this using scikit-learn's OneHotEncoder.<jupyter_code>import pandas as pd

df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/17a7c5c0acd7bfa253827ea53646cf0db7d39649/ames_unprocessed_data.csv')
print(df.info())
df.head()
# Import OneHotEncoder
from sklearn.preprocessing import OneHotEncoder

# Print the shape of the original DataFrame
print(df.shape)

# Create OneHotEncoder: ohe
ohe = OneHotEncoder(sparse=False)

# Apply OneHotEncoder to categorical columns - output is no longer a dataframe
categorical_columns = df.columns[df.dtypes == object].values
df_encoded = pd.DataFrame(ohe.fit_transform(df[categorical_columns]))
df_encoded.columns = ohe.get_feature_names()

# Removing categorical columns from original df
df.drop(categorical_columns, axis=1, inplace=True)

# Adding one hot encoded columns to the df
df = pd.concat([df, df_encoded], axis=1)

# Shape of one hot encoded array
print(df.shape)<jupyter_output>(1460, 21)
(1460, 62)<jupyter_text>As you can see, after one hot encoding, which creates binary variables out of the categorical variables, there are now 62 columns  Encoding categorical columns III: DictVectorizerAlright, one final trick before we dive into pipelines. The two step process we just went through - `LabelEncoder` followed by `OneHotEncoder` - can be simplified by using a `DictVectorizer`.Using a `DictVectorizer` on a DataFrame that has been converted to a dictionary allows us to get label encoding as well as one-hot encoding in one go.<jupyter_code># Import DictVectorizer
from sklearn.feature_extraction import DictVectorizer
import pandas as pd

# Reading dataset
df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/17a7c5c0acd7bfa253827ea53646cf0db7d39649/ames_unprocessed_data.csv')

# Convert df into a dictionary
df_dict = df.to_dict('records')

# Create the DictVectorizer object
dv = DictVectorizer(sparse=False)

# Apply dv on df
df_encoded = dv.fit_transform(df_dict)

# Print the resulting first five rows
print(df_encoded[:5,:])

# Print the vocabulary
print(dv.vocabulary_)<jupyter_output>[[3.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 1.000e+00
  0.000e+00 0.000e+00 2.000e+00 5.480e+02 1.710e+03 1.000e+00 0.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 1.000e+00 0.000e+00 0.000e+00
  8.450e+03 6.500e+01 6.000e+01 0.000e+00 0.000e+00 0.000e+00 1.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 1.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 5.000e+00 7.000e+00
  0.000e+00 0.000e+00 1.000e+00 0.000e+00 2.085e+05 2.003e+03]
 [3.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
  1.000e+00 1.000e+00 2.000e+00 4.600e+02 1.262e+03 0.000e+00 0.000e+00
  0.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
  9.600e+03 8.000e+01 2.000e+01 0.000e+00 0.000e+00 0.000e+00 1.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
 [...]<jupyter_text>Besides simplifying the process into one step, `DictVectorizer` has useful attributes such as `vocabulary_` which maps the names of the features to their indices.---  Preprocessing within a pipelineNow that we've seen what steps need to be taken individually to properly process the Ames housing data, let's use the much cleaner and more succinct `DictVectorizer` approach and put it alongside an `XGBoostRegressor` inside of a scikit-learn pipeline.<jupyter_code># Import DictVectorizer
from sklearn.feature_extraction import DictVectorizer
import pandas as pd

# Reading dataset
df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/17a7c5c0acd7bfa253827ea53646cf0db7d39649/ames_unprocessed_data.csv')
df.head()
# Setting the target variable aside 
y = df['SalePrice']

# Removing the target variable
X = df.drop('SalePrice', axis=1)
# Import necessary modules
from sklearn.feature_extraction import DictVectorizer
from sklearn.pipeline import Pipeline
import xgboost as xgb

# Fill LotFrontage missing values with 0
X.LotFrontage = X.LotFrontage.fillna(0)

# Setup the pipeline steps: steps
steps = [("ohe_onestep", DictVectorizer(sparse=False)),
         ("xgb_model", xgb.XGBRegressor())]

# Create the pipeline: xgb_pipeline
xgb_pipeline = Pipeline(steps)

# Fit the pipeline
xgb_pipeline.fit(X.to_dict('records'), y)<jupyter_output>[05:47:01] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.<jupyter_text>Incorporating XGBoost into pipelines[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch4_2.master.m3u8)  Cross-validating XGBoost modelIn this exercise, we'll go one step further by using the pipeline we've created to preprocess and cross-validate our model.<jupyter_code># Import necessary modules
from sklearn.feature_extraction import DictVectorizer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
import pandas as pd
import numpy as np

# Reading dataset
df = pd.read_csv('https://assets.datacamp.com/production/repositories/943/datasets/17a7c5c0acd7bfa253827ea53646cf0db7d39649/ames_unprocessed_data.csv')
y = df['SalePrice']
X = df.drop('SalePrice', axis=1)

# Fill LotFrontage missing values with 0
X.LotFrontage = X.LotFrontage.fillna(0)

# Setup the pipeline steps: steps
steps = [("ohe_onestep", DictVectorizer(sparse=False)),
         ("xgb_model", xgb.XGBRegressor(max_depth=2, objective="reg:linear"))]

# Create the pipeline: xgb_pipeline
xgb_pipeline = Pipeline(steps)

# Cross-validate the model
cross_val_scores = cross_val_score(xgb_pipeline, X.to_dict('records'), y, cv=10, scoring='neg_mean_squared_error')

# Print the 10-fold RMSE
print("10-fold RMSE: ", np.mean(np.sqrt(np.abs(cross_val_scores))))<jupyter_output>[05:56:01] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[05:56:02] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[05:56:02] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[05:56:02] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[05:56:02] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[05:56:02] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[05:56:02] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[05:56:03] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of re[...]<jupyter_text>--- Kidney disease case study I: Categorical ImputerWe'll now continue our exploration of using pipelines with a dataset that requires significantly more wrangling. The chronic kidney disease dataset contains both categorical and numeric features, but contains lots of missing values. The **goal** here is to predict who has chronic kidney disease given various blood indicators as features.We'll use a new library, `sklearn_pandas`, that allows us to chain many more processing steps inside of a pipeline than are currently supported in scikit-learn. Specifically, we'll be able to impute missing categorical values directly using the `Categorical_Imputer()` class in `sklearn_pandas`, and the `DataFrameMapper()` class to apply any arbitrary sklearn-compatible transformer on DataFrame columns, where the resulting output can be either a NumPy array or DataFrame.We've also created a transformer called a Dictifier that encapsulates converting a DataFrame using .to_dict("records") without you having to do it explicitly (and so that it works in a pipeline). Finally, we've also provided the list of feature names in kidney_feature_names, the target name in kidney_target_name, the features in X, and the target in y.In this exercise, your task is to apply the CategoricalImputer to impute all of the categorical columns in the dataset. You can refer to how the numeric imputation mapper was created as a template. Notice the keyword arguments input_df=True and df_out=True? This is so that you can work with DataFrames instead of arrays. By default, the transformers are passed a numpy array of the selected columns as input, and as a result, the output of the DataFrame mapper is also an array. Scikit-learn transformers have historically been designed to work with numpy arrays, not pandas DataFrames, even though their basic indexing interfaces are similar.<jupyter_code>import pandas as pd
import numpy as np

df = pd.read_csv("https://assets.datacamp.com/production/repositories/943/datasets/82c231cd41f92325cf33b78aaa360824e6b599b9/chronic_kidney_disease.csv", 
                 names=['age','bp','sg','al','su','rbc','pc','pcc','ba','bgr','bu','sc','sod','pot','hemo','pcv','wc','rc','htn','dm','cad','appet','pe','ane', 'label'],
                 na_values='?')
print(df.shape)
df.head()
# Replace label values with 0 (ckd) and 1
df['label'].replace({'ckd':0, 'notckd':1}, inplace=True)

# Separating training data and labels from DataFrame
X, y = df.iloc[:, :-1], df['label'].values
# Import necessary modules
from sklearn_pandas import DataFrameMapper, CategoricalImputer
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import FeatureUnion
from sklearn.model_selection import cross_val_score, RandomizedSearchCV
from sklearn.base import BaseEstimator, TransformerMixin
import xgboost as xgb
# Create a boolean mask for categorical columns
categorical_feature_mask = X.dtypes == object

# Get a list of categorical column names
categorical_columns = X.columns[categorical_feature_mask].tolist()

# Get a list of non-categorical column names
non_categorical_columns = X.columns[~categorical_feature_mask].tolist()
# Create empty list to hold column imputers: transformers
transformers = []

# Create numeric imputers and add to list of transformers
transformers.extend([([num_col], [SimpleImputer(strategy='median'),
                                                 StandardScaler()]) for num_col
                    in non_categorical_columns])

# Create categorical imputers and add to list of transformers
transformers.extend([(cat_col, [CategoricalImputer()]) for cat_col in
                    categorical_columns])

# Use list of transformers to create a DataFrameMapper object
numeric_categorical_union = DataFrameMapper(transformers, input_df=True,
                                            df_out=True)
# Define Dictifier class to turn df into dictionary as part of pipeline
class Dictifier(BaseEstimator, TransformerMixin):       
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return X.to_dict('records')
# Create full pipeline
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction import DictVectorizer

pipeline = Pipeline([('featureunion', numeric_categorical_union),
                    ('dictifier', Dictifier()),
                    ('vectorizer', DictVectorizer(sort=False)),
                    ('clf', xgb.XGBClassifier(max_depth=3))])

# Perform cross-validation
cross_val_scores = cross_val_score(pipeline, X, y, scoring='roc_auc', cv=3)<jupyter_output><empty_output><jupyter_text>Tuning XGBoost hyperparameters[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch4_3.master.m3u8) Tuning hyperparameters on Boston Housing prices dataset<jupyter_code>import pandas as pd
from sklearn.datasets import load_boston
import numpy as np
import xgboost as xgb
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import RandomizedSearchCV
X, y = load_boston(return_X_y=True)
# Creating pipeline
xgb_pipeline = Pipeline([('st_scaler', StandardScaler()), 
                         ('xgb_model', xgb.XGBRegressor())])

# Creating parameter grid
gbm_param_grid = {
    'xgb_model__subsample' : np.arange(0.05, 1, 0.05),
    'xgb_model__max_depth' : np.arange(3, 20, 1),
    'xgb_model__colsample_bytree' : np.arange(0.1, 1.05, 0.05)
}

randomized_neg_mse = RandomizedSearchCV(estimator=xgb_pipeline, n_iter=10,
                                        param_distributions=gbm_param_grid,
                                        scoring='neg_mean_squared_error', cv=4)

randomized_neg_mse.fit(X, y)

# Printing best RMSE
print('Best RMSE:', np.sqrt(np.abs(randomized_neg_mse.best_score_)))

# Printing best model parameters
print('Best model', randomized_neg_mse.best_estimator_)<jupyter_output>[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[02:37:33] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of re[...]<jupyter_text>Final Thoughts[VIDEO](https://videos.datacamp.com/transcoded/3679_xgboost/wb1/hls-ch4_4.master.m3u8) Things covered in this tutorial- Using XGBoost for classification tasks- Using XGBoost for regression tasks- Tuning XGBoost model's most important hyperparameters- Incorporating XGBoost into pipelines Things not covered in this tutorial- Using XGBoost for ranking and recommendation problems (Netflix, Amazon)  - This can be done by simply modifying the loss function we used previously- More sofisticated hyperparameter selection strategies for tuning XGBoost models (Bayesian Optimization)- Using XGBoost in an ensemble with other models for regression/classification**---THE END---**------<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09033065913425839, 'fraction_numerical': 0.04394228389219726, 'mean_word_length': 5.016819973718791, 'pattern_counts': {'":': 16, '<': 120, '<?xml version=': 0, '>': 129, 'https://': 32, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '306282', 'license': 'MIT', 'n_tokens_mistral': 23578, 'n_tokens_neox': 21115, 'n_words': 7141, 'path': 'Extreme_Gradient_Boosting_with_XGBoost.ipynb', 'repo_name': 'pratikgujral/Machine-Learning-Scientist-With-Python', 'size': '68681'}