<jupyter_start><jupyter_text>Classification by XGBRegressor ![Creative Commons License](https://i.creativecommons.org/l/by/4.0/88x31.png)  This work by <NAME> and <NAME> is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Introduction to House Sales in King County, USA dataset This dataset consists of Price of Houses in King County, Washington from sales between May 2014 and May 2015.The data is from Kaggle ([House Sales in King County, USA](https://www.kaggle.com/harlfoxem/housesalesprediction)) Kaggle contains various data sets.  Once you created a free account,  you may have access to and download them. Once you have downloaded the data set  you may import it by  `df = pd.read_csv('kc_house_data.csv')`.<jupyter_code>data = pd.read_csv('kc_house_data.csv')
print(data.shape)
data.head()<jupyter_output>(21613, 21)<jupyter_text>This is not a good example of data analysis  since I cannot find the reliable descriptions about the features.  (Let me know if you know the source/details of this data set.)For the illustration purposes,  we have a reasonable guess for the meanning of each column.<jupyter_code>data.columns<jupyter_output><empty_output><jupyter_text>- **price**: Price of each home sold  - bedrooms: numbers of bedrooms  - bathrooms: number of bathrooms (containing non-integer entries?)  - sqft_living: square footage of ???  - sqft_lot: square footage of ???- floors: numbers of floors (containing non-integer entries)  - waterfront: whether it has waterfront or not- view: nice view (0~4)  - condition: the condition of the apartment (1~5)  - grade: (1~13) - sqft_above: square footage of space above ground  - sqft_basement: square footage of the basement- yr_built: year built  - yr_renovated: year last renovated  - zipcode: zipcode (categorial)  - lat: lattitude  - long: longitude  - sqft_living15: square footage of ???  - sqft_lot15: square footage of ??? Use `np.unique([array[, return_counts=True)`  to check the occurence of a values in a columns.<jupyter_code>np.unique(data.lat, return_counts=True)<jupyter_output><empty_output><jupyter_text>Replace `yr_built` by the house age by the time it was sold.  Replace `yr_renovated` by the age after renovated by the time it was sold.<jupyter_code>data = pd.read_csv('kc_house_data.csv')
data['yr_built'] = data['date'].str[:4].astype(int) - data['yr_built']
mask = data['yr_renovated'] > 0
data.loc[mask, 'yr_renovated'] = data.loc[mask, 'date'].str[:4].astype(int) - data.loc[mask, 'yr_renovated']
data.head()<jupyter_output><empty_output><jupyter_text>Take `X` as all the columns other than `id`, `date`, and `price`.  Take `y` as the column of `price`.<jupyter_code>X = data.iloc[:, 3:]
y = data.iloc[:, 2]<jupyter_output><empty_output><jupyter_text>Split the data.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y)
print(X_train.shape)
print(X_test.shape)<jupyter_output>(16209, 18)
(5404, 18)<jupyter_text>XGBRegressor model See [Introduction-to-XGBClassifier](https://github.com/jephianlin/ModularPython/blob/master/Introduction-to-XGBClassifier.ipynb) for  more information about XGBoost.<jupyter_code>import xgboost as xgb
model1 = xgb.XGBRegressor()
model1.fit(X_train, y_train)
y_model1 = model1.predict(X_test)<jupyter_output><empty_output><jupyter_text>Use `mean_squared_error` to evaluate the performance.<jupyter_code>from sklearn.metrics import mean_squared_error

mean_squared_error(y_test, y_model1) / y_test.shape[0]<jupyter_output><empty_output><jupyter_text>Hyperparameters of XGBoost`colsample_bytree`: float; Subsample ratio of columns when constructing each tree.  `learning_rate`: float;  Boosting learning rate (xgb's "eta" $\eta$)  `max_depth`: int;  Maximum tree depth for base learners.  See a list [here](https://xgboost.readthedocs.io/en/latest/parameter.html).<jupyter_code>model2 = xgb.XGBRegressor(colsample_bytree=0.6, learning_rate=0.3, max_depth=5)
model2.fit(X_train, y_train)
y_model2 = model2.predict(X_test)

mean_squared_error(y_test, y_model2) / y_test.shape[0]<jupyter_output><empty_output><jupyter_text>Cross validation and hyperparameter<jupyter_code>from sklearn.model_selection import cross_val_score<jupyter_output><empty_output><jupyter_text>The default score for regression is R2.<jupyter_code>print(cross_val_score(model1, X, y, cv=3))
print(cross_val_score(model2, X, y, cv=3))<jupyter_output>[0.87884148 0.87869525 0.87208372]
[0.87876011 0.87779849 0.87089951]<jupyter_text>apply grid search to find good hyperparameter<jupyter_code>from sklearn.model_selection import GridSearchCV
model = xgb.XGBRegressor()
choices = {'colsample_bytree': [0.5, 1], 
           'learning_rate': [0.1, 0.3, 0.5],
           'max_depth': [3, 7]
          }

grid = GridSearchCV(model, cv=3, 
                    param_grid=choices)
grid.fit(X, y)
print(grid.best_score_)
print(grid.best_params_)<jupyter_output>0.884808470019013
{'colsample_bytree': 0.5, 'learning_rate': 0.1, 'max_depth': 7}<jupyter_text>Dig into the model One of the nice feature of XGBoost  is that it allows you  to know the importance of each feature.   Use `xgb.plot_importance([model_name])`  to visualize the feature importances.<jupyter_code>model = xgb.XGBRegressor()
model.fit(X_train, y_train)
xgb.plot_importance(model)<jupyter_output><empty_output><jupyter_text>Set the `figsize` to make it bigger if necessary.<jupyter_code>fig = plt.figure(figsize=(8,6))
ax = plt.gca()
xgb.plot_importance(model, ax=ax)<jupyter_output><empty_output><jupyter_text>To understand the model  and the meaning of the feature importance,  use `model.get_booster().trees_to_dataframe()`  to show  the information of the trees and their nodes.<jupyter_code>df = model.get_booster().trees_to_dataframe()
df.head()<jupyter_output><empty_output><jupyter_text>Use `model.get_booster().get_score()`  to see the importance of each feature.<jupyter_code>bster = model.get_booster()
bster.get_score()<jupyter_output><empty_output><jupyter_text>There are five `importance_type`:* `weight`: the number of times a feature is used to split the data across all trees.* `gain`: the average gain across all splits the feature is used in.* `cover`: the average coverage across all splits the feature is used in.* `total_gain`: the total gain across all splits the feature is used in.* `total_cover`: the total coverage across all splits the feature is used in.<jupyter_code>bster.get_score?<jupyter_output><empty_output><jupyter_text>For example, if `importance_type='gain'`  then the importance of a feature  is set to the sum of all gains  of nodes spliting the given feature.<jupyter_code>bster.get_score(importance_type='total_gain')
gains = {f: df.Gain[ df.Feature == f ].sum() for f in bster.feature_names}
gains<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11099971354912633, 'fraction_numerical': 0.02363219707820109, 'mean_word_length': 6.198969072164949, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 76, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '269377', 'license': 'MIT', 'n_tokens_mistral': 2520, 'n_tokens_neox': 2368, 'n_words': 640, 'path': 'Introduction-to-XGBRegressor.ipynb', 'repo_name': 'jephianlin/ModularPython', 'size': '6995'}