<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 3, Module 2*--- Permutation & BoostingYou will use your portfolio project dataset for all assignments this sprint. AssignmentComplete these tasks for your project, and document your work.- [ ] If you haven't completed assignment 1, please do so first.- [ ] Continue to clean and explore your data. Make exploratory visualizations.- [ ] Fit a model. Does it beat your baseline? - [ ] Try xgboost.- [ ] Get your model's permutation importances.You should try to complete an initial model today, because the rest of the week, we're making model interpretation visualizations.But, if you aren't ready to try xgboost and permutation importances with your dataset today, that's okay. You can practice with another dataset instead. You may choose any dataset you've worked with previously.The data subdirectory includes the Titanic dataset for classification and the NYC apartments dataset for regression. You may want to choose one of these datasets, because example solutions will be available for each. ReadingTop recommendations in _**bold italic:**_ Permutation Importances- _**[Kaggle / <NAME>: Machine Learning Explainability](https://www.kaggle.com/dansbecker/permutation-importance)**_- [<NAME>: Interpretable Machine Learning](https://christophm.github.io/interpretable-ml-book/feature-importance.html) (Default) Feature Importances  - [Ando Saabas: Selecting good features, Part 3, Random Forests](https://blog.datadive.net/selecting-good-features-part-iii-random-forests/)  - [Terence Parr, et al: Beware Default Random Forest Importances](https://explained.ai/rf-importance/index.html) Gradient Boosting  - [A Gentle Introduction to the Gradient Boosting Algorithm for Machine Learning](https://machinelearningmastery.com/gentle-introduction-gradient-boosting-algorithm-machine-learning/)  - _**[A Kaggle Master Explains Gradient Boosting](http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/)**_  - [_An Introduction to Statistical Learning_](http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf) Chapter 8  - [Gradient Boosting Explained](http://arogozhnikov.github.io/2016/06/24/gradient_boosting_explained.html)  - _**[Boosting](https://www.youtube.com/watch?v=GM3CDQfQ4sw) (2.5 minute video)**_<jupyter_code>from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.feature_selection import f_regression, SelectKBest
import itertools
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn import metrics
from sklearn.metrics import confusion_matrix
from sklearn.utils.multiclass import unique_labels
from sklearn.metrics import classification_report
from scipy.stats import randint, uniform
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_absolute_error
import warnings
import pandas as pd
import plotly_express as px
import eli5
from eli5.sklearn import PermutationImportance
import category_encoders as ce
from xgboost import XGBRegressor
df=pd.read_csv('https://raw.githubusercontent.com/VeraMendes/Project---Train-a-predictive-model/master/led.csv')
print(df.shape)
df.head()
# looking into nan's
df.isnull().sum()
# finding the nan's for lifeexpectancy
df[pd.isnull(df['Lifeexpectancy'])]
# finding the nan's for GDP
df[pd.isnull(df['GDP'])]
# finding the nan's for Polio
df[pd.isnull(df['Polio'])]
# finding the nan's for thinness5-9years
df[pd.isnull(df['thinness5-9years'])]
# checkig all my column names
df.columns
# changing my column names
df = df.rename(columns = {
    'Country':'country','Year':'year', 'Status':'development','Lifeexpectancy':'lifespan',
    'AdultMortality':'adult_mortality', 'infantdeaths':'infant_deaths',
    'Alcohol':'alcohol_conpsumption', 'percentageexpenditure':'percentage_expenditure',
    'HepatitisB':'hepatitisb','Measles':'measles','BMI':'BMI','under-fivedeaths':'baby_deaths',
    'Polio':'polio', 'Totalexpenditure':'total_expenditure','Diphtheria':'diphtheria','HIV/AIDS':'HIV/AIDS',
    'GDP':'GDP','Population':'population','thinness1-19years':'thinness_teenager',
    'thinness5-9years':'thinness_children','Incomecompositionofresources':'ICR','Schooling':'education'
})

df.head()
# dropping nan's for lifespan
df = df.dropna(axis=0, subset=['lifespan'])
# checking dimnesions of my df
df.shape
# due to leakage problems, I am creating a new column which is going to be my y_target: next_year_lifespan
df['next_year_lifespan']= df['lifespan'].shift(1)
df.head()
target = 'next_year_lifespan'
df.head(50)
df['year'].value_counts()
# 2015 does not have data on y_target for any country
# With pd.shift, I have moved into 2015 wrong values for the countries considered.
year_2015 = df[df.year == 2015]
year_2015.head()<jupyter_output><empty_output><jupyter_text>Looking into Time Series I am: using 2013 & 2014 as test using 2011 & 2012 as val I cannot use 2015 values as I don't have values for the next year lifespan (2016)<jupyter_code># For my analysis I am dropping 2015 rows to avoid errors and leakage.
df = df[df.year != 2015]
df.head()
# looking into nan's
df.isnull().sum()
# splitting my data into a time series manner
train = df[df['year']<2011]
val = df[(df.year == 2011) | (df.year == 2012)]
test = df[(df.year == 2013) | (df.year == 2014)]
train.shape, val.shape, test.shape
# using previous year lifespan, as y_pred = y_baseline
y_baseline = pd.Series(train['lifespan'], index=train.index)
print(y_baseline)
# Arrange data into X features matrix and y target vector
# removing previous year lifespan from all my data
target = 'next_year_lifespan'
baseline_values = 'lifespan'
cols_to_drop = ['next_year_lifespan', 'lifespan']
X_train = train[train.columns.drop(cols_to_drop)]
y_train = train['next_year_lifespan']
X_val = val[val.columns.drop(cols_to_drop)]
y_val = val['next_year_lifespan']
X_test = test[test.columns.drop(cols_to_drop)]
y_test = test['next_year_lifespan']
# using previous year lifespan, as y_pred = y_baseline to try to find a better model fit
mean_baseline = train['next_year_lifespan'].mean()
y_pred = val['lifespan']
baseline_mae = mean_absolute_error(y_val, y_pred)
print('mean baseline:', train['next_year_lifespan'].mean())
print(f'Mean Absolut Error of the baseline prediction (predict lifespan for years 2011 & 2012): {baseline_mae:.4f}')<jupyter_output>mean baseline: 68.68564331843021
Mean Absolut Error of the baseline prediction (predict lifespan for years 2011 & 2012): 0.8281<jupyter_text>baseline mae is equal to 0.8281 years.<jupyter_code># using RandomForestRegressor
pipeline = make_pipeline(
    ce.OrdinalEncoder(), 
    SimpleImputer(strategy='median'), 
    RandomForestRegressor(n_estimators=100, n_jobs=-1)
)

#Fit on train, score on val
pipeline.fit(X_train, y_train)
y_pred= pipeline.predict(X_val)
# mean absolute error for this model
mae = mean_absolute_error(y_val, y_pred)
print(f"My model's absolut error: {mae:,.2f}")<jupyter_output>My model's absolut error: 1.46<jupyter_text>new model is not an improvement from the baseline.<jupyter_code># feature importances
rf = pipeline.named_steps['randomforestregressor']
importances = pd.Series(rf.feature_importances_, X_train.columns)

# Plotting feature importances
%matplotlib inline
import matplotlib.pyplot as plt

n = 10
plt.figure(figsize=(10,n/2))
plt.title(f'Top {n} features')
importances.sort_values()[-n:].plot.barh(color='blue');
# using pipeline first and then permuter
transformers = make_pipeline(
    ce.OrdinalEncoder(), 
    SimpleImputer(strategy='median')
)

X_train_transformed = transformers.fit_transform(X_train)
X_val_transformed = transformers.transform(X_val)

model = RandomForestRegressor(n_estimators=100, n_jobs=-1)
model.fit(X_train_transformed, y_train)
# using feature permutation on my model
permuter = PermutationImportance(
    model, 
    scoring='neg_mean_absolute_error', 
    n_iter=5, 
    random_state=42
)

permuter.fit(X_val_transformed, y_val)
feature_names = X_val.columns.tolist()
pd.Series(permuter.feature_importances_, feature_names).sort_values(ascending=False)
# looking into feature importance and weight
eli5.show_weights(
    permuter, 
    top=None,
    feature_names = feature_names
)
# fitting a different model:xgboost
pipeline = make_pipeline(
    ce.OrdinalEncoder(), 
    XGBRegressor(n_estimators=100, n_jobs=-1)
)

pipeline.fit(X_train, y_train)
# mean absolute error for this model
mae = mean_absolute_error(y_val, y_pred)
y_pred = pipeline.predict(X_val)
print(f"My model's absolut error: {mae:,.2f}")<jupyter_output>My model's absolut error: 1.64<jupyter_text>new model is not an improvement from the baseline.<jupyter_code># trying some different parameters on xgboost
encoder = ce.OrdinalEncoder()
X_train_encoded = encoder.fit_transform(X_train)
X_val_encoded = encoder.transform(X_val)

X_train.shape, X_val.shape, X_train_encoded.shape, X_val_encoded.shape
eval_set = [(X_train_encoded, y_train), 
            (X_val_encoded, y_val)]

model = XGBRegressor(
    n_estimators=2000,
    max_depth=7,       
    learning_rate=0.1, 
    n_jobs=-1
)

model.fit(X_train_encoded, y_train, eval_set=eval_set, 
          eval_metric='mae', early_stopping_rounds=50)<jupyter_output>[15:28:14] WARNING: src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[0]	validation_0-mae:61.3862	validation_1-mae:63.6096
Multiple eval metrics have been passed: 'validation_1-mae' will be used for early stopping.

Will train until validation_1-mae hasn't improved in 50 rounds.
[1]	validation_0-mae:55.2679	validation_1-mae:57.3627
[2]	validation_0-mae:49.7611	validation_1-mae:51.7172
[3]	validation_0-mae:44.8007	validation_1-mae:46.6107
[4]	validation_0-mae:40.3368	validation_1-mae:42.0071
[5]	validation_0-mae:36.3182	validation_1-mae:37.8648
[6]	validation_0-mae:32.7016	validation_1-mae:34.1458
[7]	validation_0-mae:29.4477	validation_1-mae:30.7935
[8]	validation_0-mae:26.5177	validation_1-mae:27.7703
[9]	validation_0-mae:23.8791	validation_1-mae:25.0285
[10]	validation_0-mae:21.5038	validation_1-mae:22.5977
[11]	validation_0-mae:19.3667	validation_1-mae:20.3886
[12]	validation_0-mae:17.4434	validation_1-mae:18.3925
[13]	validation_0-mae:[...]<jupyter_text>new model is not an improvement from the baseline.<jupyter_code>results = model.evals_result()
train_error = results['validation_0']['mae']
val_error = results['validation_1']['mae']
epoch = range(1, len(train_error)+1)
plt.plot(epoch, train_error, label='Train')
plt.plot(epoch, val_error, label='Validation')
plt.ylabel('Regressor Error')
plt.xlabel('Model Complexity (n_estimators)')
plt.ylim((0, 57)) 
plt.legend();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11125056028686688, 'fraction_numerical': 0.03765127745405648, 'mean_word_length': 6.286740692357936, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 23, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '98808', 'license': 'MIT', 'n_tokens_mistral': 4080, 'n_tokens_neox': 3677, 'n_words': 930, 'path': 'module2/Vera Mendes_Assignment_applied_modeling_2.ipynb', 'repo_name': 'VeraMendes/DS-Unit-2-Applied-Modeling', 'size': '11169'}