<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 3, Module 3*--- Permutation & BoostingYou will use your portfolio project dataset for all assignments this sprint. AssignmentComplete these tasks for your project, and document your work.- [ ] If you haven't completed assignment 1, please do so first.- [ ] Continue to clean and explore your data. Make exploratory visualizations.- [ ] Fit a model. Does it beat your baseline? - [ ] Try xgboost.- [ ] Get your model's permutation importances.You should try to complete an initial model today, because the rest of the week, we're making model interpretation visualizations.But, if you aren't ready to try xgboost and permutation importances with your dataset today, that's okay. You can practice with another dataset instead. You may choose any dataset you've worked with previously.The data subdirectory includes the Titanic dataset for classification and the NYC apartments dataset for regression. You may want to choose one of these datasets, because example solutions will be available for each. ReadingTop recommendations in _**bold italic:**_ Permutation Importances- _**[Kaggle / <NAME>: Machine Learning Explainability](https://www.kaggle.com/dansbecker/permutation-importance)**_- [<NAME>: Interpretable Machine Learning](https://christophm.github.io/interpretable-ml-book/feature-importance.html) (Default) Feature Importances  - [Ando Saabas: Selecting good features, Part 3, Random Forests](https://blog.datadive.net/selecting-good-features-part-iii-random-forests/)  - [Ter<NAME>, et al: Beware Default Random Forest Importances](https://explained.ai/rf-importance/index.html) Gradient Boosting  - [A Gentle Introduction to the Gradient Boosting Algorithm for Machine Learning](https://machinelearningmastery.com/gentle-introduction-gradient-boosting-algorithm-machine-learning/)  - [An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf), Chapter 8  - _**[Gradient Boosting Explained](https://www.gormanalysis.com/blog/gradient-boosting-explained/)**_ — <NAME>  - [Gradient Boosting Explained](http://arogozhnikov.github.io/2016/06/24/gradient_boosting_explained.html) — <NAME>  - [How to explain gradient boosting](https://explained.ai/gradient-boosting/) — <NAME> & <NAME><jupyter_code>import pandas as pd
test=pd.read_csv(r"C:\Users\Aarons\Downloads\airline-passenger-satisfaction\test.csv")
train=pd.read_csv(r"C:\Users\Aarons\Downloads\airline-passenger-satisfaction\train.csv")
              
train.shape, test.shape
train = train.drop(columns=['Unnamed: 0', 'id' ])
train = train.dropna(subset=['Cleanliness'])
train['Clean'] = (train['Cleanliness'] >= 4)
# Create a binary classification for the new clean variable.
y = train['Clean'].unique()
y

train['Clean'].value_counts(normalize=True)
train = train.fillna("Missing")
from sklearn.model_selection import train_test_split

train, val = train_test_split(train, train_size=0.80, test_size=0.20, 
                             stratify=train['Clean'], random_state=42)

test = test.drop(columns=['Unnamed: 0'])

train.shape, val.shape, test.shape
import category_encoders as ce
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier

target = "Clean"
features = train.columns.drop([target, 'Flight Distance', 'Cleanliness'])
X_train = train[features]
y_train = train[target]
X_val = val[features]
y_val = val[target]
import category_encoders as ce
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

pipeline = make_pipeline(
    ce.OrdinalEncoder(), 
    SimpleImputer(strategy='median'), 
    RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
)

# Fit on train, score on val
pipeline.fit(X_train, y_train)
print('Validation Accuracy', pipeline.score(X_val, y_val))
# We can get permutation importances for model interpretation and feature selection.

rf = pipeline.named_steps['randomforestclassifier']
importances = pd.Series(rf.feature_importances_, X_train.columns)

# Plot feature importances
%matplotlib inline
import matplotlib.pyplot as plt

n = 10
plt.figure(figsize=(10,n/2))
plt.title(f'Top {n} features')
importances.sort_values()[-n:].plot.barh(color='grey');
from sklearn.metrics import accuracy_score

y_pred = pipeline.predict(X_val)
print(f'Val accuracy: {accuracy_score(y_val, y_pred)}')
transformers = make_pipeline(
    ce.ordinal.OrdinalEncoder(),
    SimpleImputer()
)

X_train_transformed = transformers.fit_transform(X_train)
X_val_transformed = transformers.transform(X_val)

model = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
model.fit(X_train_transformed, y_train)
import eli5
from eli5.sklearn import PermutationImportance

permuter = PermutationImportance(
    model,
    scoring='accuracy',
    n_iter=5,
    random_state=42
)

permuter.fit(X_val_transformed, y_val)
feature_names = X_val.columns.to_list()
pd.Series(permuter.feature_importances_, feature_names).sort_values(ascending=True)
permuter.feature_importances_std_
eli5.show_weights(permuter, 
                  top=None, 
                  feature_names=feature_names)
from xgboost import XGBClassifier

pipeline = make_pipeline(
    ce.ordinal.OrdinalEncoder(),
    XGBClassifier(n_estimators=100, random_state=42, n_jobs=-1)
)

pipeline.fit(X_train, y_train)
print(f'Shape before removing features: {X_train.shape}')
minimum_importance = 0
mask = permuter.feature_importances_ > minimum_importance
features = X_train.columns[mask]
X_train = X_train[features]
# Remove features with 0 importance.
print(f'Shape after removing features: {X_train.shape}')
X_val = X_val[features]
pipeline = make_pipeline(
    ce.OrdinalEncoder(), 
    SimpleImputer(strategy='median'), 
    RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
)
pipeline.fit(X_train, y_train)
pipeline.score(X_val, y_val)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10763371419109125, 'fraction_numerical': 0.01092896174863388, 'mean_word_length': 5.73355629877369, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 14, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '404993', 'license': 'MIT', 'n_tokens_mistral': 1956, 'n_tokens_neox': 1853, 'n_words': 465, 'path': 'module3-permutation-boosting/Jason Robinson_DS_233_assignment.ipynb', 'repo_name': 'techthumb1/DS-Unit-2-Applied-Modeling', 'size': '6083'}