<jupyter_start><jupyter_text>Lambda School Data Science, Unit 2: Predictive Modeling Kaggle Challenge, Module 2 Assignment- [ ] Read [“Adopting a Hypothesis-Driven Workflow”](https://outline.com/5S5tsB), a blog post by a Lambda DS student about the Tanzania Waterpumps challenge.- [ ] Continue to participate in our Kaggle challenge.- [ ] Try Ordinal Encoding.- [ ] Try a Random Forest Classifier.- [ ] Submit your predictions to our Kaggle competition. (Go to our Kaggle InClass competition webpage. Use the blue **Submit Predictions** button to upload your CSV file. Or you can use the Kaggle API to submit your predictions.)- [ ] Commit your notebook to your fork of the GitHub repo. Stretch Goals Doing- [ ] Add your own stretch goal(s) !- [ ] Do more exploratory data analysis, data cleaning, feature engineering, and feature selection.- [ ] Try other [categorical encodings](https://contrib.scikit-learn.org/categorical-encoding/).- [ ] Get and plot your feature importances.- [ ] Make visualizations and share on Slack. ReadingTop recommendations in _**bold italic:**_ Decision Trees- A Visual Introduction to Machine Learning, [Part 1: A Decision Tree](http://www.r2d3.us/visual-intro-to-machine-learning-part-1/),  and _**[Part 2: Bias and Variance](http://www.r2d3.us/visual-intro-to-machine-learning-part-2/)**_- [Decision Trees: Advantages & Disadvantages](https://christophm.github.io/interpretable-ml-book/tree.htmladvantages-2)- [How a Russian mathematician constructed a decision tree — by hand — to solve a medical problem](http://fastml.com/how-a-russian-mathematician-constructed-a-decision-tree-by-hand-to-solve-a-medical-problem/)- [How decision trees work](https://brohrer.github.io/how_decision_trees_work.html)- [Let’s Write a Decision Tree Classifier from Scratch](https://www.youtube.com/watch?v=LDRbO9a6XPU) Random Forests- [_An Introduction to Statistical Learning_](http://www-bcf.usc.edu/~gareth/ISL/), Chapter 8: Tree-Based Methods- [Coloring with Random Forests](http://structuringtheunstructured.blogspot.com/2017/11/coloring-with-random-forests.html)- _**[Random Forests for Complete Beginners: The definitive guide to Random Forests and Decision Trees](https://victorzhou.com/blog/intro-to-random-forests/)**_ Categorical encoding for trees- [Are categorical variables getting lost in your random forests?](https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/)- [Beyond One-Hot: An Exploration of Categorical Variables](http://www.willmcginnis.com/2015/11/29/beyond-one-hot-an-exploration-of-categorical-variables/)- _**[Categorical Features and Encoding in Decision Trees](https://medium.com/data-design/visiting-categorical-features-and-encoding-in-decision-trees-53400fa65931)**_- _**[Coursera — How to Win a Data Science Competition: Learn from Top Kagglers — Concept of mean encoding](https://www.coursera.org/lecture/competitive-data-science/concept-of-mean-encoding-b5Gxv)**_- [Mean (likelihood) encodings: a comprehensive study](https://www.kaggle.com/vprokopev/mean-likelihood-encodings-a-comprehensive-study)- [The Mechanics of Machine Learning, Chapter 6: Categorically Speaking](https://mlbook.explained.ai/catvars.html) Imposter Syndrome- [Effort Shock and Reward Shock (How The Karate Kid Ruined The Modern World)](http://www.tempobook.com/2014/07/09/effort-shock-and-reward-shock/)- [How to manage impostor syndrome in data science](https://towardsdatascience.com/how-to-manage-impostor-syndrome-in-data-science-ad814809f068)- ["I am not a real data scientist"](https://brohrer.github.io/imposter_syndrome.html)- _**[Imposter Syndrome in Data Science](https://caitlinhudon.com/2018/01/19/imposter-syndrome-in-data-science/)**_ More Categorical Encodings**1.** The article **[Categorical Features and Encoding in Decision Trees](https://medium.com/data-design/visiting-categorical-features-and-encoding-in-decision-trees-53400fa65931)** mentions 4 encodings:- **"Categorical Encoding":** This means using the raw categorical values as-is, not encoded. Scikit-learn doesn't support this, but some tree algorithm implementations do. For example, [Catboost](https://catboost.ai/), or R's [rpart](https://cran.r-project.org/web/packages/rpart/index.html) package.- **Numeric Encoding:** Synonymous with Label Encoding, or "Ordinal" Encoding with random order. We can use [category_encoders.OrdinalEncoder](https://contrib.scikit-learn.org/categorical-encoding/ordinal.html).- **One-Hot Encoding:** We can use [category_encoders.OneHotEncoder](http://contrib.scikit-learn.org/categorical-encoding/onehot.html).- **Binary Encoding:** We can use [category_encoders.BinaryEncoder](http://contrib.scikit-learn.org/categorical-encoding/binary.html).**2.** The short video **[Coursera — How to Win a Data Science Competition: Learn from Top Kagglers — Concept of mean encoding](https://www.coursera.org/lecture/competitive-data-science/concept-of-mean-encoding-b5Gxv)** introduces an interesting idea: use both X _and_ y to encode categoricals.Category Encoders has multiple implementations of this general concept:- [CatBoost Encoder](http://contrib.scikit-learn.org/categorical-encoding/catboost.html)- [James-Stein Encoder](http://contrib.scikit-learn.org/categorical-encoding/jamesstein.html)- [Leave One Out](http://contrib.scikit-learn.org/categorical-encoding/leaveoneout.html)- [M-estimate](http://contrib.scikit-learn.org/categorical-encoding/mestimate.html)- [Target Encoder](http://contrib.scikit-learn.org/categorical-encoding/targetencoder.html)- [Weight of Evidence](http://contrib.scikit-learn.org/categorical-encoding/woe.html)Category Encoder's mean encoding implementations work for regression problems or binary classification problems. For multi-class classification problems, you will need to temporarily reformulate it as binary classification. For example:```pythonencoder = ce.TargetEncoder(min_samples_leaf=..., smoothing=...)  Both parameters > 1 to avoid overfittingX_train_encoded = encoder.fit_transform(X_train, y_train=='functional')X_val_encoded = encoder.transform(X_train, y_val=='functional')```For this reason, mean encoding won't work well within pipelines for multi-class classification problems.**3.** The **[dirty_cat](https://dirty-cat.github.io/stable/)** library has a Target Encoder implementation that works with multi-class classification.```python dirty_cat.TargetEncoder(clf_type='multiclass-clf')```It also implements an interesting idea called ["Similarity Encoder" for dirty categories](https://www.slideshare.net/GaelVaroquaux/machine-learning-on-non-curated-data-154905090).However, it seems like dirty_cat doesn't handle missing values or unknown categories as well as category_encoders does. And you may need to use it with one column at a time, instead of with your whole dataframe.**4. [Embeddings](https://www.kaggle.com/learn/embeddings)** can work well with sparse / high cardinality categoricals._**I hope it’s not too frustrating or confusing that there’s not one “canonical” way to encode categorcals. It’s an active area of research and experimentation! Maybe you can make your own contributions!**_  SetupYou can work locally (follow the [local setup instructions](https://lambdaschool.github.io/ds/unit2/local/)) or on Colab (run the code cell below).<jupyter_code>%%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Kaggle-Challenge/master/data/'
    !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'
import numpy as np

import category_encoders as ce
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split

data = pd.merge(pd.read_csv(DATA_PATH+'waterpumps/train_features.csv'), 
                 pd.read_csv(DATA_PATH+'waterpumps/train_labels.csv'))
test = pd.read_csv(DATA_PATH+'waterpumps/test_features.csv')
sample_submission = pd.read_csv(DATA_PATH+'waterpumps/sample_submission.csv')

data.shape, test.shape
#train/validate split
train, val = train_test_split(data, random_state=42)

train.shape, val.shape, test.shape
#Define a function to wrangle train, validate, and test sets in the same way. 
#Clean outliers and engineer features.

def wrangle(X):
  """Wrangle train, validate, and test sets in the same way"""
    
  # Prevent SettingWithCopyWarning
  X = X.copy()
    
  # About 3% of the time, latitude has small values near zero,
  # outside Tanzania, so we'll treat these values like zero.
  X['latitude'] = X['latitude'].replace(-2e-08, 0)
    
  # When columns have zeros and shouldn't, they are like null values.
  # So we will replace the zeros with nulls, and impute missing values later.
  #confirmed model is .2% worse when excluding population here
  cols_with_zeros = ['longitude', 'latitude','construction_year','population']
  for col in cols_with_zeros:
      X[col] = X[col].replace(0, np.nan)
  
  #drop duplicate columns
  # quantity & quantity_group are duplicates, so drop one
  # payment == payment_type
  X = X.drop(columns=['quantity_group','payment_type'])
  
  #recode permit to 1/0
  #this doesn't turn out make a meaningful difference
  #X['permit'] = X['permit'].replace({True: 1, False: 0})
  
  #new feature: whether amount_tsh is 0 or not
  #it turns out to be wildly unimportant!
  #(might be helpful on a regression; trees don't need it)
  #X['0_tsh'] = X['amount_tsh'] == 0
  
  #convert date_recorded to datetime
  X['date_recorded'] = X['date_recorded'].apply(pd.to_datetime)
  
  #create 'age' column, the number of years between construction and evaluation
  X['age'] = X['date_recorded'].apply(lambda d: d.year) - X['construction_year']
  
  #pare down high-cardinality variables prior to encoding
  #if a category has fewer than threshold observations, nan it
  #since we're ordinal encoding now, let's try removing this
  #removing it = modest validation accuracy increase
  
  #threshold = 600 #tweaking threshold value for best results
  #value_counts = X['funder'].value_counts()
  #to_remove = value_counts[value_counts < threshold].index
  #X['funder'] = X['funder'].replace(to_remove, np.nan)
  
  #value_counts = X['installer'].value_counts()
  #to_remove = value_counts[value_counts < threshold].index
  #X['installer'] = X['installer'].replace(to_remove, np.nan)
  
  #value_counts = X['lga'].value_counts()
  #to_remove = value_counts[value_counts < threshold].index
  #X['lga'] = X['lga'].replace(to_remove, np.nan)
  
  #scheme_management has fine cardinality but a pesky "None" observed once
  X['scheme_management'] = X['scheme_management'].replace('None', np.nan)
  
  #bools for whether the waterpoint is marked as rural or urban
  #...make it a tiny bit worse
  #X['rural'] = X['lga'].apply(lambda s: "Rural" in s)
  #X['urban'] = X['lga'].apply(lambda s: "Urban" in s)
  
  #fill nans in latitude and longitude with averages for the region/basin
  #instead of relying on the simple imputer
  #using region makes the model worse by .3%
  #using basin makes it better by about the same amount!
  X2 = pd.DataFrame(columns=X.columns)

  for basin in X['basin'].value_counts().index:
    df = X[X['basin'] == basin]
    lat_mean = df['latitude'].mean()
    long_mean = df['longitude'].mean()
    df['latitude'] = df['latitude'].replace(np.nan,lat_mean)
    df['longitude'] = df['longitude'].replace(np.nan,long_mean)
    X2 = X2.append(df)
    
  # return the wrangled dataframe
  #return X
  return X2

train = wrangle(train)
val = wrangle(val)
test = wrangle(test)

train.head()
#feature selection!
#just dumping in all the features (aside from date_recorded, which causes an error)
#isn't actually all that much better than this list I used yesterday

target = 'status_group'
features = ['amount_tsh','funder','gps_height','installer','longitude','latitude',
            'basin','region','lga','population','permit','construction_year',
           'extraction_type_group','management','payment','water_quality',
           'quantity','source','waterpoint_type','age']

#X_train = train[features]
X_train = train.drop(columns=[target,'date_recorded'])
y_train = train[target]

#X_val = val[features]
X_val = val.drop(columns=[target,'date_recorded'])
y_val = val[target]

#X_test = test[features]
X_test = test.drop(columns=['date_recorded'])
#pipeline includes:
#encoding of categorical variables as ordinal/numeric
#imputer for nans (median and mean are within .1% of each other in performance)
#random forest (many trees)
pipeline = make_pipeline(
    ce.OrdinalEncoder(), 
    SimpleImputer(strategy='median'), 
    RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)
)

pipeline.fit(X_train, y_train)
print('Validation Accuracy', pipeline.score(X_val, y_val))
encoder = pipeline.named_steps['ordinalencoder']
encoded = encoder.transform(X_train)

# Get feature importances
rf = pipeline.named_steps['randomforestclassifier']
importances = pd.Series(rf.feature_importances_, encoded.columns)

# Plot feature importances
n = 100
plt.figure(figsize=(10,n/2))
plt.title(f'Top {n} features')
importances.sort_values()[-n:].plot.barh(color='grey');

#WHAT are id and name doing there, honestly
#make submission for kaggle
X_test = test.drop(columns=['date_recorded'])

y_pred = pipeline.predict(X_test)

y_pred
submission = pd.DataFrame(data={'id':test['id'],'status_group':y_pred})

submission.head()
submission.to_csv('third_submission.csv', index=False)<jupyter_output><empty_output><jupyter_text>testing code for feature engineering<jupyter_code>train['region'].value_counts().index
#there are no nans in region, so what if 0 lats/longs were 
#recoded to the mean of their region's values?
shin = train[train['region'] == 'Shinyanga']

print(shin.shape)
shin['longitude'].describe()
train['latitude'] = train['latitude'].replace(-2e-08, 0)
train['latitude'] = train['latitude'].replace(0, np.nan)
train['longitude'] = train['longitude'].replace(0, np.nan)
train['latitude'].mean()
#means = pd.DataFrame(columns=['lat_mean','long_mean'])

train2 = pd.DataFrame(columns=train.columns)

#cuts data up by region, then assigns missing lat and long values to regional mean
for region in train['region'].value_counts().index:
  df = train[train['region'] == region]
  lat_mean = df['latitude'].mean()
  long_mean = df['longitude'].mean()
  df['latitude'] = df['latitude'].replace(np.nan,lat_mean)
  df['longitude'] = df['longitude'].replace(np.nan,long_mean)
  train2 = train2.append(df)
  
train2.describe()
  
  
  
  #row = pd.DataFrame(data={'lat_mean':lat_mean,'long_mean':long_mean},
  #                   index=[region],
  #                   columns=['lat_mean','long_mean'])
  #means = means.append(row)
  
#means


#step one: turn zeroes into nans
#step two: turn nans into the mean for that region
train2.head()
train2[train2['region'] == 'Shinyanga']['latitude'].describe()
means.loc['Shinyanga']
means.loc['Shinyanga'][0] #latitude mean for Shinyanga
means.loc['Shinyanga'][1] #longitude mean
train.shape
train2['longitude'].describe()
test['basin'].value_counts(dropna=False)
test['subvillage'].value_counts(dropna=False)
test['lga'].value_counts(dropna=False)
#could urban/rural make a difference?
test['ward'].value_counts(dropna=False)
#'urban' = whether the word 'urban' appears in the lga
test['urban'] = test['lga'].apply(lambda s: "Urban" in s)

test[test['lga'] == 'Shinyanga Urban']
test.head()
#'rural' = whether the word 'rural' appears in the lga
test['rural'] = test['lga'].apply(lambda s: "Rural" in s)

test[test['lga'] == 'Shinyanga Rural']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13161732073796995, 'fraction_numerical': 0.01001711785963355, 'mean_word_length': 5.345132743362832, 'pattern_counts': {'":': 1, '<': 12, '<?xml version=': 0, '>': 10, 'https://': 24, 'lorem ipsum': 0, 'www.': 11, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '548455', 'license': 'MIT', 'n_tokens_mistral': 5186, 'n_tokens_neox': 4907, 'n_words': 1327, 'path': 'module2/assignment_kaggle_challenge_2.ipynb', 'repo_name': 'srfinley/DS-Unit-2-Kaggle-Challenge', 'size': '15773'}