<jupyter_start><jupyter_text>14 - Data Preparationby [<NAME>](http://www.albahnsen.com/) & [<NAME>](http://www.ivantorroledo.com/)and [<NAME>](https://github.com/jesugome)version 1.5, February 2019 Part of the class [Practical Machine Learning](https://github.com/albahnsen/PracticalMachineLearningClass)This notebook is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US). Special thanks goes to [<NAME>](https://github.com/justmarkham)  Handling missing valuesscikit-learn models expect that all values are **numeric** and **hold meaning**. Thus, missing values are not allowed by scikit-learn.<jupyter_code>import pandas as pd
url = 'https://raw.githubusercontent.com/albahnsen/PracticalMachineLearningClass/master/datasets/titanic.csv.zip'
titanic = pd.read_csv(url, index_col=0)
titanic.head()
# check for missing values
titanic.isnull().sum()<jupyter_output><empty_output><jupyter_text>One possible strategy is to **drop missing values**:<jupyter_code># drop rows with any missing values
titanic.dropna().shape
# drop rows where Age is missing
titanic[titanic.Age.notnull()].shape<jupyter_output><empty_output><jupyter_text>Sometimes a better strategy is to **impute missing values**:<jupyter_code># mean Age
titanic.Age.mean()
# median Age
titanic.Age.median()
titanic.loc[titanic.Age.isnull()]<jupyter_output><empty_output><jupyter_text>most frequent Agetitanic.Age.mode()<jupyter_code># fill missing values for Age with the median age
titanic.Age.fillna(titanic.Age.median(), inplace=True)<jupyter_output><empty_output><jupyter_text>Another strategy would be to build a **KNN model** just to impute missing values. How would we do that?If values are missing from a categorical feature, we could treat the missing values as **another category**. Why might that make sense?How do we **choose** between all of these strategies?  Handling categorical featuresHow do we include a categorical feature in our model?- **Ordered categories:** transform them to sensible numeric values (example: small=1, medium=2, large=3)- **Unordered categories:** use dummy encoding (0/1)<jupyter_code>titanic.head(10)
# encode Sex_Female feature
titanic['Sex_Female'] = titanic.Sex.map({'male':0, 'female':1})
# create a DataFrame of dummy variables for Embarked
embarked_dummies = pd.get_dummies(titanic.Embarked, prefix='Embarked')
embarked_dummies.drop(embarked_dummies.columns[0], axis=1, inplace=True)

# concatenate the original DataFrame and the dummy DataFrame
titanic = pd.concat([titanic, embarked_dummies], axis=1)
titanic.head(1)<jupyter_output><empty_output><jupyter_text>- How do we **interpret** the encoding for Embarked?- Why didn't we just encode Embarked using a **single feature** (C=0, Q=1, S=2)?- Does it matter which category we choose to define as the **baseline**?- Why do we only need **two dummy variables** for Embarked?<jupyter_code># define X and y
feature_cols = ['Pclass', 'Parch', 'Age', 'Sex_Female', 'Embarked_Q', 'Embarked_S']
X = titanic[feature_cols]
y = titanic.Survived

# train/test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

# train a logistic regression model
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(C=1e9)
logreg.fit(X_train, y_train)

# make predictions for testing set
y_pred_class = logreg.predict(X_test)

# calculate testing accuracy
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred_class))<jupyter_output>0.7937219730941704<jupyter_text>Advanced Categorical Encoding  Mushroom DatabaseThis data set includes descriptions of hypothetical samples    corresponding to 23 species of gilled mushrooms in the Agaricus and    Lepiota Family (pp. 500-525).  Each species is identified as    definitely edible, definitely poisonous, or of unknown edibility and    not recommended.  This latter class was combined with the poisonous    one.  The Guide clearly states that there is no simple rule for    determining the edibility of a mushroom; no rule like leaflets three, let it be for Poisonous Oak and Ivy.Mushroom records drawn from The Audubon Society Field Guide to North        American Mushrooms (1981). <NAME> (Pres.), New York: Alfred        <NAME><jupyter_code>url = 'https://raw.githubusercontent.com/albahnsen/PracticalMachineLearningClass/master/datasets/agaricus-lepiota.zip'
data = pd.read_csv(url, index_col=None)
data = data.drop(['capcolor', 'stalkcolorabovering', 'odor', 'gillsize', 'sporeprintcolor', 'stalksurfaceabovering',
                  'ringtype', 'stalkroot', 'bruises'], axis=1)
data = data.sample(frac=1, random_state=42)
data.head()
data.columns<jupyter_output><empty_output><jupyter_text>Attribute Information: (classes: edible=e, poisonous=p)     1. cap-shape:                bell=b,conical=c,convex=x,flat=f,                                  knobbed=k,sunken=s     2. cap-surface:              fibrous=f,grooves=g,scaly=y,smooth=s     3. cap-color:                brown=n,buff=b,cinnamon=c,gray=g,green=r,                                  pink=p,purple=u,red=e,white=w,yellow=y     4. bruises?:                 bruises=t,no=f     5. odor:                     almond=a,anise=l,creosote=c,fishy=y,foul=f,                                  musty=m,none=n,pungent=p,spicy=s     6. gill-attachment:          attached=a,descending=d,free=f,notched=n     7. gill-spacing:             close=c,crowded=w,distant=d     8. gill-size:                broad=b,narrow=n     9. gill-color:               black=k,brown=n,buff=b,chocolate=h,gray=g,                                  green=r,orange=o,pink=p,purple=u,red=e,                                  white=w,yellow=y    10. stalk-shape:              enlarging=e,tapering=t    11. stalk-root:               bulbous=b,club=c,cup=u,equal=e,                                  rhizomorphs=z,rooted=r,missing=?    12. stalk-surface-above-ring: fibrous=f,scaly=y,silky=k,smooth=s    13. stalk-surface-below-ring: fibrous=f,scaly=y,silky=k,smooth=s    14. stalk-color-above-ring:   brown=n,buff=b,cinnamon=c,gray=g,orange=o,                                  pink=p,red=e,white=w,yellow=y    15. stalk-color-below-ring:   brown=n,buff=b,cinnamon=c,gray=g,orange=o,                                  pink=p,red=e,white=w,yellow=y    16. veil-type:                partial=p,universal=u    17. veil-color:               brown=n,orange=o,white=w,yellow=y    18. ring-number:              none=n,one=o,two=t    19. ring-type:                cobwebby=c,evanescent=e,flaring=f,large=l,                                  none=n,pendant=p,sheathing=s,zone=z    20. spore-print-color:        black=k,brown=n,buff=b,chocolate=h,green=r,                                  orange=o,purple=u,white=w,yellow=y    21. population:               abundant=a,clustered=c,numerous=n,                                  scattered=s,several=v,solitary=y    22. habitat:                  grasses=g,leaves=l,meadows=m,paths=p,                                  urban=u,waste=w,woods=d  Dummies<jupyter_code>y = (data['class'] == 'p') * 1.0
y.mean(), y.shape
X = data.drop(['class'], axis=1)
X = pd.get_dummies(X)
X.head()
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
pd.Series(cross_val_score(RandomForestClassifier(n_estimators=10), X, y, cv=10, scoring='accuracy')).describe()<jupyter_output><empty_output><jupyter_text>PCA<jupyter_code>from sklearn.decomposition import PCA
X_ = PCA(n_components=8).fit_transform(X)
pd.Series(cross_val_score(RandomForestClassifier(n_estimators=10), X_, y, cv=10, scoring='accuracy')).describe()<jupyter_output><empty_output><jupyter_text>PCA can only be estimated if num_columns < num_observations  Other encoders http://contrib.scikit-learn.org/categorical-encoding/<jupyter_code>!pip install category_encoders
import category_encoders as ce<jupyter_output><empty_output><jupyter_text>BinaryBinary encoding for categorical variables, similar to onehot, but stores categories as binary bitstrings.<jupyter_code>X_ = ce.BinaryEncoder().fit_transform(data.drop(['class'], axis=1))
X_.head()
pd.Series(cross_val_score(RandomForestClassifier(n_estimators=10), X_, y, cv=10, scoring='accuracy')).describe()<jupyter_output><empty_output><jupyter_text>Feature HashingFeature Hashing for Large Scale Multitask Learninghttps://alex.smola.org/papers/2009/Weinbergeretal09.pdf``Empirical evidence suggests that hashing is an effective strategy for dimensionality reduction and practical nonparametric estimation. In this paper we provide exponential tail bounds for feature hashing and show that the interaction between random subspaces is negligible with high probability. We demonstrate the feasibility of this approach with experimental results for a new use case — multitask learning with hundreds of thousands of tasks.``<jupyter_code>X_ = ce.HashingEncoder(n_components=8).fit_transform(data.drop(['class'], axis=1))
X_.head()
pd.Series(cross_val_score(RandomForestClassifier(n_estimators=10), X_, y, cv=10, scoring='accuracy')).describe()<jupyter_output><empty_output><jupyter_text>Helmert CodingHelmert coding compares each level of a categorical variable to the mean of the subsequent levels.  Hence, the first contrast compares the mean of the dependent variable for level 1 of race with the mean of all of the subsequent levels of race (levels 2, 3, and 4), the second contrast compares the mean of the dependent variable for level 2 of race with the mean of all of the subsequent levels of race (levels 3 and 4), and the third contrast compares the mean of the dependent variable for level 3 of race with the mean of all of the subsequent levels of race (level 4).<NAME> (2003). Coding Categorical Variableshttp://psych.colorado.edu/~carey/courses/psyc5741/handouts/Coding%20Categorical%20Variables%202006-03-03.pdf<jupyter_code>X_ = ce.HelmertEncoder().fit_transform(data.drop(['class'], axis=1))
X_.head()
pd.Series(cross_val_score(RandomForestClassifier(n_estimators=10), X_, y, cv=10, scoring='accuracy')).describe()<jupyter_output><empty_output><jupyter_text>Ordinal<jupyter_code>X_ = ce.OrdinalEncoder().fit_transform(data.drop(['class'], axis=1))
X_.head()
pd.Series(cross_val_score(RandomForestClassifier(n_estimators=10), X_, y, cv=10, scoring='accuracy')).describe()<jupyter_output><empty_output><jupyter_text>Polynomial CodingPolynomial contrast coding for the encoding of categorical featuresOrthogonal polynomial coding is a form of trend analysis in that it is looking for the linear, quadratic and cubic trends in the categorical variable. This type of coding system should be used only with an ordinal variable in which the levels are equally spaced. Examples of such a variable might be income or education.<jupyter_code>X_ = ce.PolynomialEncoder().fit_transform(data.drop(['class'], axis=1))
X_.head()
pd.Series(cross_val_score(RandomForestClassifier(n_estimators=10), X_, y, cv=10, scoring='accuracy')).describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11089108910891089, 'fraction_numerical': 0.01575157515751575, 'mean_word_length': 4.6515768056968465, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 67, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '574577', 'license': 'MIT', 'n_tokens_mistral': 3738, 'n_tokens_neox': 3495, 'n_words': 880, 'path': 'notebooks/14-data_preparation.ipynb', 'repo_name': 'andaoba/PracticalMachineLearningClass', 'size': '11165'}