<jupyter_start><jupyter_text>Feature EngineeringIn order to get comfortable with common feature engineering techniques and how to implement them in python, we will use a toy dataset.  First, let's create the dataset, and we can also read in the necessary libraries.<jupyter_code>import numpy as np
import pandas as pd
import statsmodels.api as sm;
import sklearn.preprocessing as p

df = pd.DataFrame({'response': [2.4, 3.3, -4.2, 5.6, 1.5, 8.7], 
                         'x1': ['yes','no','yes','maybe','no','yes'],
                         'x2': [-1,-3,np.nan, 0, np.nan, 1],
                         'x3': [2.4, 15, 3.3, 2.4, 1.8, 0.4],
                         'x4': [np.nan, np.nan, 1, 1, 1, 1],
                         'x5': ['A', 'B', np.nan, 'A', 'A', 'A']})
df<jupyter_output><empty_output><jupyter_text>`1.` Fit a linear model between the response and the three x-variables in the dataset.  Also add an intercept.  Use the results to answer the first quiz question below.<jupyter_code>df['intercept'] = 1
lm = sm.OLS(df['response'], df[['intercept','x2','x3','x4']])
results = lm.fit()
results.summary()<jupyter_output><empty_output><jupyter_text>`2.` Use the sklearn documetation [here](http://scikit-learn.org/stable/modules/preprocessing.html) and the previous video to assist in filling in the missing values for each of the quantitative columns with the column mean.  Now, use the new columns to re-fit the linear model from question `1.`, and use the results to answer quiz 2 below.<jupyter_code>imp = p.Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(df[['x2', 'x3', 'x4']])
df[['x2', 'x3', 'x4']] = imp.transform(df[['x2', 'x3', 'x4']])
lm = sm.OLS(df['response'], df[['intercept','x2','x3','x4']])
results = lm.fit()
results.summary()<jupyter_output>/opt/conda/lib/python3.6/site-packages/statsmodels/stats/stattools.py:72: ValueWarning: omni_normtest is not valid with less than 8 observations; 6 samples were given.
  "samples were given." % int(n), ValueWarning)<jupyter_text>`3.` Another common way to scale features is by subtracting the mean and dividing by the standard deviation.  There are certain machine learning algorithms where you should always consider this type of scaling (or other ways of normalizing), as discussed [here](https://stats.stackexchange.com/questions/189652/is-it-a-good-practice-to-always-scale-normalize-data-for-machine-learning).  Use the [sklearn documentation](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.htmlsklearn.preprocessing.StandardScaler) and the previous video to assist with performing this scaling on the three new, quantitative columns in your dataset.  To assure you performed these transformations correctly, answer quiz 3 below.<jupyter_code>norm = p.StandardScaler()
norm.fit(df[['x2','x3','x4']])
norm.transform(df[['x2','x3','x4']])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14468232044198895, 'fraction_numerical': 0.026933701657458564, 'mean_word_length': 4.384758364312268, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '269551', 'license': 'MIT', 'n_tokens_mistral': 969, 'n_tokens_neox': 915, 'n_words': 288, 'path': 'Practical_Statistics/Practical_Statistics/13_MLR/Feature Engineering.ipynb', 'repo_name': 'christineton/Udacity_Data_Analysis_Nanodegree', 'size': '2896'}