<jupyter_start><jupyter_text>Cleaning, EDA<jupyter_code>import numpy as np
import scipy.stats as stats
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

plt.style.use('fivethirtyeight')
sns.set_style('whitegrid')

%matplotlib inline
df = pd.read_csv('Feature Elimination 1.csv')
df.head()
df.shape
df.dtypes<jupyter_output><empty_output><jupyter_text>Looks like we have have some object data types. Probably because of commas, dollar signs, whitespaces, etc. Let's clean them up and convert them into float data types. We have some null values in this data frame. We could fill them in, but let's see what percentage of the data frame has null values.<jupyter_code>df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Okay dropping the null values isn't too bad. Lets' go ahead and drop them with at most 4% of the data frame.<jupyter_code>df.isnull().sum() / df.shape[0] 
(df.dropna(axis=0, how='any').shape[0]) / df.shape[0]
df = df.dropna(how='any', axis=0)<jupyter_output><empty_output><jupyter_text>Let's try dropping the nulls to see how the models perform on a preliminary basis...<jupyter_code>df['Med_HHD_Inc_ACS_09_13'] = df['Med_HHD_Inc_ACS_09_13'].str.replace('$', '')
df['Med_HHD_Inc_ACS_09_13'] = df['Med_HHD_Inc_ACS_09_13'].str.replace(',', '')
df['Med_HHD_Inc_ACS_09_13'] = df['Med_HHD_Inc_ACS_09_13'].astype(float)
df['Med_House_value_ACS_09_13'] = df['Med_House_value_ACS_09_13'].str.replace('$', '')
df['Med_House_value_ACS_09_13'] = df['Med_House_value_ACS_09_13'].str.replace(',', '').astype(float)
df['avg_Agg_HH_INC_ACS_09_13'] = df['avg_Agg_HH_INC_ACS_09_13'].str.replace('$', '')
df['avg_Agg_HH_INC_ACS_09_13'] = df['avg_Agg_HH_INC_ACS_09_13'].str.replace(',', '').astype(float)
df['avg_Agg_House_Value_ACS_09_13'] = df['avg_Agg_House_Value_ACS_09_13'].str.replace('$', '')
df['avg_Agg_House_Value_ACS_09_13'] = df['avg_Agg_House_Value_ACS_09_13'].str.replace(',', '').astype(float)
df['Aggr_House_Value_ACS_09_13'] = df['Aggr_House_Value_ACS_09_13'].str.replace('$', '')
df['Aggr_House_Value_ACS_09_13'] = df['Aggr_House_Value_ACS_09_13'].str.replace(',', '').astype(float)
df['Aggregate_HH_INC_ACS_09_13'] = df['Aggregate_HH_INC_ACS_09_13'].str.replace('$', '')
df['Aggregate_HH_INC_ACS_09_13'] = df['Aggregate_HH_INC_ACS_09_13'].str.replace(',', '').astype(float)
df.dtypes #Cool, the only object data types are the state and county names. We're good with data cleaning here!
df.to_csv('Cleaned up data.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15605095541401273, 'fraction_numerical': 0.04418789808917197, 'mean_word_length': 8.238970588235293, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '165304', 'license': 'MIT', 'n_tokens_mistral': 1104, 'n_tokens_neox': 982, 'n_words': 180, 'path': '01 Cleaning.ipynb', 'repo_name': 'joshsungasong/Capstone-Project', 'size': '2512'}