<jupyter_start><jupyter_text>---  Focusing on H1N1, so dropping seasonal columns (and respondent_id which isn't needed)<jupyter_code>train.drop(columns=['respondent_id', 'doctor_recc_seasonal', 'opinion_seas_vacc_effective', 'opinion_seas_risk', 'opinion_seas_sick_from_vacc'], inplace=True)<jupyter_output><empty_output><jupyter_text>Drop rows without response on h1n1 opinion<jupyter_code>train.dropna(subset=['opinion_h1n1_vacc_effective', 'opinion_h1n1_risk', 'opinion_h1n1_sick_from_vacc'], inplace=True)<jupyter_output><empty_output><jupyter_text>Drop rows without response on behavioral questions<jupyter_code>train.dropna(subset=['behavioral_antiviral_meds', 'behavioral_avoidance', 'behavioral_face_mask', 'behavioral_wash_hands', 
                     'behavioral_large_gatherings', 'behavioral_outside_home', 'behavioral_touch_face'], inplace=True)<jupyter_output><empty_output><jupyter_text>Drop rows without response on h1n1 concern or knowledge<jupyter_code>train.dropna(subset=['h1n1_concern', 'h1n1_knowledge'], inplace=True)<jupyter_output><empty_output><jupyter_text>To reduce coding complexity, drop rows with missing binary values that won't greatly reduce total samples.<jupyter_code>binary_cols_drop = ['doctor_recc_h1n1', 'chronic_med_condition', 'child_under_6_months', 'health_worker']
train.dropna(subset=binary_cols_drop, inplace=True)
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 23018 entries, 0 to 26706
Data columns (total 32 columns):
 #   Column                       Non-Null Count  Dtype  
---  ------                       --------------  -----  
 0   h1n1_concern                 23018 non-null  float64
 1   h1n1_knowledge               23018 non-null  float64
 2   behavioral_antiviral_meds    23018 non-null  float64
 3   behavioral_avoidance         23018 non-null  float64
 4   behavioral_face_mask         23018 non-null  float64
 5   behavioral_wash_hands        23018 non-null  float64
 6   behavioral_large_gatherings  23018 non-null  float64
 7   behavioral_outside_home      23018 non-null  float64
 8   behavioral_touch_face        23018 non-null  float64
 9   doctor_recc_h1n1             23018 non-null  float64
 10  chronic_med_condition        23018 non-null  float64
 11  child_under_6_months         23018 non-null  float64
 12  health_worker                23018 non-null  float64
 13  health_insurance[...]<jupyter_text>---  Impute health_insurance with -1 for NaN.  Will one-hot encode for training and predicting<jupyter_code>train['health_insurance'].value_counts()
train['health_insurance'].fillna(value=-1, inplace=True)
train['health_insurance'].value_counts()<jupyter_output><empty_output><jupyter_text>---  Impute string categorical values with 'unknown'.  Will one-hot encode if using for training and predicting.<jupyter_code>cat_cols = ['education', 'marital_status', 'employment_status', 'employment_industry', 'employment_occupation', 'rent_or_own']
train['education'].value_counts()
train['rent_or_own'].value_counts()
train[cat_cols] = train[cat_cols].fillna(value='unknown')
train['education'].value_counts()<jupyter_output><empty_output><jupyter_text>----  Impute with most frequent<jupyter_code>train['income_poverty'].value_counts()
most_freq_cols = ['income_poverty', 'household_adults', 'household_children']
si = SimpleImputer(strategy='most_frequent')
train[most_freq_cols] = si.fit_transform(train[most_freq_cols])
train['marital_status'].value_counts()
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 23018 entries, 0 to 26706
Data columns (total 32 columns):
 #   Column                       Non-Null Count  Dtype  
---  ------                       --------------  -----  
 0   h1n1_concern                 23018 non-null  float64
 1   h1n1_knowledge               23018 non-null  float64
 2   behavioral_antiviral_meds    23018 non-null  float64
 3   behavioral_avoidance         23018 non-null  float64
 4   behavioral_face_mask         23018 non-null  float64
 5   behavioral_wash_hands        23018 non-null  float64
 6   behavioral_large_gatherings  23018 non-null  float64
 7   behavioral_outside_home      23018 non-null  float64
 8   behavioral_touch_face        23018 non-null  float64
 9   doctor_recc_h1n1             23018 non-null  float64
 10  chronic_med_condition        23018 non-null  float64
 11  child_under_6_months         23018 non-null  float64
 12  health_worker                23018 non-null  float64
 13  health_insurance[...]<jupyter_text>---<jupyter_code># save clean version
train.to_csv('../Data/combin_vaccine_data_clean.csv')<jupyter_output><empty_output><jupyter_text>Updates from EDA- Make age_group a binary split at 55- make income a binary split at $75,000- Group unkown in with not married for marital_status<jupyter_code># read in saved clean data
train_update = pd.read_csv('../Data/combin_vaccine_data_clean.csv', index_col=0)<jupyter_output><empty_output><jupyter_text>Age group<jupyter_code>train_update['age_group'].value_counts()
train_update['age_group'] = np.where((train_update['age_group']=='65+ Years') | 
                                     (train_update['age_group']=='55 - 64 Years'), 1, 0)
train_update['age_group'].value_counts()<jupyter_output><empty_output><jupyter_text>Income<jupyter_code>train_update['income_poverty'].value_counts()
train_update['income_poverty'] = np.where(train_update['income_poverty']=='> $75,000', 1, 0)
train_update['income_poverty'].value_counts()<jupyter_output><empty_output><jupyter_text>Marital Status<jupyter_code>train_update['marital_status'].value_counts()
train_update['marital_status'] = np.where(train_update['marital_status']=='Married', 1, 0)
train_update['marital_status'].value_counts()
# save update clean df
train_update.to_csv('../Data/combin_vaccine_data_clean_updated.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10737453055650392, 'fraction_numerical': 0.05257767156025948, 'mean_word_length': 4.601338432122371, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '464094', 'license': 'MIT', 'n_tokens_mistral': 2204, 'n_tokens_neox': 1926, 'n_words': 357, 'path': 'Code/01_Cleaning.ipynb', 'repo_name': 'ericBayless/H1N1-Vaccine-Classification', 'size': '5858'}