<jupyter_start><jupyter_text>Loading the file shared by Shreyas<jupyter_code>import pandas as pd
import numpy as np
pd.set_option('display.max_columns', 50)
pd.set_option('display.max_rows', 1000)

df_SJ = pd.read_csv("fema_dataset.csv")
# Checking the first few rows 
df_SJ.head()
#  check the data type
df_SJ.dtypes<jupyter_output><empty_output><jupyter_text>FEMA Dataset from my local machine<jupyter_code>df_fema = pd.read_excel("data.gov.FEMADeclarations.3.15.19.xlsx",  sheet_name="Sheet4")
df_fema.shape
# these two datasets have the same shape
df_fema.head()<jupyter_output><empty_output><jupyter_text>difference in data type df_SJ vs df_FEMAI might want to choose df_FEMA because I want to keep the datatime data format for columns containing data time info<jupyter_code>df_fema.dtypes
# Let's check the missing vlaues
df_fema.isna().sum()
df_fema.columns<jupyter_output><empty_output><jupyter_text>drop columns that aren't necessary for now<jupyter_code>df_fema_columns_dropped = df_fema.drop(columns=["Disaster Close Out Date", "Declaration Request Number" ])
df_fema_columns_dropped['Declared County/Area'].isna().sum()<jupyter_output><empty_output><jupyter_text>Let's drop the null counties (misisng county name/code)<jupyter_code>df_fema_no_missing = df_fema_columns_dropped.dropna()
df_fema_no_missing.isna().sum()
df.shape<jupyter_output><empty_output><jupyter_text>Load the US state and County files<jupyter_code>#  states' names
df_states = pd.read_csv("us-state-names.csv", sep='\t')
df_states.head()
# US counties 
df_counties = pd.read_csv("us-county-names.csv", sep='\t')
df_counties.head()
df_fema_no_missing.columns
df_fema_sorted_by_state = df_fema_no_missing.sort_values(by=['State '])
df_fema_sorted_by_state.head()
df_states.sort_values(by=['code'])
state_dict = dict(zip(df_states.code, df_states.name))
state_dict
df_fema_no_missing['State_name'] = np.nan 
df_fema_no_missing.head()
for key, value in state_dict.items():
    print(key, value)  
# this code below doesn't work
# because of the missing index values
# because I deleted some misisng rows earlier 
# thus iterating over index doesn't work
# %%time
# for i in range(0, 500):
#     state_short = df_fema_no_missing['State '][i]
#     state_full_name = state_dict[state_short]
# #     print(state_short, state_full_name)
#     df_fema_no_missing['State_name'][i] = state_full_name<jupyter_output><empty_output><jupyter_text>adding State names to the dataframe<jupyter_code>for index, row_ptr in df_fema_no_missing.iterrows():
    state_short = row_ptr['State ']
    state_full_name = state_dict[state_short]
#     print(state_short, state_full_name)
    df_fema_no_missing.loc[index,'State_name'] = state_full_name<jupyter_output><empty_output><jupyter_text>Let's bring State_name column close to the State abbriviation column<jupyter_code>df_fema_no_missing = df_fema_no_missing[['Disaster Number', 
                                         'IH Program Declared', 
                                         'IA Program Declared',
       'PA Program Declared', 'HM Program Declared', 'State ','State_name',
       'Declaration Date', 'Disaster Type', 'Incident Type', 'Title',
       'Incident Begin Date', 'Incident End Date', 'Place Code',
       'Declared County/Area']]
df_fema_no_missing.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10940325497287523, 'fraction_numerical': 0.004822182037371911, 'mean_word_length': 5.787321063394683, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '329046', 'license': 'MIT', 'n_tokens_mistral': 1212, 'n_tokens_neox': 1180, 'n_words': 240, 'path': 'notebooks/MM01_Working_with_FEMA_Dataset_Manjula.ipynb', 'repo_name': 'KimDuclos/liveSafe-data', 'size': '3318'}