<jupyter_start><jupyter_text>NFIRS CODE CHECKThis notebook was created to check and clean the NFIRS_2009_2016_geocoded_with_tract.csv Dataset.<jupyter_code>import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt


# Change to your filepath to data and uncomment if NFIRS_2009_2016.csv not in the main path 
os.chdir('../Data/Raw')
NFIR = pd.read_csv('NFIRS_2009_2016_geocoded_with_tract.csv',
                   encoding='latin_1',
                   index_col=0,
                   low_memory=False)<jupyter_output><empty_output><jupyter_text>Null Analysis<jupyter_code>NFIR['X'].isnull().value_counts(normalize = True)<jupyter_output><empty_output><jupyter_text>16% of data is not properly geocoded, as long as its uniformly missing it shouldn't affect results drastically.  From the Code below it looks like it is all above  75% accounted for except for West Virginia, which has 60%  Also we can see there were some malformed queries as there are 2 Digit state codes such as CZ,DD and VI.<jupyter_code>MATCH =  NFIR["MATCH"].groupby(NFIR['STATE']).value_counts(normalize = True)

name = 'PERCENT_MATCHED'

MATCH
# Data Cleaning 
 

NFIR = NFIR[~NFIR['X'].isnull()]
NFIR = NFIR[NFIR.STATE.str.match('[A-Z]{2}')]


plt.scatter(NFIR['X'].values,NFIR['Y'].values)
plt.show()<jupyter_output><empty_output><jupyter_text>As we can see most of the points form the Continental US, Alaska and hawaii in the correct orientation indicating that most points are in the correct (X,Y) orientation. From looking at the rows it seems as though there are some rows in the form (Y,X) since we don't have any visible gaps in the map of the US it does not seem to be a problem that happens at the state level.  Considering the fact that our improperly geocoded rows cluster so nicely We should be able to create a criterion to search for them and correct them<jupyter_code>((NFIR['X'] > 0) & (NFIR['Y'] < 0)).value_counts()
# here is the code to swap elements if they appear backwards

NFIR[['X','Y']] = NFIR[['Y','X']].where( (NFIR['X'] > 0) & (NFIR['Y'] < 0), NFIR[['X','Y']].values )
((NFIR['X'] > 0) & (NFIR['Y'] < 0)).value_counts()<jupyter_output><empty_output><jupyter_text>now things are looking much better!<jupyter_code>plt.scatter(NFIR['X'].values,NFIR['Y'].values)
plt.show()

NFIR.to_csv('NFIR_Clean.csv',header= True)
NFIR<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11757269279393173, 'fraction_numerical': 0.01685630004214075, 'mean_word_length': 5.134366925064599, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '596778', 'license': 'MIT', 'n_tokens_mistral': 874, 'n_tokens_neox': 785, 'n_words': 253, 'path': 'Code/Munging/NFIRS/NFIRS_Geocode_Cleaner.ipynb', 'repo_name': 'LaurenHa/rcp2', 'size': '2373'}