<jupyter_start><jupyter_text>Clean Data  Remove Cities with a 0 populations<jupyter_code># Population cannot be 0
combined_csv.loc[combined_csv['Population']==0.0]
combined_csv = combined_csv[combined_csv.Population != 0]<jupyter_output><empty_output><jupyter_text>Fill empty columns with violent crime and property crime sums if available.- property crime is the sum of 'Burglary', 'Larceny- theft', 'Motor vehicle theft'- violent crime is the sum of 'Murder and nonnegligent manslaughter', 'Rape','Robbery', 'Aggravated assault'<jupyter_code>violent_crimes = ['Murder and nonnegligent manslaughter', 'Rape','Robbery', 
                  'Aggravated assault']
combined_csv['Violent crime']= combined_csv[violent_crimes].sum(axis=1)
property_crimes = ['Burglary', 'Larceny- theft', 'Motor vehicle theft']
combined_csv['Property crime']= combined_csv[property_crimes].sum(axis=1)<jupyter_output><empty_output><jupyter_text>Feature Engineering- Creating a crime rate per 1000 column - Creating a crime rating column  Crime RateCrime Rate per 1,000 inhabitants: This represents the number of Index offenses per 1,000 inhabitants.For example: What is the crime rate for a municipality with 513 Index offenses (murder, rape, robbery,aggravated assault, burglary, larceny-theft and motor vehicle theft), with a population of 8,280?513 (Index offenses) ÷ 8,280 (population) = .061957 x 1,000 = 62.0 (crime per 1,000 inhabitants) https://www.njsp.org/info/ucr2000/pdf/calc_ucr2000.pdf<jupyter_code># Crime is broken down into violent crime and property crime
# Arson is reported in conjunction with property or violent crime depending on the nature
columns = ['Violent crime', 'Property crime']
combined_csv[columns] = combined_csv[columns].astype(int)
def sum_frame_by_column(frame, new_col_name, list_of_cols_to_sum, divideby):
    frame[new_col_name] = ((frame[list_of_cols_to_sum].astype(int).sum(axis=1)/frame[divideby].astype(int))*1000).round(2)
    return(frame)
sum_frame_by_column(combined_csv, 'Crime Rate (per 1000 residents)', columns, 'Population' )
min(combined_csv['Crime Rate (per 1000 residents)']), max(combined_csv['Crime Rate (per 1000 residents)'])
# Just wanted to see the type of crime for such a high percentage
combined_csv.loc[combined_csv['Crime Rate (per 1000 residents)']==25625.0]<jupyter_output><empty_output><jupyter_text>Using pd.cut() to create a Crime Ratings Columns<jupyter_code>combined_csv['Crime Rate (per 1000 residents)'].describe()
combined_csv['Crime Rating'] = pd.qcut(combined_csv['Crime Rate (per 1000 residents)'], q=3, labels=['Low', 'Medium', 'High'])
combined_csv.head()
combined_csv.to_csv("../fbi_crime_uscities.csv", index=False)
fbi_crime_filepath = os.path.join(path, "../fbi_crime_uscities.csv")
fbi_crime_uscities = pd.read_csv(fbi_crime_filepath)
fbi_crime_uscities.head()
fbi_crime_uscities.loc[fbi_crime_uscities['Crime Rate (per 1000 residents)']>=100]
len(fbi_crime_uscities.loc[fbi_crime_uscities['Crime Rate (per 1000 residents)']>=100])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11702479338842975, 'fraction_numerical': 0.03603305785123967, 'mean_word_length': 7.5, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '244293', 'license': 'MIT', 'n_tokens_mistral': 1107, 'n_tokens_neox': 977, 'n_words': 260, 'path': 'Datasets/Data/Crime_Data/fbi_crime_us.ipynb', 'repo_name': 'CityScape-Datasets/Workspace_Datasets', 'size': '3025'}