<jupyter_start><jupyter_text>Phase 3: Merging Datasets and Building a Model<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import re
from datetime import datetime
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import warnings
import statsmodels.formula.api as smf
from sklearn.linear_model import Lasso
import statsmodels.api as sm<jupyter_output><empty_output><jupyter_text>The code below loads and cleans the original NYPD Felonies dataset. The code specifically selects the following features (fields within the dataset outlined in red below). The selected fields do not contain null values.<jupyter_code>def cleaning_data(data):
    # useful features
    features = ['objectid','occurrence_date','day_of_week','occurrence_month','occurrence_day','occurrence_year','occurrence_hour'
               , 'compstat_month','compstat_day','compstat_year','offense','offense_classification','sector','precinct','jurisdiction'
               , 'xcoordinate','ycoordinate']
    # select useful featuers from original data
    clean_data = data[features]
    return clean_data
    
data = pd.read_csv('./NYPD_Dataset.csv')
data = cleaning_data(data)<jupyter_output><empty_output><jupyter_text>Merging the NY Population Dataset with the NYPD Felonies Dataset The code below loads the population data, filters the Borough field for Manhattan, and outputs the first five rows of the dataset.<jupyter_code>pop_data = pd.read_csv('./New_York_City_Population_By_Community_Districts.csv')
man_mask = pop_data['Borough'] == 'Manhattan'
pop_data = pop_data[man_mask]
pop_data.head()<jupyter_output><empty_output><jupyter_text>After filtering on the Borough field for Manhattan, the 2010 Population field will be merged to the original NYPD Felonies dataset. The limitation here is that my original dataset is not organized at the community-district level and therefore the merging excercise at the community-district level is elevated to the borough level. Therefore, a new field is added to the original dataset that contains the entire Manhattan population. The code below prints the population of Manhattan in 2010.<jupyter_code>print("Manhattan 2010 population: " + str(sum(pop_data['2010 Population'])))<jupyter_output>Manhattan 2010 population: 1577385<jupyter_text>The code below merges the new field, Manhattan_Population, to the original NYPD Felonies dataset and provides a snapshot of the output.<jupyter_code>total_pop = sum(pop_data['2010 Population'])
total_pop_col = [total_pop] * len(data)
data['Manhattan_Population'] = total_pop_col
data<jupyter_output><empty_output><jupyter_text>Building a ModelThe following code counts the number of felonies per precinct for grand larceny and felony assault offense types. The code below applied no additional filtering criteria for time and covers the years 2006 through 2016.<jupyter_code>pre_list = pd.unique(data['precinct'])
offense_type_list = ['GRAND LARCENY','FELONY ASSAULT']
felony_count_dic = {}

for off in offense_type_list:
    off_mask = data['offense'] == off
    off_data = data[off_mask]
    if len(off_data) > 0 :
        felony_count_dic.setdefault(off,{})
        for p in pre_list:
            p_mask = off_data['precinct'] == p
            p_data = off_data[p_mask]
            if len(p_data) > 0 :
                felony_count_dic[off].setdefault(p,len(p_data))
                
fel_df = pd.DataFrame(felony_count_dic)
fel_df = fel_df.rename(index=str, columns={"GRAND LARCENY": "GRAND_LARCENY", "FELONY ASSAULT": "FELONY_ASSAULT"})<jupyter_output><empty_output><jupyter_text>The code below forms the model and presents the ordinary least squares regression outputs.<jupyter_code>precinct_model = smf.ols('GRAND_LARCENY ~ FELONY_ASSAULT', data=fel_df).fit()
precinct_model.summary()<jupyter_output><empty_output><jupyter_text>The code below plots the relevant data points and forms a regression line.<jupyter_code>fig = plt.figure(figsize=(12,8))
fig = sm.graphics.plot_partregress_grid(precinct_model, fig=fig)<jupyter_output><empty_output><jupyter_text>The above regression model (second chart directly above this markdown) regresses the occurrences of felony assault crimes against the occurrences of grand larceny crimes at the precinct level over the period 2006 through 2016. The number of unique precincts included in the above regression was 22. While the model was successfully run, the output of the regression analysis did not yield statistically significant findings. For example, the coefficient of determination (R2) was 0.022, which implies a low statistical fit between the felonies under comparison. In other words, the occurrances of felony assault cannot be caused or explained by its relationship to grand larceny crimes.<jupyter_code>###<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07629768605378362, 'fraction_numerical': 0.012507817385866166, 'mean_word_length': 5.834757834757835, 'pattern_counts': {'":': 2, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '120579', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1482, 'n_tokens_neox': 1367, 'n_words': 465, 'path': 'fimofimo/Phase3-final.ipynb', 'repo_name': 'em212/applied_ds', 'size': '4797'}