<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Data Description2&nbsp;&nbsp;Imports3&nbsp;&nbsp;Useful Scripts4&nbsp;&nbsp;Load the data5&nbsp;&nbsp;Data Processing and Feature Engineering5.1&nbsp;&nbsp;Date time features5.2&nbsp;&nbsp;Categorical Features6&nbsp;&nbsp;Missing Values7&nbsp;&nbsp;Save clean data  Data Description- Data source: https://www.kaggle.com/AnalyzeBoston/crimes-in-boston**Context**  Crime incident reports are provided by Boston Police Department (BPD) to document the initial details surrounding an incident to which BPD officers respond. This is a dataset containing records from the new crime incident report system, which includes a reduced set of fields focused on capturing the type of incident as well as when and where it occurred.**Content**  Records begin in June 14, 2015 and continue to September 3, 2018.**Acknowledgements**  The data is provided by Analyze Boston. The most up-to-date version can be found [here](https://data.boston.gov/dataset/crime-incident-reports-august-2015-to-date-source-new-system).**Inspiration**  - What types of crimes are most common? - Where are different types of crimes most likely to occur? - Does the frequency of crimes change over the day? Week? Year?  Imports<jupyter_code>import bhishan
%load_ext autoreload
autoreload 2
import numpy as np
import pandas as pd
import seaborn as sns
sns.set(color_codes=True)

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
sns.set(context='notebook', style='whitegrid', rc={'figure.figsize': (12,8)})
plt.style.use('ggplot') # better than sns styles.
matplotlib.rcParams['figure.figsize'] = 12,8

import os
import time


# random state
random_state=100
np.random.seed(random_state)

# Jupyter notebook settings for pandas
#pd.set_option('display.float_format', '{:,.2g}'.format) # numbers sep by comma
from pandas.api.types import CategoricalDtype
np.set_printoptions(precision=3)
pd.set_option('display.max_columns', 100)
pd.set_option('display.max_rows', 100) # None for all the rows
pd.set_option('display.max_colwidth', 200)

import IPython
from IPython.display import display, HTML, Image, Markdown

print([(x.__name__,x.__version__) for x in [np, pd,sns,matplotlib]])<jupyter_output>[('numpy', '1.16.4'), ('pandas', '0.25.0'), ('seaborn', '0.9.0'), ('matplotlib', '3.1.1')]<jupyter_text>Useful Scripts<jupyter_code>def show_method_attributes(method, ncols=7,start=None):
    """ Show all the attributes of a given method.
    Example:
    ========
    show_method_attributes(list)
     """
    x = [I for I in dir(method) if I[0]!='_' ]
    x = [I for I in x 
         if I not in 'os np pd sys time psycopg2'.split() ]
    if start:
        x = [I for I in x if I.startswith(start)]

    return pd.DataFrame(np.array_split(x,ncols)).T.fillna('')<jupyter_output><empty_output><jupyter_text>Load the data<jupyter_code>df = pd.read_csv('../data/raw/crime.csv',encoding='latin-1')
print(df.shape)
df.head()
df.columns
df.columns = df.columns.str.lower()
df.columns
df.columns = ['incident', 'code', 'group',
       'description', 'district', 'area', 'shooting',
       'date', 'year', 'month', 'dayofweek', 'hour', 'ucr_part',
       'street', 'lat', 'long', 'location']
df.head()<jupyter_output><empty_output><jupyter_text>Data Processing and Feature Engineering<jupyter_code>from bhishan.util_ds import get_column_descriptions<jupyter_output><empty_output><jupyter_text>Date time features<jupyter_code>from pandas.api.types import CategoricalDtype
# we already have year, month and hour as numeric type
# we need to make occurred on date as datetime object
df['date'] = pd.to_datetime(df['date'], errors='coerce')
show_method_attributes(df['date'].dt)
df['dayofweek']  = df['date'].dt.dayofweek.astype(CategoricalDtype())
df['dayofmonth'] = df['date'].dt.day.astype(CategoricalDtype())
df['dayofyear']  = df['date'].dt.dayofyear.astype(CategoricalDtype())
df['weekofyear'] = df['date'].dt.weekofyear.astype(CategoricalDtype())
df['quarter']    = df['date'].dt.quarter.astype(CategoricalDtype())

df['day_name']   = df['date'].dt.day_name().astype(CategoricalDtype())
df['month_name'] = df['date'].dt.month_name().astype(CategoricalDtype())

df.head()<jupyter_output><empty_output><jupyter_text>Categorical FeaturesRef: https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html- Some categorical features may be number type, make them string type.- Some categorical features may be nans make them 'Missing'- We may fill categorical features by mode.<jupyter_code>get_column_descriptions(df)
# 99% shooting data is nans, we can drop this column
cols_obj = df.select_dtypes(['object']).columns
cols_obj
df.select_dtypes(['object']).apply(pd.Series.nunique)
len(df)
df.select_dtypes(['object']).apply(pd.Series.nunique).loc[lambda x: x<20]
cols_obj_small = df.select_dtypes(['object']).apply(pd.Series.nunique).loc[lambda x: x<20].index

for c in cols_obj_small:
    print('\n=========================================')
    print(c) 
    print(df[c].value_counts())<jupyter_output>=========================================
district
B2     49945
C11    42530
D4     41915
A1     35717
B3     35442
C6     23460
D14    20127
E13    17536
E18    17348
A7     13544
E5     13239
A15     6505
Name: district, dtype: int64

=========================================
shooting
N    318054
Y      1019
Name: shooting, dtype: int64

=========================================
ucr_part
Part Three    158553
Part Two       97569
Part One       61629
Other           1232
Name: ucr_part, dtype: int64<jupyter_text>Missing Values<jupyter_code># make shootings nans as N
df['shooting'] = df['shooting'].fillna('N')

df['shooting'].value_counts()
df['lat'] = df['lat'].replace(-1, None)
df['long'] = df['long'].replace(-1, None)<jupyter_output><empty_output><jupyter_text>Save clean data<jupyter_code>df.to_csv('../data/processed/data_cleaned.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16098294884653963, 'fraction_numerical': 0.03092611166833835, 'mean_word_length': 5.699888017917133, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 31, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '585510', 'license': 'MIT', 'n_tokens_mistral': 2138, 'n_tokens_neox': 1991, 'n_words': 490, 'path': 'Crimes_in_Boston/notebooks/a01_timeseries_data_processing.ipynb', 'repo_name': 'bhishanpdl/Projects', 'size': '5982'}