<jupyter_start><jupyter_text>Crimes analysis of San Francisco and Seattle by Visualization <NAME>This notebook contains an analysis of the crimes of San Fransisco and Seattle, with some cross city features, by means of data visualization. All the code is reproducible with Jupypter v4.0.6, Python v3.5, Pandas v0.18.0 and Seaborn v0.7.0 (and the corresponding dependencies). For this you can install [Anaconda](https://www.continuum.io/downloads)<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from scipy import stats
from ipywidgets import interact
%matplotlib inline

# Plot visual parameters
plt.rcParams["axes.labelsize"] = 20
plt.rcParams["xtick.labelsize"] = 15
plt.rcParams["ytick.labelsize"] = 15
plt.rcParams["figure.figsize"] = (8*(1+np.sqrt(5))/2,8)<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>San FranciscoThe data is imported with the pandas library. The next output show a brief of the dataframe, and the first 3 rows for some insight of the raw data<jupyter_code>raw_data_sf = pd.read_csv("./sanfrancisco_incidents_summer_2014.csv")
raw_data_sf.info()
raw_data_sf.iloc[:3,:]<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 28993 entries, 0 to 28992
Data columns (total 13 columns):
IncidntNum    28993 non-null int64
Category      28993 non-null object
Descript      28993 non-null object
DayOfWeek     28993 non-null object
Date          28993 non-null object
Time          28993 non-null object
PdDistrict    28993 non-null object
Resolution    28993 non-null object
Address       28993 non-null object
X             28993 non-null float64
Y             28993 non-null float64
Location      28993 non-null object
PdId          28993 non-null int64
dtypes: float64(2), int64(2), object(9)
memory usage: 2.9+ MB<jupyter_text>The variable for this analysis will be __sf__, which is a dataframe with PdDistrict and Category as categorical types (because these features are Nominal) and Timestamp with the actual timestamp of the crime (with type datetime64 for custom search). The IncidntNum and PdId are droped; these are Ordinal values with no meaning for this analysis.<jupyter_code>sf = raw_data_sf.copy()
sf.Category = sf.Category.astype("category")
sf.PdDistrict = sf.PdDistrict.astype("category")

sf["Timestamp"] = pd.to_datetime(sf.Date + " " + sf.Time)
sf["HourDay"] = sf.Timestamp.dt.hour
sf.drop(["IncidntNum"], axis=1, inplace=True)
print(sf.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 28993 entries, 0 to 28992
Data columns (total 14 columns):
Category      28993 non-null category
Descript      28993 non-null object
DayOfWeek     28993 non-null object
Date          28993 non-null object
Time          28993 non-null object
PdDistrict    28993 non-null category
Resolution    28993 non-null object
Address       28993 non-null object
X             28993 non-null float64
Y             28993 non-null float64
Location      28993 non-null object
PdId          28993 non-null int64
Timestamp     28993 non-null datetime64[ns]
HourDay       28993 non-null int64
dtypes: category(2), datetime64[ns](1), float64(2), int64(2), object(7)
memory usage: 2.7+ MB
None<jupyter_text>Count of crimes by typeWith the data clean ready for plotting, the first visualization to be shown is the count of the crimes by type of crimes.<jupyter_code>sns.set_palette(sns.color_palette("pastel"))
ordVal = sf.groupby("Category")["PdId"].describe().unstack().sort_values(by=["count"], ascending=False).index.values
sns.countplot(data=sf, y="Category", order=ordVal);<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>As expected, LARCENY/THEFT is the most commont tpye of crimes, by a clear margin. Then OTHER OFFENCES (which is a vague category), NON-CRIMINAL and the fourth category is ASSAULT. The amount of ASSAULT crimes shows that San Francisco has an important amount violent crimes.  Count of crimes by time period Hour of the day<jupyter_code>sns.set_palette(sns.color_palette("pastel"))
sns.countplot(y=sf.Timestamp.dt.hour, order=range(0,24));<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>Hour of day and category of crimeThe last plot can have a stacked form, with the count by category. The categories with less than 10 crimes in one hour are ploted in another graphic, for the sake of understanding<jupyter_code>crime_type_hour = sf[["HourDay","Category"]]#.pivot_table(index="HourDay", columns="Category", aggfunc=lambda x: len(x.unique()))
sns.set_palette(sns.color_palette("coolwarm", crime_type_hour.Category.cat.categories.size))
crtb = pd.crosstab(crime_type_hour.Category, crime_type_hour.HourDay).dropna()
more = crtb[crtb > 10].dropna(thresh=24)
sns.set_palette(sns.color_palette("coolwarm", more.index.size))
more.T.sort_index(ascending=False).plot(kind="barh", stacked=True,)
plt.legend(bbox_to_anchor=(1.2, 1.0));

plt.figure();
less = crtb[crtb <= 10].dropna(thresh=24)
sns.set_palette(sns.color_palette("coolwarm", less.index.size))
less.T.sort_index(ascending=False).plot(kind="barh", stacked=True)
plt.legend(bbox_to_anchor=(1.3, 1.0))<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>Here is noted that the most common crime is the LANCERY/THEAF (as expected), and are more common in the late night and much less common in the morning. The same are for the crimes with less or equal than 10 ocurrences, but some crimes of these crimes occurs also in the late morning and evening (7AM to 1PM), like RUNAWAY and LIQUOUR LAWS; it's expected because these type of crimes does not correlate with time (means, the crime develops at any time).  By day of week<jupyter_code>sns.countplot(x=sf.Timestamp.dt.dayofweek, order=range(0,7))
plt.xlabel("Day of week")
plt.ylabel("Crime count");<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>Noted, that the crimes were developed in uniform way the days of the week  By day of week and type of crimeTwo plots, one with more than 20 crimes by day of week and type, and the other less than 20 crimes.<jupyter_code>crime_type_week = sf[["DayOfWeek","Category"]]
crtb = pd.crosstab(crime_type_week.Category, crime_type_week.DayOfWeek).dropna()
more = crtb[crtb > 20].dropna(thresh=7)
sns.set_palette(sns.color_palette("coolwarm", more.index.size))
more.T.sort_index(ascending=False).plot(kind="barh", stacked=True,)
plt.legend(bbox_to_anchor=(1.2, 1.0));

plt.figure();
less = crtb[crtb <= 20].dropna(thresh=7)
sns.set_palette(sns.color_palette("coolwarm", less.index.size))
less.T.sort_index(ascending=False).plot(kind="barh", stacked=True)
plt.legend(bbox_to_anchor=(1.3, 1.0))<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>By month<jupyter_code>sns.countplot(x=sf.Timestamp.dt.month, order=range(6,9))
plt.xlabel("Month")
plt.ylabel("Crime count");<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>As you can see, the crimes count almost don't change month to month. Nevertheless, there is a linear increasing (if you want to plot it). It can be meaningless  Count of crimes by district Without the type of crime<jupyter_code>sns.set_palette(sns.color_palette("pastel"))
sns.countplot(data=sf, y="PdDistrict");<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>With the type of crime specified<jupyter_code>crime_type_district = sf[["PdDistrict","Category"]]

crtb = pd.crosstab(crime_type_district.Category, crime_type_district.PdDistrict).dropna()
more = crtb[crtb > 10].dropna(thresh=10)
sns.set_palette(sns.color_palette("coolwarm", more.index.size))
more.T.plot(kind="barh", stacked=True)
plt.legend(bbox_to_anchor=(1.2, 1.0));

plt.figure();
less = crtb[crtb <= 10].dropna(thresh=10)
sns.set_palette(sns.color_palette("coolwarm", less.index.size))
less.T.plot(kind="barh", stacked=True)
plt.legend(bbox_to_anchor=(1.3, 1.0))<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>Only in the BAYVIEW, the LARCERY/THIEF crimes are not the most common type of crime. In that place, OTHER OFFENCES crimes are also common. It is not know if that have a explanation, because OTHER OFFENCES is a vague classification.  Seattle crimesNow the analysis for Seattle crimes<jupyter_code>raw_data_st = pd.read_csv("./seattle_incidents_summer_2014.csv")
raw_data_st.info()
raw_data_st.iloc[:3,:]<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 32779 entries, 0 to 32778
Data columns (total 19 columns):
RMS CDW ID                           32779 non-null int64
General Offense Number               32779 non-null int64
Offense Code                         32779 non-null object
Offense Code Extension               32779 non-null int64
Offense Type                         32779 non-null object
Summary Offense Code                 32779 non-null object
Summarized Offense Description       32779 non-null object
Date Reported                        32779 non-null object
Occurred Date or Date Range Start    32779 non-null object
Occurred Date Range End              16499 non-null object
Hundred Block Location               32779 non-null object
District/Sector                      32722 non-null object
Zone/Beat                            32723 non-null object
Census Tract 2000                    32645 non-null float64
Longitude                            32779 non-null float64
Latitud[...]<jupyter_text>The data to be analysed has few features, because this dataset have a lot of reducances. For example, the category of crime is created by spliting the values of Offence Type by the hyphen (-); the the Offence Type is droped. Then the columns names are lowercased and the spaces replaced by lower hyphens (_)_. The dates are all keeped, because it can be possible to plot the timespan of the crimes<jupyter_code>st = raw_data_st.copy()
st.drop(["General Offense Number","Offense Code","Offense Code Extension","Summarized Offense Description", "Location",
         "Summary Offense Code", "Hundred Block Location", 
         "Census Tract 2000"], axis=1, inplace=True)

st["Date Reported"] = pd.to_datetime(st["Date Reported"])
st.rename(columns={'Occurred Date or Date Range Start': 'Date Start', 'Occurred Date Range End': 'Date End'}, inplace=True)
st.columns = st.columns.str.strip().str.lower().str.replace(' ', '_')


st["category"] = st["offense_type"].str.split("-").str[0]
print(st.info())
st.drop(["offense_type"],axis=1,inplace=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 32779 entries, 0 to 32778
Data columns (total 12 columns):
rms_cdw_id         32779 non-null int64
offense_type       32779 non-null object
date_reported      32779 non-null datetime64[ns]
date_start         32779 non-null object
date_end           16499 non-null object
district/sector    32722 non-null object
zone/beat          32723 non-null object
longitude          32779 non-null float64
latitude           32779 non-null float64
month              32779 non-null int64
year               32779 non-null int64
category           32779 non-null object
dtypes: datetime64[ns](1), float64(2), int64(3), object(6)
memory usage: 3.0+ MB
None<jupyter_text>Count by time period  By hour of day<jupyter_code>sns.set_palette(sns.color_palette("pastel"))
sns.countplot(y=st.date_reported.dt.hour, order=range(0,24));
plt.ylabel("Hor of day");<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>Count by type of crimeThe plot is sort by count. It must be used a unique value to make the sort<jupyter_code>plt.figure(figsize=(11*(1+np.sqrt(5))/2,11))
sns.set_palette(sns.color_palette("pastel"))
ordVal = st.groupby("category")["rms_cdw_id"].describe().unstack().sort([('count')], ascending=False).index.values
sns.countplot(data=st, y="category", order=ordVal);<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
C:\Users\sadeus\Anaconda3\lib\site-packages\ipykernel\__main__.py:3: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....)
  app.launch_new_instance()
C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>Count of crimes by district/sector<jupyter_code>sns.set_palette(sns.color_palette("pastel"))
sns.countplot(data=st, y="district/sector");<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:892: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))<jupyter_text>Inter city comparizon  Count of crimes by type, weighted by the population of each city Population of each city taken from [Wikipedia](http://en.wikipedia.org)<jupyter_code>n_st = 608660 ## Seattle 
n_sf = 852469 ## San Francisco
plt.figure(figsize=(11*(1+np.sqrt(5))/2,11))
sns.set_palette(sns.color_palette("pastel"))
ordVal = st.groupby("category")["rms_cdw_id"].describe().unstack().sort([('count')], ascending=False).index.values
(st.groupby("category")["rms_cdw_id"].count().sort_values(ascending=True) / n_st).plot(kind="barh")

plt.figure()
ordVal = sf.groupby("Category")["PdId"].describe().unstack().sort([('count')], ascending=False).index.values
(sf.groupby("Category")["PdId"].count().sort_values(ascending=True) / n_st).plot(kind="barh")<jupyter_output>C:\Users\sadeus\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
C:\Users\sadeus\Anaconda3\lib\site-packages\ipykernel\__main__.py:3: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....)
  app.launch_new_instance()
C:\Users\sadeus\Anaconda3\lib\site-packages\ipykernel\__main__.py:7: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....)
{'fraction_non_alphanumeric': 0.11348684210526316, 'fraction_numerical': 0.03630169172932331, 'mean_word_length': 5.024416135881104, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '623214', 'license': 'MIT', 'n_tokens_mistral': 6158, 'n_tokens_neox': 5630, 'n_words': 1457, 'path': 'SF_Crimes_SSchiavinato.ipynb', 'repo_name': 'sadeus/datasci_course_assigment6', 'size': '17032'}