<jupyter_start><jupyter_text>Crime data visualization in San FranciscoSan Francisco has one of the most "open data" policies of any large city. In this lab, we are going to download about 85M of data (238,456) describing all police incidents since 2018 (I'm grabbing data on August 5, 2019). Getting startedDownload [Police Department Incident Reports 2018 to present](https://data.sfgov.org/Public-Safety/Police-Department-Incident-Reports-2018-to-Present/wg3w-h783) or, if you want, all [San Francisco police department incident since 1 January 2003](https://data.sfgov.org/Public-Safety/SFPD-Incidents-from-1-January-2003/tmnf-yvry). Click the "Export" button and then save in "CSV for Excel" format. (It's fairly large at all 140MB so it could take a while if you have a slow connection.)We can easily figure out how many records there are:```bash$ wc -l Police_Department_Incident_Reports__2018_to_Present.csv   388670 Police_Department_Incident_Reports__2018_to_Present.csv```So 388,669 not including the header row.  You can name that data file whatever you want but I will call it `SFPD.csv` for these exercises and save it in `/tmp`.  Sniffing the dataLet's assume the file you downloaded and is in `/tmp`:<jupyter_code>import pandas as pd

df_sfpd = pd.read_csv('/tmp/SFPD.csv')
df_sfpd.head(10).T<jupyter_output><empty_output><jupyter_text>To get a better idea of what the data looks like, let's do a simple histogram of the categories and crime descriptions.  Here is the category histogram:<jupyter_code>df_sfpd['Incident Category'].unique()
from collections import Counter
counter = Counter(df_sfpd['Incident Category'])
counter.most_common(10)
from collections import Counter
counter = Counter(df_sfpd['Incident Description'])
counter.most_common(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08356545961002786, 'fraction_numerical': 0.03788300835654596, 'mean_word_length': 5.828897338403042, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '450749', 'license': 'MIT', 'n_tokens_mistral': 609, 'n_tokens_neox': 519, 'n_words': 202, 'path': 'notes/sfpd.ipynb', 'repo_name': 'Victor-Palacios/msds692', 'size': '1795'}