<jupyter_start><jupyter_text>> First, I made a mistake naming the data set! **It's 2015 data, not 2014 data.** But yes, still use `311-2014.csv`. You can rename it. Importing and preparing your dataImport your data, but **only the first 200,000 rows**. You'll also want to change the index to be a datetime based on the **Created Date** column - you'll want to check if it's already a datetime, and parse it if not.<jupyter_code>import datetime
import datetime as dt

dt.datetime.strptime('07/06/2015 10:58:27 AM', '%m/%d/%Y %I:%M:%S %p')
#datetime.datetime(2015, 7, 6, 0, 0)
parser = lambda date: pd.datetime.strptime(date, '%m/%d/%Y %H:%M:%S')
df = pd.read_csv("311-2015.csv", low_memory=False, parse_dates=[1], dtype=str , nrows=200000)
df.info()
df.index = df['Created Date']
del df['Created Date']
df.head(2)<jupyter_output><empty_output><jupyter_text>What was the **most popular type of complaint**, and how many times was it filed?<jupyter_code>df['Complaint Type'].value_counts().head(1)<jupyter_output><empty_output><jupyter_text>Make a horizontal bar graph of the **top 5 most frequent complaint types**.<jupyter_code>ax = df['Complaint Type'].value_counts().sort_values(ascending=True).tail(5).plot(kind='barh', figsize=(6,4), fontsize=9)
ax.set_title("Top 5 Most Frequent 311-Complaints in 2015")
ax.set_xlabel("Complaint Count")
ax.set_ylabel("Complaint Type")
plt.savefig("5 Most Frequent 311 Complaints in 2015.svg", bbox_inches='tight')
#http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html<jupyter_output><empty_output><jupyter_text>Which borough has the **most complaints per capita?** Since it's only 5 boroughs, you can do the math manually.<jupyter_code>df_summed_complaints = df['Borough'].value_counts()
summed_complaints = pd.DataFrame(df_summed_complaints)
summed_complaints.reset_index(inplace=True)
summed_complaints.columns = ['Borough', 'Complaint Count']
Borough_head_count = pd.read_csv("NYC_Boroughs.csv")
summed_complaints_merged = summed_complaints.merge(Borough_head_count, left_on='Borough', right_on='borough name')
del summed_complaints_merged['borough name']
summed_complaints_merged
summed_complaints_merged['Per Capita'] = summed_complaints_merged['Total'] / summed_complaints_merged['Complaint Count']
summed_complaints_merged['Per Capita'].sort_values(ascending=False)
summed_complaints_merged[['Borough', 'Per Capita']]
Sorted_complaints = summed_complaints_merged.sort_values(by='Per Capita', ascending=False)
Sorted_complaints<jupyter_output><empty_output><jupyter_text>According to your selection of data, **how many cases were filed in March?** How about May?<jupyter_code>df['2015-03']['Unique Key'].count()
df['2015-05']['Unique Key'].count()<jupyter_output><empty_output><jupyter_text>I'd like to see all of the 311 complaints **called in on April 1st.**> **Surprise!** We couldn't do this in class, but it was just a limitation of our data set<jupyter_code>df['2015-04-01']['Unique Key'].count()<jupyter_output><empty_output><jupyter_text>What was the most popular type of complaint on April 1st?<jupyter_code>df['2015-04-01']['Complaint Type'].value_counts().head(1)<jupyter_output><empty_output><jupyter_text>What were the **most popular three types of complaint** on April 1st<jupyter_code>df['2015-04-01']['Complaint Type'].value_counts().head(3)<jupyter_output><empty_output><jupyter_text>**What month has the most reports filed?** How many? Graph it.<jupyter_code>#PANDAS resample: http://stackoverflow.com/questions/17001389/pandas-resample-documentation/17001474#17001474
ax = df.resample('M')['Unique Key'].count().plot(kind='barh')
#ax.set_yticks(['2015-01-31 00:00:00, 2015-02-28 00:00:00, 2015-03-30 00:00:00, 2015-04-30 00:00:00, 2015-05-31 00:00:00, 2015-06-30 00:00:00, 2015-07-31 00:00:00, 2015-08-31 00:00:00, 2015-09-30 00:00:00, 2015-10-31 00:00:00, 2015-11-30 00:00:00, 2015-12-31 00:00:00'])
ax.set_yticklabels(['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
ax.set_title('Number of Complaints per Month')
ax.set_ylabel('Months of the Year')
ax.set_xlabel('Complaint Counts')<jupyter_output><empty_output><jupyter_text>**What week of the year has the most reports filed?** How many? Graph the weekly complaints.<jupyter_code>df_week_count = df.resample('W')['Unique Key'].count()
df_week_count = pd.DataFrame(df_week_count)
df_week_count.sort_values(by='Unique Key', ascending=False).head(5)<jupyter_output><empty_output><jupyter_text>Noise complaints are a big deal. Use `.str.contains` to select noise complaints, and make an chart of when they show up annually. **Then** make a chart about when they show up every day (cyclic).<jupyter_code>df['Noise'] = df['Complaint Type'].str.contains('Noise')
df_noise = df[df['Noise'] == True]
ax = df_noise.resample('D')['Unique Key'].count().plot(kind='bar', figsize=(15,4))
ax.set_xticklabels('')
ax.set_title('Number of Noise Complaints per Day in 2015')
ax.set_ylabel('Noise Complaint Count')
ax.set_xlabel('From January - December 2015 ')
df_noise.groupby(by=df_noise.index.hour)['Unique Key'].count().plot(kind='bar', figsize=(12,6))<jupyter_output><empty_output><jupyter_text>**Which were the top five days of the year for filing complaints?** How many on each of those days? Graph it.<jupyter_code>df_day_count = df.resample('D')['Unique Key'].count()
df_day_count = pd.DataFrame(df_day_count)
ax = df_day_count.sort_values(by='Unique Key', ascending=True).tail(5).plot(kind='barh', legend=False)
ax.set_yticklabels(['10. Oktober', '6. August', '3. August', '25. September', '19. Oktober'])
ax.set_title('Top five complaint days in 2015')
ax.set_ylabel('')
ax.set_xlabel('Complaint Counts')
plt.savefig("Top 5 Complaint Days", bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>**What hour of the day are the most complaints?** Graph a day of complaints.<jupyter_code>ax = df.groupby(by=df.index.hour)['Unique Key'].count().plot(kind='bar', figsize=(12,6))
ax.set_title('Number of Complaints per hour in 2015')<jupyter_output><empty_output><jupyter_text>One of the hours has an odd number of complaints. What are the most common complaints at that hour, and what are the most common complaints the hour before and after?<jupyter_code>df.groupby(by=df.index.hour)['Complaint Type'].value_counts().head(2)
df_complaint_type_count_per_hour = df.groupby(by=df.index.hour)['Complaint Type'].value_counts()
Top_complaints_by_hour = pd.DataFrame(df_complaint_type_count_per_hour)
Top_complaints_by_hour['Complaint Type'][0].head(1)
Top_complaints_by_hour['Complaint Type'][1].head(1)
Top_complaints_by_hour['Complaint Type'][23].head(1)
#More Reading: http://pandas.pydata.org/pandas-docs/version/0.13.1/timeseries.html<jupyter_output><empty_output><jupyter_text>So odd. What's the **per-minute breakdown** of complaints between 12am and 1am? You don't need to include 1am.<jupyter_code>df.groupby(by=df.index.hour == 0)['Unique Key'].count()<jupyter_output><empty_output><jupyter_text>Looks like midnight is a little bit of an outlier. Why might that be? Take the 5 most common agencies and graph the times they file reports at (all day, not just midnight).<jupyter_code>df['Agency'].value_counts().head(5)
df['Agency Name'].value_counts().head(5)
df_NYPD = df[df['Agency Name'] == 'New York City Police Department']
df_NYPD.groupby(by=df_NYPD.index.hour)['Unique Key'].count().plot(kind='bar')
df_HPD = df[df['Agency Name'] == 'Department of Housing Preservation and Development']
df_HPD.groupby(by=df_HPD.index.hour)['Unique Key'].count().plot(kind='bar')
df_DOT = df[df['Agency Name'] == 'Department of Transportation']
df_DOT.groupby(by=df_DOT.index.hour)['Unique Key'].count().plot(kind='bar')
df_DPR = df[df['Agency Name'] == 'Department of Parks and Recreation']
df_DPR.groupby(by=df_DPR.index.hour)['Unique Key'].count().plot(kind='bar')
df_DOHMH = df[df['Agency Name'] == 'Department of Health and Mental Hygiene']
df_DOHMH.groupby(by=df_DOHMH.index.hour)['Unique Key'].count().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>df_NYPD = df[df['Agency Name'] == 'New York City Police Department']df_NYPD.groupby(by=df_NYPD.index.hour)['Unique Key'].count().plot(kind='bar')  Dont understand why I can't change the labels here:<jupyter_code>ax = df[df['Agency Name'] == 'New York City Police Department'].groupby(by=df_NYPD.index.hour)['Unique Key'].count().plot(kind='bar', stacked=True, figsize=(15,6))
df[df['Agency Name'] == 'Department of Housing Preservation and Development'].groupby(by=df_HPD.index.hour)['Unique Key'].count().plot(kind='bar', ax=ax, stacked=True, color='lightblue')
df[df['Agency Name'] == 'Department of Transportation'].groupby(by=df_DOT.index.hour)['Unique Key'].count().plot(kind='bar', ax=ax, stacked=True, color='purple')
df[df['Agency Name'] == 'Department of Health and Mental Hygiene'].groupby(by=df_DOHMH.index.hour)['Unique Key'].count().plot(kind='bar', ax=ax, stacked=True, color='grey')
df[df['Agency Name'] == 'Department of Parks and Recreation'].groupby(by=df_DPR.index.hour)['Unique Key'].count().plot(kind='bar', ax=ax, stacked=True, color='green')
ax.set_title('Time of Day Agencies file complaints')
ax.set_ylabel('Complaint Count')
ax.set_xlabel("""
              Red: New York City Police Department
              Blue: Department of Housing Preservation and Development
              Purple: Department of Transportation
              Grey: Department of Health and Mental Hygiene
              Green: Department of Parks and Recreation""")
plt.savefig("Time of Day Agencies File Complaints.svg", bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>Graph those same agencies on an annual basis - make it **weekly**. When do people like to complain? When does the NYPD have an odd number of complaints?<jupyter_code>ax = df[df['Agency Name'] == 'New York City Police Department'].resample('W')['Agency'].count().plot(figsize=(15,4), linewidth=3)
df[df['Agency Name'] == 'Department of Housing Preservation and Development'].resample('W')['Agency'].count().plot(color='lightblue', linewidth=2)
df[df['Agency Name'] == 'Department of Transportation'].resample('W')['Agency'].count().plot(color='purple', linewidth=2)
df[df['Agency Name'] == 'Department of Health and Mental Hygiene'].resample('W')['Agency'].count().plot(color='grey', linewidth=2)
df[df['Agency Name'] == 'Department of Parks and Recreation'].resample('W')['Agency'].count().plot(color='green', linewidth=2)
ax.set_xticklabels(['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
ax.set_title('When the Angencies Filed the Reports 2015')
ax.set_xlabel("""
              Red: New York City Police Department
              Blue: Department of Housing Preservation and Development
              Purple: Department of Transportation
              Grey: Department of Health and Mental Hygiene
              Green: Department of Parks and Recreation""")
plt.savefig("When the Angencies Filed the Reports 2015.svg", bbox_inches='tight')
ax = df_NYPD.resample('W')['Agency'].count().plot(kind='bar', figsize=(15,4))
ax.set_xticklabels([''])
ax.set_xlabel('January to December 2015')<jupyter_output><empty_output><jupyter_text>Maybe the NYPD deals with different issues at different times? Check the most popular complaints in July and August vs the month of May. Also check the most common complaints for the Housing Preservation Bureau (HPD) in winter vs. summer.<jupyter_code>df['2015-08']['Complaint Type'].value_counts().head(3)
df['2015-07']['Complaint Type'].value_counts().head(3)
df['2015-05']['Complaint Type'].value_counts().head(3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1573159206117364, 'fraction_numerical': 0.036085574362058594, 'mean_word_length': 7.044229440221147, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '347934', 'license': 'MIT', 'n_tokens_mistral': 4325, 'n_tokens_neox': 3878, 'n_words': 942, 'path': '13/311 time series homework.ipynb', 'repo_name': 'barjacks/foundations-homework', 'size': '11639'}