<jupyter_start><jupyter_text>I could not get the data to load properly from its URL because it is password-gated.  This was not an issue on the previous homework assignment.  I loaded it into jupyter, so that I could actually use it.  I know this isnt how we are supposed to do it, but I will try to figure it out later.  Frequency of reports by state:plots number of reports by state.  In the future, I would like to figure out how to format the plot so that all of the state names show up without being too crowded.<jupyter_code>bigfoot.state.value_counts().plot(kind='bar')
plt.xticks(np.arange(0, 50, step=5))
plt.xticks(rotation=90, horizontalalignment='center')<jupyter_output><empty_output><jupyter_text>Cleaning data - removing rows with no date value:Creates a new dataframe, df, that has removed all rows with null date values<jupyter_code>df = pd.DataFrame(bigfoot)
df['date'].replace('', np.nan, inplace=True)
df.dropna(subset=['date'], inplace=True)<jupyter_output><empty_output><jupyter_text>Creates a new column in the dataframe with the date values converted to datetime<jupyter_code>df['date_parsed'] = pd.to_datetime(df['date'], format = "%Y/%m/%d")
counts_by_year = df['date_parsed'].dt.year.value_counts()
counts_by_month = df['date_parsed'].dt.month.value_counts()<jupyter_output><empty_output><jupyter_text>Frequency of reports by yearTakes the number of reports for each year and plots frequency on a graph<jupyter_code>plt.plot(counts_by_year, "k+")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10046573519627412, 'fraction_numerical': 0.003992015968063872, 'mean_word_length': 6.4088669950738915, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '434126', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 490, 'n_tokens_neox': 478, 'n_words': 164, 'path': 'jnovak01/bigfoot.ipynb', 'repo_name': 'pezLyfe/applied_ds', 'size': '1503'}