<jupyter_start><jupyter_text>Why It's Important to Explore Your Data A Cautionary Tale Let's do some imports. Make sure you have all this stuff installed.<jupyter_code>import seaborn as sns
import pandas as pd
import feather<jupyter_output><empty_output><jupyter_text>Read in the feather file to a pandas dataframe. If something goes wrong, it's probably a path issue. This notebook should be running from the "notebooks" folder, so you should need to go up one level, into /data, and find /beer-info.feather.<jupyter_code>df = feather.read_dataframe('../data/beer-info.feather')
df.head()<jupyter_output><empty_output><jupyter_text>Well that's a good start, reasonable columns and all. How much data do we have?<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>Also reasonable. Let's check types:<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 700 entries, 0 to 699
Data columns (total 10 columns):
abv            695 non-null float64
brewery        700 non-null object
date           700 non-null datetime64[ns]
description    700 non-null object
ibu            489 non-null float64
id             700 non-null int64
name           700 non-null object
rating         700 non-null float64
style          700 non-null object
num ratings    700 non-null int64
dtypes: datetime64[ns](1), float64(3), int64(2), object(4)
memory usage: 54.8+ KB<jupyter_text>Hmm, I wonder if we have correlation between some of the numerical columns.<jupyter_code>df[['abv','ibu','num ratings','rating']].corr()<jupyter_output><empty_output><jupyter_text>ABV seems like it might correlate to rating, which isn't terribly surprising. Is there a trend over time?<jupyter_code>%matplotlib inline
fig = df.plot('date','rating',marker='o',linestyle='None')
sns.regplot('date','rating',df)<jupyter_output><empty_output><jupyter_text>Jury is still out. Maybe one of those fancy seaborn plots will be enlightening.<jupyter_code>%matplotlib inline
fig = sns.pairplot(df)<jupyter_output>C:\Users\Brian\Anaconda3\envs\cicerone\lib\site-packages\numpy\lib\histograms.py:754: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
C:\Users\Brian\Anaconda3\envs\cicerone\lib\site-packages\numpy\lib\histograms.py:755: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>Nice, we've got a bunch of scatter plots and distributions. ABV and rating even look like normal distributions! Num ratings tapers off quick, but I guess that isn't too exciting. And IBU looks... wait what? Everything is at zero? Has the Zero IBU craze really gone that nuts...oh wait, the max value is...wtf?<jupyter_code>df.ibu.mean()<jupyter_output><empty_output><jupyter_text>That's....insane...<jupyter_code>df.ibu.max()<jupyter_output><empty_output><jupyter_text>WHOSE RESPONSIBLE THIS????<jupyter_code>df[df.ibu == df.ibu.max()]<jupyter_output><empty_output><jupyter_text>OK, what's the deal here?<jupyter_code>print(df[df.ibu == df.ibu.max()].description.values[0])<jupyter_output>Like the dirty neck beard it self this hazy IPA our take on a New England style IPA is unfiltered and dirty. It smells of orange, peach and melon. First sips are orangy tapering off to a nice bitter grapefruity finish.
{'fraction_non_alphanumeric': 0.10390006093845217, 'fraction_numerical': 0.022242535039609993, 'mean_word_length': 5.325626204238921, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '291875', 'license': 'MIT', 'n_tokens_mistral': 1150, 'n_tokens_neox': 1064, 'n_words': 322, 'path': "notebooks/Why It's Important to Explore Your Data.ipynb", 'repo_name': 'bcminch/cicerone', 'size': '3282'}