<jupyter_start><jupyter_text>Next thing I would be curious about is the actual distribution of data.. let's explore the distribution of styles of beer. We'll do this by getting a high level summary. This will include a mean and a standard deviation then to that we will crunch a Z-Score and use that metric to graph our results for easier visualization.<jupyter_code>from __future__ import print_function
# setup the essentials
import pandas as pd
import numpy as np

# load the data and take a quick peek
beer_all_data = pd.read_csv("../data/beer_reviews.csv")

# store off details by beer style
beer_style_details = beer_all_data.groupby('beer_style').agg({
       'brewery_name': pd.Series.nunique,
       'review_profilename': pd.Series.nunique,
       'beer_name': pd.Series.nunique
    }).rename(columns={
                        'brewery_name' : 'number_of_distinct_brewerys',
                        'review_profilename' : 'number_of_distinct_reviewers',
                        'beer_name' : 'number_of_distinct_beer'                          
                      })

# store off summary by beer styles
beer_style_sum = beer_style_details.agg({
        'number_of_distinct_brewerys' : ['mean', 'std'],
        'number_of_distinct_reviewers' : ['mean', 'std'],
        'number_of_distinct_beer' : ['mean', 'std']    
    })

# store off high level summary across all beer styles
beer_style_tots = beer_all_data.agg({
       'brewery_name': pd.Series.nunique,
       'review_profilename': pd.Series.nunique,
       'beer_name': pd.Series.nunique
    }).rename(columns={
                        'brewery_name' : 'number_of_distinct_brewerys',
                        'review_profilename' : 'number_of_distinct_reviewers',
                        'beer_name' : 'number_of_distinct_beer'                          
                      })

# some basic conversions for later
#beer_style_sum = pd.DataFrame(beer_style_sum).reset_index()
#-----------------------------------------------------------
# some updates to the detail table from the sum and tots
#-----------------------------------------------------------

#--------- Brewery
beer_style_details['total_brewerys'] = beer_style_tots['brewery_name']
beer_style_details['pct_brewerys'] = beer_style_details['number_of_distinct_brewerys'] / beer_style_details['total_brewerys']
beer_style_details['mean_distinct_brewerys'] = beer_style_sum.loc['mean','number_of_distinct_brewerys']
beer_style_details['sd_distinct_brewerys'] = beer_style_sum.loc['std','number_of_distinct_brewerys']

beer_style_details['Z_distinct_brewerys'] = (
                                                (beer_style_details['number_of_distinct_brewerys'] - beer_style_details['mean_distinct_brewerys'])
                                                    / beer_style_details['sd_distinct_brewerys']
                                                )

#--------- Reviewers
beer_style_details['total_reviewers'] = beer_style_tots['review_profilename']
beer_style_details['pct_reviewers'] = beer_style_details['number_of_distinct_reviewers'] / beer_style_details['total_reviewers']
beer_style_details['mean_distinct_reviewers'] = beer_style_sum.loc['mean','number_of_distinct_reviewers']
beer_style_details['sd_distinct_reviewers'] = beer_style_sum.loc['std','number_of_distinct_reviewers']

beer_style_details['Z_distinct_reviewers'] = (
                                                (beer_style_details['number_of_distinct_reviewers'] - beer_style_details['mean_distinct_reviewers'])
                                                    / beer_style_details['sd_distinct_reviewers']
                                                )

#--------- Beer
beer_style_details['total_beer'] = beer_style_tots['beer_name']
beer_style_details['pct_Beer'] = beer_style_details['number_of_distinct_beer'] / beer_style_details['total_beer']
beer_style_details['mean_distinct_beer'] = beer_style_sum.loc['mean','number_of_distinct_beer']
beer_style_details['sd_distinct_beer'] = beer_style_sum.loc['std','number_of_distinct_beer']

beer_style_details['Z_distinct_beer'] = (
                                                (beer_style_details['number_of_distinct_beer'] - beer_style_details['mean_distinct_beer'])
                                                    / beer_style_details['sd_distinct_beer']
                                                )

beer_style_details.head()

# Save this off to csv for later consumption
beer_style_details.to_csv("../data/beer_style_summary.csv")<jupyter_output><empty_output><jupyter_text>Now for some graphs<jupyter_code># import the goodies needed
%matplotlib inline
import seaborn as sns
# Brewery Distribution across styles of beer
print('Brewery Distribution\n')
sns.distplot(beer_style_details['Z_distinct_brewerys'])
# Acutal Distinct Reviewer Distribution across styles of beer
print('\nReviewer\n')
sns.distplot(beer_style_details['Z_distinct_reviewers'])
# Distinct Beer Distribution across styles of beer
print('\nBeer\n')
sns.distplot(beer_style_details['Z_distinct_beer'])
# Put them all together
sns.kdeplot(beer_style_details['Z_distinct_brewerys'])
sns.kdeplot(beer_style_details['Z_distinct_reviewers'])
sns.kdeplot(beer_style_details['Z_distinct_beer'])<jupyter_output><empty_output><jupyter_text>Observations so farFrom this it would seem all of our scales are positively skewed - but that's a good thing - right?  Time will tell if we take advantage of this by creating some subsets of certain styles of beer - perhaps heavily rated styles (Z-Score over 3) or lightly rate (Z-Score under -1).  Further we could do similar datasets for the other categories within this dataset (Reviewer, Beer) So what's nextDo the same analysis against reviewer_profilename<jupyter_code>beer_style_details = pd.read_csv("../data/beer_style_summary.csv")
beer_style_details.head()
beer_all_data.head()
# store off details by beer style
beer_reviewer_details = beer_all_data.groupby('review_profilename').agg({
       'brewery_name': pd.Series.nunique,
       'beer_name': pd.Series.nunique
    }).rename(columns={
                        'brewery_name' : 'number_of_distinct_brewerys',
                        'beer_name' : 'number_of_distinct_beer'                          
                      })

# store off summary by beer styles
beer_reviewer_sum = beer_reviewer_details.agg({
        'number_of_distinct_brewerys' : ['mean', 'std'],
        'number_of_distinct_beer' : ['mean', 'std']    
    })

# store off high level summary across all beer styles
beer_reviewer_tots = beer_all_data.agg({
       'brewery_name': pd.Series.nunique,
       'beer_name': pd.Series.nunique
    }).rename(columns={
                        'brewery_name' : 'number_of_distinct_brewerys',
                        'beer_name' : 'number_of_distinct_beer'                          
                      })
beer_reviewer_details.head()
#-----------------------------------------------------------
# some updates to the detail table from the sum and tots
#-----------------------------------------------------------

#--------- Brewery
beer_reviewer_details['total_brewerys'] = beer_reviewer_tots['brewery_name']
beer_reviewer_details['pct_brewerys'] = beer_reviewer_details['number_of_distinct_brewerys'] / beer_reviewer_details['total_brewerys']
beer_reviewer_details['mean_distinct_brewerys'] = beer_reviewer_sum.loc['mean','number_of_distinct_brewerys']
beer_reviewer_details['sd_distinct_brewerys'] = beer_reviewer_sum.loc['std','number_of_distinct_brewerys']

beer_reviewer_details['Z_distinct_brewerys'] = (
                                                (beer_reviewer_details['number_of_distinct_brewerys'] - beer_reviewer_details['mean_distinct_brewerys'])
                                                    / beer_reviewer_details['sd_distinct_brewerys']
                                                )

#--------- Beer
beer_reviewer_details['total_beer'] = beer_reviewer_tots['beer_name']
beer_reviewer_details['pct_Beer'] = beer_reviewer_details['number_of_distinct_beer'] / beer_reviewer_details['total_beer']
beer_reviewer_details['mean_distinct_beer'] = beer_reviewer_sum.loc['mean','number_of_distinct_beer']
beer_reviewer_details['sd_distinct_beer'] = beer_reviewer_sum.loc['std','number_of_distinct_beer']

beer_reviewer_details['Z_distinct_beer'] = (
                                                (beer_reviewer_details['number_of_distinct_beer'] - beer_reviewer_details['mean_distinct_beer'])
                                                    / beer_reviewer_details['sd_distinct_beer']
                                                )

beer_reviewer_details.head()
beer_reviewer_details.describe()
# Save this off to csv for later consumption
beer_reviewer_details.to_csv("../data/beer_reviewer_summary.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12270356089816285, 'fraction_numerical': 0.00022680880018144704, 'mean_word_length': 2.8696796840719614, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '164206', 'license': 'MIT', 'n_tokens_mistral': 2647, 'n_tokens_neox': 2614, 'n_words': 440, 'path': 'notebooks/2 - Additional Feature Exploration.ipynb', 'repo_name': 'tomdynares/BeerRatings', 'size': '8818'}