<jupyter_start><jupyter_text>Udacity Data Analysis Nanodegree      Project: Wine Quality Dataset      <NAME>, June 2020   Table of ContentsData WranglingExploratory Data AnalysisConclusions<jupyter_code>#import modules
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
#display visuals in notebook
%matplotlib inline
#set theme for visualisations
sns.set_style('darkgrid')<jupyter_output><empty_output><jupyter_text>Data Wrangling General Properties<jupyter_code>#read in red_wine dataset
df_r = pd.read_csv(r"C:\Users\noama\OneDrive\My Documents\OneDrive\Python\Projects\Wine Quality\winequality-red.csv", sep=';')
#validate first 5 elements of red_wine dataset
df_r.head(5) 
#read in white_wine dataset
df_w = pd.read_csv(r"C:\Users\noama\OneDrive\My Documents\OneDrive\Python\Projects\Wine Quality\winequality-white.csv", sep=';')
#validate first 5 elements of white_wine dataset
df_w.head(5) 
#inspect number of columns and rows in the red_wine dataset
df_r.shape
#inspect name of columns in the red_wine dataset
df_r.columns
#inspect number of columns and rows in the white_wine dataset
df_w.shape
#inspect name of columns in the white_wine dataset
df_w.columns
#inspect data types of in the red_wine dataset
df_r.dtypes
#inspect data types of in the white_wine dataset
df_w.dtypes
#inspect summary information of ther red_wine dataset
df_r.info()
#inspect summary information of ther white_wine dataset
df_w.info()
#generate summary statistics of numerical columns of red_wine dataset
df_r.describe()
#generate summary statistics of numerical columns of white_wine dataset
df_w.describe()
#count number of unique values in red_wine dataset
df_r.nunique()
#count number of unique values in white_wine dataset
df_w.nunique()
#univarate analysis of varibales for red_wine dataset
df_r.hist(figsize=(8,8));
#univarate analysis of varibales for white_wine dataset
df_w.hist(figsize=(8,8));
#bivariate analysis of variables in the red_wine dataset
pd.plotting.scatter_matrix(df_r, figsize=(8,8));
#bivariate analysis of variables in the white_wine dataset
pd.plotting.scatter_matrix(df_w, figsize=(8,8));<jupyter_output><empty_output><jupyter_text>**Verify Data Quality**Examine the quality of the data, addressing questions such as:>1. Is the data complete (does it cover all the cases required)? >2. Is it correct, or does it contain errors and, if there are errors, how common are they?>3. Are there missing values in the data? If so, how are they represented, where do they occur, and how common are they? **Missing Data**In addition to incorrect datatypes, another common problem when dealing with real-world data is missing values. These can arise for many reasons and have to be either filled in or removed before we train a machine learning model. First, let’s get a sense of how many missing values are in each column.<jupyter_code>#create user define function to capture number of missing values
def missing_values(df):
    miss_val = df.isnull().sum()
    #print(miss_val)
    miss_val_percent = 100 * df.isnull().sum() / len(df)
    #print(miss_val_percent)
    miss_val_table = pd.concat([miss_val, miss_val_percent], axis=1)
    #print(miss_val_table)
    miss_val_table_rename_col = miss_val_table.rename(columns = {0 : 'Missing Values', 1 : '% of Total Values'})
    #print(miss_val_table_rename_col)
    miss_val_table_sorted = miss_val_table_rename_col[miss_val_table_rename_col.iloc[:,1] != 0].sort_values('% of Total Values', ascending=False).round(1)
    #print(miss_val_table_sorted)
    print ("Your selected dataframe has " + str(df.shape[1]) + " columns.\n"
           "There are " + str(miss_val_table_sorted.shape[0]) +
           " columns that have missing values.")
    return miss_val_table_sorted
#apply misisng values function to red_wine dataset
missing_values(df_r)
#apply misisng values function to white_wine dataset
missing_values(df_w)
#inspect number of duplicates in red_wine dataset
sum(df_r.duplicated())
#inspect number of duplicates in white_wine dataset
sum(df_w.duplicated())
#How many samples and variables are there in the red wine dataset?
df_r.shape
#How many samples and variables are there in the white wine dataset?
df_w.shape
#which features have missing values in the red wine dataset?
df_r.isnull().sum()
#which features have missing values in the white wine dataset?
df_w.isnull().sum()<jupyter_output><empty_output><jupyter_text>Duplicates There may be duplicates in the data. However, these may be legitimate new rows depending on the structure of the data. We need to discover them, then decide what to do with them.<jupyter_code>#how many duplicate records are there in the red wine dataset?
df_r.duplicated().sum()
#how many duplicate records are there in the white wine dataset?
df_w.duplicated().sum()
#how many unique values of quality are there in the red wine dataset?
df_r.nunique()
#how many unique values of quality are there in the white wine dataset?
df_w.nunique()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>#what is the mean density of the red wine dataset?
df_r['density'].mean()
##is a certain type of wine (red or white) associated with higher quality?
#Column name differences between the files, so change to a matching name
df_r = df_r.rename(columns = {'total_sulfur-dioxide':'total_sulfur_dioxide'})

# create color array for red dataframe 
color_red = np.repeat('red', df_r.shape[0]) 
# create color array for white dataframe 
color_white = np.repeat('white', df_w.shape[0])

#assign newly created coloured arrays to respective dataframes
df_r['color'] = color_red 
df_w['color'] = color_white

#combine red wine dataset with white wine dataset
wine_df = df_r.append(df_w) 
#inspect first five rows of merged dataframe
wine_df.head()
#rename column
wine_df = wine_df.rename(columns = {'residual sugar':'residual_sugar'})
#univarate analysis of varibales of merged dataset
wine_df.hist(figsize=(8,8));
#bivariate analysis of variables in the merged dataset
pd.plotting.scatter_matrix(wine_df, figsize=(8,8));
# Find the mean quality of each wine type (red and white) with groupby 
wine_df.groupby('color').quality.mean()
##what level of acidity (PH value) receives the highest average rating?
wine_df.pH.describe()
# Bin edges that will be used to "cut" the data into groups 
bin_edges = [2.72, 3.11, 3.21, 3.32, 4.01]
# Labels for the four acidity level groups 
bin_names = ['high', 'mod_high', 'medium', 'low']
# Creates acidity_levels column 
wine_df['acidity_levels'] = pd.cut(wine_df['pH'], bin_edges, labels=bin_names)
# Checks for successful creation of this column 
wine_df.head()
# Find the mean quality of each acidity level with groupby 
wine_df.groupby('acidity_levels').quality.mean()
##Do wines with higher alcohol content receive higher ratings?
median = wine_df['alcohol'].median() 

low = wine_df.query('alcohol < {}'.format(median)) 
high = wine_df.query('alcohol >= {}'.format(median))

mean_quality_low = low['quality'].mean()
mean_quality_high = high['quality'].mean()
#create bar chart with labels of preceeding analysis
locations = [1, 2]
heights = [mean_quality_low, mean_quality_high]
labels = ['Low', 'High']
plt.bar(locations, heights, tick_label=labels)
plt.title('Average Quality Ratings by Alcohol Content')
plt.xlabel('Alcohol Content')
plt.ylabel('Average Quality Rating');
##do sweeter wines (more resiudal sugar) recieve higher ratings?
# Use query to select each group and get its mean quality 
median = wine_df['residual_sugar'].median()

low = wine_df.query('residual_sugar < {}'.format(median))
high = wine_df.query('residual_sugar >= {}'.format(median)) 

mean_quality_low = low['quality'].mean()
mean_quality_high = high['quality'].mean()
#create bar chart with labels of preceeding analysis
locations = [1, 2]
heights = [mean_quality_low, mean_quality_high]
labels = ['Low', 'High']
plt.bar(locations, heights, tick_label=labels)
plt.title('Average Quality Ratings by Residual Sugar')
plt.xlabel('Residual Sugar')
plt.ylabel('Average Quality Rating')
##what level of acidity receives the highest average rating?
acidity_level_quality_means = wine_df.groupby('acidity_levels').mean().quality 
acidity_level_quality_means
# reorder values above to go from low to high 
locations = [4, 1, 2, 3]  
heights = acidity_level_quality_means 

# labels = ['Low', 'Medium', 'Moderately High', 'High'] 
labels = acidity_level_quality_means.index.str.replace('_', ' ').str.title() # alternative to commented out line above 

#create bar chart with labels for title and axis 
plt.bar(locations, heights, tick_label=labels)
plt.title('Average Quality Ratings by Acidity Level')
plt.xlabel('Acidity Level') 
plt.ylabel('Average Quality Rating');
# get counts for each rating and color 
color_counts = wine_df.groupby(['color', 'quality']).count()['pH'] 
color_counts 
#get count by color
color_totals = wine_df.groupby('color').count()['pH'] 
color_totals
# get proportions by dividing red rating counts by total # of red samples 
red_proportions = color_counts['red'] / color_totals['red'] 
red_proportions
#create additional layer of quality
red_proportions['9'] = 0 
red_proportions
# get proportions by dividing white rating counts by total # of white samples 
white_proportions = color_counts['white'] / color_totals['white'] 
white_proportions
# the x locations for the groups 
ind = np.arange(len(red_proportions))  

# the width of the bars
width = 0.35
# plot bars 
red_bars = plt.bar(ind, red_proportions, width, color='r', alpha=.7, label='Red Wine') 
white_bars = plt.bar(ind + width, white_proportions, width, color='w', alpha=.7, label= 'White Wine') 

# title and labels 
plt.ylabel('Proportion') 
plt.xlabel('Quality') 
plt.title('Proportion by Wine Color and Quality') 

# xtick locations
locations = ind + width / 2  
# xtick labels 
labels = ['3', '4', '5', '6', '7', '8', '9']  
plt.xticks(locations, labels) 

# legend 
plt.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10108303249097472, 'fraction_numerical': 0.007420778178900923, 'mean_word_length': 5.7476319350473615, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '418167', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3284, 'n_tokens_neox': 3101, 'n_words': 1066, 'path': 'Introduction/Wine_Quality_Dataset/Data Analysis -- Wine Quality Dataset .ipynb', 'repo_name': 'NMangera/Udacity_DataAnalysis_Nanodegree', 'size': '9980'}