<jupyter_start><jupyter_text>Business Understanding:1. Is white wine or red wine better in quality?2. What properties are different between Vinho Verde white and red wines?3. Can we predict if it is Vinho Verde white wine or red wine based on its properties? Data Understanding:1. Get quality Distribution of Red and White wine, and also use Kendall's Rank Correlation to see if there is signifiant quality difference.2. Examine chemical properties of red and white wines, and use t-test or Wilcoxon-Mann-Whitney test to compare these properties between red and white wines.3. Based on chemical properties, select features and build a logistic regression model to predict if the wine is red wine or white wine.  Downloaded winequality_red.csv and winequality_white.csv from https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
!pip install researchpy
import researchpy as rp
import matplotlib.pyplot as plt
import sys
import scipy.stats as stats
from scipy.stats import ttest_ind
import statsmodels.api as sm
pd.set_option('display.max_columns', None)
from sklearn.metrics import f1_score
from statsmodels.formula.api import logit
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score, accuracy_score, confusion_matrix
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>Quic Look at the data (red wine and white wine datasets) for completeness and correlations<jupyter_code>white = pd.read_csv("winequality-white.csv", delimiter=';')
#Summary of columns
print(white.describe())    
#Check completeness of data   
print(white.isna().mean())    
red=pd.read_csv("winequality-red.csv", delimiter=';')
#Summary of columns
print(red.describe())        
#Check completeness of data
print(red.isna().mean())      
#See correlation of properties in white wine
sns.heatmap(white.corr())
#See correlation of properties in red wine
sns.heatmap(red.corr())<jupyter_output><empty_output><jupyter_text>1. Quality difference between white and red wines<jupyter_code>#Combine datasets of white wine and red wine
white['Type']= "white"
red['Type'] ="red"
dataall=red.append(white)
print(white.shape)
print(red.shape)
print(dataall.shape)<jupyter_output>(4898, 13)
(1599, 13)
(6497, 13)<jupyter_text>There are 4898 white wine entries and 1599 red wine entries<jupyter_code>#Density plot for quality of wines
qualitybyType=dataall.pivot(columns='Type',values='quality')
qualitybyType.plot.density()
plt.xlabel('Quality')
plt.title('Wine Quality')
#Boxplot for quality of wines 
bp = sns.boxenplot(x = "Type", y = "quality", hue ="Type",  
              data = dataall) 
plt.legend(bbox_to_anchor=(1.05, 1.0), loc='upper left')
plt.title('Wine Quality by Type')
plt.show()<jupyter_output><empty_output><jupyter_text>Measure Kendall's Rank Correlation between qualities of red wine and white wine:Null hypothesis: There is no quality difference between red wine and white wineAlternate hypothesis: There is quality difference between red wine and white wine<jupyter_code>crosstab = pd.crosstab(dataall['Type'],dataall['quality'])
print(crosstab)
stats.kendalltau(crosstab.iloc[0],crosstab.iloc[1])<jupyter_output>quality   3    4     5     6    7    8  9
Type                                     
red      10   53   681   638  199   18  0
white    20  163  1457  2198  880  175  5<jupyter_text>With p value < 0.05, we reject the null hypothese that there is no quality difference between red wine and white wine. Based on previous boxplots, we conclude that white wines generally have higher quality.  Property Discrepancies between White wine and Red wine<jupyter_code>#Histogram and density plots of all the properties of white and red wines
for i in dataall.columns.drop(['Type']):
  print(i)
  dataall[i].hist(by=dataall["Type"])
  dataall.pivot(columns='Type',values=i).plot.density()
  plt.xlabel(i)
  plt.title(i)
  plt.show()<jupyter_output>fixed acidity<jupyter_text>As we see, many of the distributions for the properties of the wines are not normally distributed, thus to identify properties that are significantly different between white and red wines, we want to see results of non-parametric Wilcoxon-Mann-Whitney test aside from t-test results.<jupyter_code>#Wilcoxon-Mann-Whitney test and t-test results on properties of wines
for i in dataall.columns.drop(['Type', 'quality']):
  print (i)
  stat, p = ttest_ind(white[i], red[i])
  print('T-test result for %s: t=%.3f, p=%.8f' % (i, stat, p))
  stat, p = stats.mannwhitneyu(white[i], red[i])
  print('Wilcoxon-Mann-Whitney result for %s: t=%.3f, p=%.8f\n' % (i, stat, p))<jupyter_output>fixed acidity
T-test result for fixed acidity: t=-44.906, p=0.00000000
Wilcoxon-Mann-Whitney result for fixed acidity: t=1693395.000, p=0.00000000

volatile acidity
T-test result for volatile acidity: t=-69.493, p=0.00000000
Wilcoxon-Mann-Whitney result for volatile acidity: t=772278.500, p=0.00000000

citric acid
T-test result for citric acid: t=15.375, p=0.00000000
Wilcoxon-Mann-Whitney result for citric acid: t=3070088.500, p=0.00000000

residual sugar
T-test result for residual sugar: t=29.996, p=0.00000000
Wilcoxon-Mann-Whitney result for residual sugar: t=2569687.000, p=0.00000000

chlorides
T-test result for chlorides: t=-48.123, p=0.00000000
Wilcoxon-Mann-Whitney result for chlorides: t=424886.500, p=0.00000000

free sulfur dioxide
T-test result for free sulfur dioxide: t=43.106, p=0.00000000
Wilcoxon-Mann-Whitney result for free sulfur dioxide: t=1186396.500, p=0.00000000

total sulfur dioxide
T-test result for total sulfur dioxide: t=79.075, p=0.00000000
Wilcoxon-Mann-Whitney[...]<jupyter_text>Aside from the amount of alcohol content, all other properties are significantly different between red and white wine. Specifically for the amount of alcohol between red wine and white wine, we see that t test gives a p value of 0.0079, which means that amount of alcohol is also signifanctly different between red wine and white wine. Nonetheless, as we mentioned previously, the alcohol distribution is skewed and thus the Wilcoxon-Mann-Whitney is another measurement we want to look at, and the p-value by Wilcoxon-Mann-Whitney shows that the distribution for the amount of alcohol between red wine and white wine is not significantly different, which is probably more accurate due to the more appropriate test with its distribution. Predicting Wine Type<jupyter_code>#Get features and response columns
X=dataall.drop(['Type'], axis=1)
Y=dataall['Type']
#see feature importance by Chi square test
bestfeatures = SelectKBest(score_func=chi2, k=3)
fit = bestfeatures.fit(X,Y)
featureScores = pd.concat([pd.DataFrame(X.columns),pd.DataFrame(fit.scores_)],axis=1)
featureScores.columns = ['Properties','Score']  #naming the dataframe columns
print(featureScores.sort_values(by=['Score']))<jupyter_output>Properties         Score
7                density      0.008961
10               alcohol      0.957399
8                     pH      5.652320
11               quality     12.122074
2            citric acid     15.118752
4              chlorides     37.398554
9              sulphates     64.271565
1       volatile acidity    221.064880
0          fixed acidity    358.497091
3         residual sugar   3287.056863
5    free sulfur dioxide  14913.554060
6   total sulfur dioxide  87946.248732<jupyter_text>Use logistic regression to build model for distinguishing red wine and white wine<jupyter_code>#Use Statsmodel Logit for logistic regression
dataallc=red.append(white)
dataallc.rename(columns={'volatile acidity':'volatile_acidity', 'free sulfur dioxide':'fsd', 'total sulfur dioxide':'tsd', 'fixed acidity':'facidity', 'citric acid':'ca', 'residual sugar':'rs'}, inplace=True)
dataallc.Type[dataallc['Type']=="red"]= 0
dataallc.Type[dataallc['Type']=="white"]= 1
dataallc['Type'] = dataallc['Type'].astype(str).astype(int)

#Separate training and test sets
train, test = train_test_split(dataallc, test_size= .30, random_state=42)

#Use Statsmodel Logit for logistic regression
formula=('Type ~ alcohol + volatile_acidity + density+ fsd + tsd + facidity + ca + rs + pH')
model=logit(formula = formula, data=train).fit()
#model=logit(formula = formula, data=train).fit(method='bfgs')   ###This works worse
model.summary()
# Evaluate the Model using confusion matrix and f1 score
prediction=model.predict(exog=test)
cutoff=0.5
yprediction=np.where(prediction > cutoff, 1,0)
yactual=test["Type"]
conf_matrix=pd.crosstab(yactual, yprediction, rownames=['Actual'], colnames=['Predict'], margins=True)
print(conf_matrix)

f1_score(yactual, yprediction, average='weighted')<jupyter_output>Predict    0     1   All
Actual                  
0        495     7   502
1          8  1440  1448
All      503  1447  1950
{'fraction_non_alphanumeric': 0.09454148471615721, 'fraction_numerical': 0.05185589519650655, 'mean_word_length': 4.7435736677115985, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 13, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '230722', 'license': 'MIT', 'n_tokens_mistral': 3250, 'n_tokens_neox': 2780, 'n_words': 947, 'path': 'Wine_Project_draft1.ipynb', 'repo_name': 'chaomacau/DSND_Wine_Project', 'size': '9160'}