<jupyter_start><jupyter_text>The data contains the following fields:For binary variables, 1 means yes, 0 means no.Header | Description-------|------------chocolate | Does it contain chocolate?fruity | Is it fruit flavored?caramel | Is there caramel in the candy?peanutalmondy | Does it contain peanuts, peanut butter or almonds?nougat | Does it contain nougat?crispedricewafer | Does it contain crisped rice, wafers, or a cookie component?hard | Is it a hard candy?bar | Is it a candy bar?pluribus | Is it one of many candies in a bag or box?sugarpercent | The percentile of sugar it falls under within the data set.pricepercent | The unit price percentile compared to the rest of the set.winpercent | The overall win percentage according to 269,000 matchups.<jupyter_code># Getting a glimpse of the dataset as well as checking for missing values 
df.info()
# Overview about the descriptive stats
df.describe().T<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code># Clean the letter Õ
df['competitorname'] = df['competitorname'].str.replace('Õ', "")
# Div Winpercent by 100 to have it in the same scale as the other two percentage variables 
df['winpercent'] = round(df['winpercent']/100, 3)
table = df.head(10) 
table.set_index("competitorname").style.applymap(color_df)
# Combine the two variables
# Higher value -> the candy is sweet as well as cheap.
### df['sugarbyprice'] = round(df['sugarpercent'].div(df['pricepercent']), 3) 

# Higher value -> the candy is more liked as well as cheap.
### df['winbyprice'] = round(df['winpercent'].div(df['pricepercent']), 3) 

# Check the transformations 
### df.sample(5)<jupyter_output><empty_output><jupyter_text>EDA All the most fav candies contain choccolate, none of them are either fruity or hard. Perhaps, penaut or almond helps in be liked by people. Also, interesting to see there are not one of many candies in the box.<jupyter_code># Most fav candies 
table = df.sort_values('winpercent', ascending=False).head(10)
table.set_index("competitorname").style.applymap(color_df)
# Least fav candies 
df.sort_values('winpercent', ascending=False).tail(10)<jupyter_output><empty_output><jupyter_text>If we sort by price the game changes a lot, here the most fav candies can be either chocco or fruity. Probably, plays a good part the fact that you get many of them in one box for cheap, compared to the other case.<jupyter_code>#df.sort_values('winbyprice', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Analyzing the candy components and some possible combinations<jupyter_code>print(f"N of choco and fruity {len(df[(df['chocolate']==1)&(df['fruity']==1)])}")
print(f"N of choco and peanutyalmondy {len(df[(df['chocolate']==1)&(df['peanutyalmondy']==1)])}")
print(f"N of choco and pluribus {len(df[(df['chocolate']==1)&(df['pluribus']==1)])}")
print(f"N of choco candies {len(df[df['chocolate']==1])}")
print(f"N of fruity {len(df[df['fruity']==1])}")
# This is a good combo but also quite expansive
df[(df['chocolate']==1)&(df['pluribus']==1)].sort_values('winpercent', ascending=False)[:5]
# Which is the one with both choco and fruit? 
df[(df['chocolate']==1)&(df['fruity']==1)]<jupyter_output><empty_output><jupyter_text>Check perc yes/no for categorical values<jupyter_code>for cat_val in df[df.columns[1:-5]]: 
    print(cat_val)
    print(df[cat_val].value_counts(normalize = True)*100)
    print()<jupyter_output>chocolate
0    56.470588
1    43.529412
Name: chocolate, dtype: float64

fruity
0    55.294118
1    44.705882
Name: fruity, dtype: float64

caramel
0    83.529412
1    16.470588
Name: caramel, dtype: float64

peanutyalmondy
0    83.529412
1    16.470588
Name: peanutyalmondy, dtype: float64

nougat
0    91.764706
1     8.235294
Name: nougat, dtype: float64

crispedricewafer
0    91.764706
1     8.235294
Name: crispedricewafer, dtype: float64

hard
0    82.352941
1    17.647059
Name: hard, dtype: float64<jupyter_text>Pearson Corr<jupyter_code>corr = df.corr()
mask = np.zeros_like(corr)
mask[np.triu_indices_from(mask)] = True
with sns.axes_style("white"):
    f, ax = plt.subplots(figsize=(24, 12))
    ax = sns.heatmap(df.corr(), mask=mask, vmax=.3, square=True, annot=True, cmap = 'coolwarm', linewidths=.5)<jupyter_output><empty_output><jupyter_text>There's multicollinearity we'll adress with PCA. Chocolate is the most corr with winpercent, also with price perc though. We need to find a good combo to keep the price low and use chocolate.   Spearman Corr<jupyter_code>print("winpercent vs sugar percent {}".format(stats.spearmanr(df.winpercent, df.sugarpercent)))
print("winpercent vs price percent {}".format(stats.spearmanr(df.winpercent, df.pricepercent)))<jupyter_output>winpercent vs sugar percent SpearmanrResult(correlation=0.21917627188619349, pvalue=0.04386672037824133)
winpercent vs price percent SpearmanrResult(correlation=0.3546312501902001, pvalue=0.0008682735199554266)<jupyter_text>Apriori Algo<jupyter_code># filter candies greater than 
bests = df[df['winpercent']>df['winpercent'].quantile(.5)]
df_bests =  bests[df.columns[1:-3]]

# use apriori alg with min support .4
result = apriori(df_bests, min_support=0.4,use_colnames=True).sort_values(by='support', ascending=False)

# plot as a bar chart
result.plot(kind='barh',x='itemsets',y='support',title=f'Most Frequent Combination by Top Candies',
                 sort_columns=True,figsize = (12,6),legend=False)
plt.subplot2grid
plt.figure(figsize=(6, 4))
sns.boxplot(x="chocolate", y="winpercent", data=bests)
plt.figure(figsize=(6, 4))
sns.boxplot(x="fruity", y="winpercent", data=bests)
df.columns
#set a empty canva where to put the bargrahp 
fig, ax = plt.subplots()

#create the bargraphs by player's race
sns.barplot(data = df, 
            x = "chocolate", 
            y = "winpercent", 
            hue = "peanutyalmondy",  palette="Set2")

#Set title and labels for the axis 
plt.title("Chocolate with or without peanutyalmondy by Winperc")
plt.xlabel(" ")
plt.ylabel("winperc")

#set the graph's size
fig.set_size_inches(8, 6)
#Scatter plot with a regression line 
sns.lmplot(x = "sugarpercent", y = "winpercent", data = df, 
           hue= "chocolate", col = "chocolate", col_wrap=3, 
           palette="Set2", ci = None)
#Scatter plot with a regression line 
sns.lmplot(x = "pricepercent", y = "winpercent", data = df, 
           hue= "chocolate", col = "chocolate", col_wrap=3, 
           palette="Set2", ci = None)
#Scatter plot with a regression line 
sns.lmplot(x = "sugarpercent", y = "pricepercent", data = df, 
           hue= "chocolate", col = "chocolate", col_wrap=3, 
           palette="Set2", ci = None)<jupyter_output><empty_output><jupyter_text>Of course if the price increase the winperent is lower, compared if the sugar increase. Though, with chocolate, the slope it still positive. Also, more sugar means higher price. Perhaps, a choco candy with less sugar might be a good starting point. Howver, there are lots of the in the market already.<jupyter_code>no_choco_no_alm = df[(df["chocolate"] == 0) & (df["peanutyalmondy"] == 0)]["winpercent"]

print(f"No choco no alm candy average winperc: {np.mean(no_choco_no_alm)}")

with_choco_alm = df[(df["chocolate"] == 1) & (df["peanutyalmondy"] == 1)]["winpercent"]

print(f"Choco and alm candy average winperc: {np.mean(with_choco_alm)}")

with_choco_no_alm = df[(df["chocolate"] == 1) & (df["peanutyalmondy"] == 0)]["winpercent"]

print(f"With Choco and no alm candy average winperc: {np.mean(with_choco_no_alm)}")
# No much offering and this combo (choco and almond) has a high winpercent 
len(no_choco_no_alm), len(with_choco_alm), len(with_choco_no_alm)
#Indipendent two sample t_test: no_choco_no_alm vs with_choco_alm 
t_val, p_val = stats.ttest_ind(no_choco_no_alm, with_choco_alm)
print(f"t-value: {t_val}, p-value: {p_val}")
 #Indipendent two sample t_test: no_choco_no_alm vs with_choco_no_alm 
t_val, p_val = stats.ttest_ind(no_choco_no_alm, with_choco_no_alm)
print(f"t-value: {t_val}, p-value: {p_val}")
 #Indipendent two sample t_test: no_choco_no_alm vs with_choco_no_alm 
t_val, p_val = stats.ttest_ind(with_choco_alm, with_choco_no_alm)
print(f"t-value: {t_val}, p-value: {p_val}")<jupyter_output>t-value: 2.70019542967287, p-value: 0.010603859044829765<jupyter_text>We have always got a t-value: +-2.44 < +-1.96 and a p-value: 0.015 < 0.05, hence we always reject the null hypothesis that the difference between the two population mean is zero. However, the p-value in the last test is bigger and result might change with a larger sample size. Anyway, with_choco_alm might be a really valuable choice.   OLS<jupyter_code>df.columns
# all unique let's drop this col 
len(df.competitorname.unique())
#Store all the predictors in X
X = df.drop(["winpercent", "competitorname"], axis = 1)

#Store the target as y 
y = df["winpercent"].copy()
#Fit the OLS using all the possible predictors in the dataset 
model = sm.OLS.from_formula("y ~ X", data = df).fit()

#Chech the model's summary
model.summary()
X_parsimonious = X.drop(["nougat", "caramel", "pluribus", "bar", "hard", "sugarpercent", "pricepercent"], axis=1)
#Fit the OLS using all the possible predictors in the dataset 
model = sm.OLS.from_formula("y ~ X_parsimonious", data = df).fit()

#Chech the model's summary
model.summary()
def my_ols(choco: int, fruit: int, alm: int, crisp: int): 
    win_perc =  0.3541 + (0.2025*choco) + (0.0818*fruit) + (0.1028*alm) + (0.0905*crisp)
    return f"The predicted salary has a value of: {win_perc * 100} %"
my_ols(choco=1, fruit=0, alm=1, crisp=1) 
my_ols(choco=1, fruit=0, alm=1, crisp=0)
my_ols(choco=1, fruit=1, alm=1, crisp=1)<jupyter_output><empty_output><jupyter_text>PCAThere is a bit of multicollinearity. We could use PCA for dimensionality reduction<jupyter_code>pca = PCA(svd_solver='randomized', random_state=42)
pca.fit(X, y=None)
print("Explained variance by first two components: {}%".format(pca.explained_variance_ratio_[0:2].sum()*100))
print("Explained variance by first three components: {}%".format(pca.explained_variance_ratio_[0:3].sum()*100))
print("Explained variance by first four components: {}%".format(pca.explained_variance_ratio_[0:4].sum()*100))
print("Explained variance by first five components: {}%".format(pca.explained_variance_ratio_[0:5].sum()*100))<jupyter_output>Explained variance by first two components: 56.98597003452521%
Explained variance by first three components: 65.26829069757302%
Explained variance by first four components: 72.48596230819707%
Explained variance by first five components: 79.2260831259417%<jupyter_text>First two should be okay.<jupyter_code>pca_df = pd.DataFrame({'PC1':pca.components_[0],'PC2':pca.components_[1], 'Feature':X.columns})
pca_df<jupyter_output><empty_output><jupyter_text>PCA ANLYSIS PC1 | PC2 | Feature-------|------------ | -----------0.504117 | 0.268853 | chocolate-0.492122 | -0.346325 | fruity0.196329 | -0.124990 | caramel0.207407 | 0.153608 | peanutyalmondy0.145375 | -0.124163 | nougat0.132925 | -0.033906 | crispedricewafer-0.188254 | -0.363448 | hard0.419428 | -0.251120 | bar-0.360884 | 0.742920 | pluribus0.038526 | 0.017161 | sugarpercent0.200588| 0.067914 | pricepercen* PCA 1: chocolate, caramel, peanutyalmondy, nougat, crispedricewafer, hard, bar, pricepercen* PCA 2: fruity, pluribus, sugarpercentPCA1 are typical choco candies in all their forms, PCA2 are the ones you buy in a box/pack, they're probabily small fruity candy with a lot of sugar. First ones are  more expansive, second one are cheaper.<jupyter_code>pca_df.Feature
fig = plt.figure(figsize = (12,8))

plt.scatter(pca_df.PC1, pca_df.PC2)

plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')

for i, txt in enumerate(pca_df.Feature):
    plt.annotate(txt, (pca_df.PC1[i],pca_df.PC2[i]))
plt.tight_layout()
# Tranform the dataset, keep only first two components 
df_pca = pca.transform(X) 
df_pca = df_pca[:, 0:2]

scaler = RobustScaler()
df_scaled = scaler.fit_transform(df_pca)
hopkins_test(pd.DataFrame(df_scaled))<jupyter_output><empty_output><jupyter_text>Clusters are statistically significant.<jupyter_code>distortions = []
K = range(1,10)
for k in K:
    kmeanModel = KMeans(n_clusters=k)
    kmeanModel.fit(df_scaled)
    distortions.append(kmeanModel.inertia_)
    
plt.figure(figsize=(12,6))
plt.plot(K, distortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Distortion')
plt.title('The Elbow Method showing the optimal k')
plt.show()
kmeanModel = KMeans(n_clusters=4, random_state=42)
kmeanModel.fit(df_scaled)
len(kmeanModel.labels_) == 85
kmeanModel.labels_
# creation of data frame with original features for analysis of clusters formed.

df_dummy = pd.DataFrame.copy(df)
dfkm4c = pd.concat([df_dummy, pd.Series(kmeanModel.labels_)], axis=1) # df-dataframe, km-kmeans, 5c-5clusters.
dfkm4c
dfkm4c.rename(columns={0:'Cluster ID'}, inplace=True)
dfkm4c.head()
# creation of data frame with features as principal components for analysis of clusters formed.

df_dummy = pd.DataFrame.copy(pd.DataFrame(df_scaled))
dfpcakm4c = pd.concat([df_dummy, pd.Series(kmeanModel.labels_)], axis=1)
dfpcakm4c.columns = ['PC1', 'PC2', 'Cluster ID']
dfpcakm4c.head()
sns.pairplot(data = dfpcakm4c, vars=['PC1', 'PC2'], hue='Cluster ID')
dfkm4c.groupby('Cluster ID').sum()<jupyter_output><empty_output><jupyter_text>* Cluster 1: Box/Pack of fruity candies* Cluster 2: Fruity candies with caramel * Cluster 3: Choco bars with different tasstes* Cluster 4: Box/Pack of choco candies with different tastes<jupyter_code>table = dfkm4c.groupby("Cluster ID").agg({"winpercent": "mean", "pricepercent": "mean", "sugarpercent": "mean"}
                             ).rename(columns={"winpercent" : "average winpercent", 
                                               "pricepercent" : "average pricepercent", 
                                               "sugarpercent" : "average sugarpercent", }
                             ).sort_values("average winpercent", ascending = False)
table
table.plot(kind = "bar", rot = 0.90, legend = True, figsize = [10, 6])
plt.title("Average winpercent, pricepercent, sugarpercent")
plt.xlabel("Cluster ID")
plt.ylabel("Percentage")<jupyter_output><empty_output><jupyter_text>Modelling<jupyter_code>X_train = df.drop(['competitorname', 'winpercent', 'sugarpercent', 'pricepercent'], axis=1)
target = df['winpercent']
import time
#Instantiate the model setting the weights
rfr = RandomForestRegressor(random_state=42, n_jobs=-1, bootstrap=True)

#Hyperparameters grid to be tuned
param_grid = {
        'n_estimators': [10, 30],
        'max_depth': [2, 5, None], 
}


# we'll use 5-fold cross-validation
rf_grid = GridSearchCV(rfr, param_grid, cv=5, 
                              scoring='neg_mean_squared_error',return_train_score=True, n_jobs=-1)

#Fit the model 
start = time.time()
rf_grid.fit(X_train, target)
end = time.time() - start
print(f"Took {end} seconds")
rf_grid.best_params_
np.sqrt(-rf_grid.best_score_)
y_pred = rf_grid.predict(X_train)

fig, ax = plt.subplots()
plt.scatter(x=target, y=y_pred, alpha=1)

#Set title and labels for the axis 
plt.title("") 

fig.set_size_inches(12, 8)
rfr = RandomForestRegressor(n_estimators=10, max_depth=5, random_state=42, n_jobs=-1, bootstrap=True)
rfr.fit(X_train, target)
feat_importances = pd.Series(rfr.feature_importances_, index=X_train.columns)
feat_importances.nlargest(10).plot(kind='barh')
for k, v in sorted(zip(feat_importances, X_train.columns), reverse=True):
    print(f"{v}: {k}")<jupyter_output>chocolate: 0.5495422956120739
peanutyalmondy: 0.12173661367694703
bar: 0.081913763648309
caramel: 0.0696368233293193
pluribus: 0.047780793204208756
crispedricewafer: 0.04176577053352297
fruity: 0.03938475471161708
nougat: 0.027019562261178475
hard: 0.021219623022823528
{'fraction_non_alphanumeric': 0.13175782987103743, 'fraction_numerical': 0.054952036084111554, 'mean_word_length': 5.383617193836172, 'pattern_counts': {'":': 3, '<': 66, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '564255', 'license': 'MIT', 'n_tokens_mistral': 6276, 'n_tokens_neox': 5625, 'n_words': 1428, 'path': 'analysis/Candy_Analysis.ipynb', 'repo_name': 'nicholas-dinicola/Candy-Power-Ranking-Analysis', 'size': '15741'}