<jupyter_start><jupyter_text>EH Mini Project - Recommender Mark 1  Input File  : Input File about Beer Data  Output File : NA<jupyter_code># Importing Libraries 
import numpy as np
import pandas as pd
import warnings
warnings.filterwarnings('ignore')

from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise        import cosine_similarity
from nltk.stem.snowball              import SnowballStemmer
from itertools                       import islice
# Beer Dataset Path
FILE_PATH = 'C://Users//Deshm//Downloads//'
FILE_NAME = 'BeerDataScienceProject.csv'

beer_df  = pd.read_csv(FILE_PATH+FILE_NAME, encoding='latin-1')
# Checking File Shape
beer_df.shape
beer_df.info()
#beer_df.drop(['beer_beerId','beer_brewerId'], axis =1, inplace = True)
#beer_df.drop(['beer_brewerId'], axis =1, inplace = True)
# Converting Unix time to Readable timestamp
beer_df['Review_Timestamp'] = pd.to_datetime(beer_df['review_time'], unit='s')
beer_df.sort_values(by='beer_beerId').head()
beer_df.drop(['review_time'], axis =1, inplace = True)
beer_df.nunique()
filtered_df = beer_df.dropna()
filtered_df.shape<jupyter_output><empty_output><jupyter_text>Question 1: Rank Top 3 Breweries which produce Strongest Beers<jupyter_code># Filtering out strongest beers on the basis of ABV > 7
# Domain Reference : https://www.thebeerstore.ca/beer-101/beer-types/ 
# This is a broad grouping that can describe any beer over 7% ABV. Strong beers are typically dark in colour, some are almost black. Different styles can include old ales, double IPAs, and barleywines.

df_q1 = filtered_df[filtered_df['beer_ABV'] > 7]
df_q1.columns
df_grouped = df_q1.groupby('beer_brewerId', sort=False)['beer_brewerId'].count().to_frame().rename(columns={'beer_brewerId': 'Count'})
df_grouped.sort_values(by='Count', ascending=False).head(3)<jupyter_output><empty_output><jupyter_text>Answer :   As shown above BrewerId : 1199. 3818, 158 have the strongest beer  Question 2: Which year did beers enjoy highest rating<jupyter_code>filtered_df.columns
filtered_df['Year'] = pd.DatetimeIndex(filtered_df['Review_Timestamp']).year
filtered_df.head(1)
filtered_df.columns
q2_df = filtered_df.groupby(['beer_name','Year'], sort=False)['review_overall'].max()
q2_df_new = q2_df.reset_index()
idx=q2_df_new.groupby(by='beer_name')['review_overall'].idxmax()
q2_final = q2_df_new.loc[idx,]
q2_final<jupyter_output><empty_output><jupyter_text>Answer: The Above datafame gives you the maximum rating of beers yearwise  Question 3: Based on the user’s ratings which factors are important among taste, aroma, appearance, and palette?<jupyter_code>filtered_df.columns
df_q3 = filtered_df.filter(['review_appearance', 'review_palette', 'review_overall', 'review_taste', 'review_aroma'], axis=1)
df_q3.head()
from sklearn.model_selection import train_test_split

np.random.seed(0)
df_train, df_test = train_test_split(df_q3, train_size = 0.7, test_size = 0.3, random_state = 100)
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
# Apply scaler() to all the columns except the 'yes-no' and 'dummy' variables
num_vars = ['review_appearance', 'review_palette', 'review_overall', 'review_taste', 'review_aroma']
df_train[num_vars] = scaler.fit_transform(df_train[num_vars])
df_train.head()
y_train = df_train.pop('review_overall')
x_train = df_train
import statsmodels.api as sm  
lm = sm.OLS(y_train,x_train).fit()
print(lm.summary())
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize = (20,10))  
sns.heatmap(x_train.corr(),annot = True,cmap="YlGnBu")<jupyter_output><empty_output><jupyter_text>Answer: Review Palette and Review_Taste seems to be more important to user than appearance<jupyter_code>y_train_price = lm.predict(x_train)

fig = plt.figure()
sns.distplot((y_train - y_train_price), bins = 20)
fig.suptitle('Error Terms', fontsize = 20)                  # Plot heading 
plt.xlabel('Errors', fontsize = 18)                         # X-label
#sns.pairplot(df_q3)
#plt.show()<jupyter_output><empty_output><jupyter_text>Q4 - If you were to recommend 3 beers to your friends based on this data which ones will you recommend?<jupyter_code>filtered_df.columns
idx = filtered_df.groupby(by=['review_palette','review_taste'])['review_overall'].idxmax()

q4_df = filtered_df.loc[idx,].sort_values(by='review_overall', ascending=False)
top_3_beers = q4_df[q4_df['review_overall'] > 4 ].groupby(['beer_beerId']).agg({'review_palette':'sum','review_taste':'sum'}).sort_values(by=['review_palette','review_taste'''], ascending=False).head(3)
a = top_3_beers.reset_index().beer_beerId
q4_final = q4_df[q4_df['beer_beerId'].isin(a)][['beer_beerId','beer_name','beer_style']]
q4_final.drop_duplicates()<jupyter_output><empty_output><jupyter_text>Answer : I would recommend the above beers to my friends  NLP Based Recommendation Engine Code<jupyter_code>beer_df.columns
# Finding all beers with Null Description
beer_df[beer_df['review_text'].isnull() == True].head()
# Choosing Text Fields for NLP Processing
beer_descr = beer_df[~beer_df['review_text'].isnull()][['beer_name','beer_style','review_text']]
beer_descr.shape
beer_descr.head(10)
# Initializing Stemmer for English Language
stemmer = SnowballStemmer("english")
beer_descr['stemmed'] = beer_descr.review_ text.map(lambda x: ' '.join([stemmer.stem(y) for y in x.split(' ')]))
beer_descr.head()
# Using countVectorizer to count how many times each term shows up,
# tfidf transformer to generated weight matrix
cvec = CountVectorizer(
    stop_words = "english",
    min_df = 0.01,
    max_df = 0.60, # Optimization would be needed
    ngram_range = (1,2)
)
cvec
cvec.fit(beer_descr['stemmed'])
list(islice(cvec.vocabulary_.items(),20))
cvec_counts = cvec.transform(beer_descr['stemmed'])
print('sparse matrix shape', cvec_counts.shape)
print('nonzero count:', cvec_counts.nnz)
print('Sparsity : %.2f%%' % (100.0 * cvec_counts.nnz / (cvec_counts.shape[0] * cvec_counts.shape[1])))
# top 20 terms
occ = np.asarray(cvec_counts.sum(axis=0)).ravel().tolist()
count_df = pd.DataFrame({'term': cvec.get_feature_names(), 'occurances': occ})
count_df.sort_values(by='occurances', ascending=False).head(20)
# Using TfIdf Transformer to calculate weights
transformer = TfidfTransformer()
transformed_weights = transformer.fit_transform(cvec_counts)

# compute the similarity of a beer choice
idx = beer_descr.index.get_indexer_for((beer_descr[beer_descr['beer_style']=='Tripel'].index))[0]
sims = cosine_similarity(transformed_weights[idx,:],transformed_weights)
beer_descr['sims'] = sims.T
beer_descr.sort_values(by='sims', ascending=False)[['beer_name','beer_style','sims']].head(20)
# Domain Assumption : Based on the Inputs from : https://www.thebeerstore.ca/beers/?beer_style=strong 
#              Strong
#              This is a broad grouping that can describe any beer over 7% ABV. 
#              Strong beers are typically dark in colour, some are almost black. 
#               Different styles can include old ales, double IPAs, and barleywines.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12533110274641016, 'fraction_numerical': 0.014219991635299039, 'mean_word_length': 6.357948717948718, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '268499', 'license': 'Apache-2.0', 'n_tokens_mistral': 2649, 'n_tokens_neox': 2522, 'n_words': 540, 'path': 'EH Mini Project-Copy1.ipynb', 'repo_name': 'SanketSDeshmukh/BeedDataScienceProject', 'size': '7173'}