<jupyter_start><jupyter_text>Wine reviews analysis Table of contents * [1. Overview of dataset](dataset-overview)    * [1.1 Variables overview](variables-overview)* [2. Goal of analysis](goal-of-analysis)* [3. Data insights and cleaning](data-insights-and-cleaning)    * [3.1 First insights](first-insights)    * [3.2 Imputation and cleaning](imputation-and-cleaning)* [4. Additional features](additional-features)    * [4.1 Reviewer's count of followers on Twitter](twitter-followers)    * [4.2 Analysis of keywords in descriptions](description-keywords)* [5. Dummies for categorical variables](dummies-for-categorical)* [6. Predictions](predictions)    * [6.1 Points](points)    * [6.2 Price](price)* [7. Conclusions and further actions](conclusions)  1. Overview of dataset [Back to Table of Contents](toc)Dataset used is taken from [Kaggle](https://www.kaggle.com/zynicide/wine-reviews): *winemag-data-130k-v2.csv*. As pointed by author, it is scraped from [Wine Enthusiast](https://www.winemag.com/) website and contains wine decriptions by tasters, prices and scores granted by experts.Description of reviewing and scoring methodology (provided on website of every wine, for example [here](https://www.winemag.com/buying-guide/nicosia-2013-vulka-bianco-white-etna/)):**Tasting Methodology and Goals**All tastings reported in the Buying Guide are performed blind. Typically, products are tasted in peer-group flights of from 5–8 samples. Reviewers may know general information about a flight to provide context—vintage, variety or appellation—but never the producer or retail price of any given selection. When possible, products considered flawed or uncustomary are retasted.**About the Scores**Ratings reflect what our editors felt about a particular product. Beyond the rating, we encourage you to read the accompanying tasting note to learn about a product’s special characteristics.|Score|Value|Description||-----|-----------|-----------||98–100|Classic|The pinnacle of quality.||94–97|Superb|A great achievement.||90–93|Excellent|Highly recommended.||87–89|Very Good|Often good value; well recommended.||83–86|Good|Suitable for everyday consumption; often good value.||80–82|Acceptable|Can be employed in casual, less-critical circumstances.|Products deemed Unacceptable (receiving a rating below 80 points) are not reviewed. 1.1 Variables overview [Back to Table of Contents](toc)    *From [Kaggle](https://www.kaggle.com/zynicide/wine-reviews)*|Variable|Description||-----|-----------||country|The country that the wine is from||description|||designation|The vineyard within the winery where the grapes that made the wine are from||points|The number of points WineEnthusiast rated the wine on a scale of 1-100||price|The cost for a bottle of the wine||province|The province or state that the wine is from||region_1|The wine growing area in a province or state||region_2|Sometimes there are more specific regions specified within a wine growing area||taster_name|||taster_twitter_handle|||title|The title of the wine review, which often contains the vintage if you're interested in extracting that feature|variety|The type of grapes used to make the wine||winery|The winery that made the wine|  2. Goal of analysis[Back to Table of Contents](toc)Using data provided (including description), try to predict wine price and wine score.  3. Data insights and cleaning  3.1 First insights[Back to Table of Contents](toc)<jupyter_code># Import packages
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb
from IPython.display import Markdown as md
from bs4 import BeautifulSoup
import unicodedata
import json
%matplotlib inline
# Import data
wine_data = pd.read_csv('winemag-data-130k-v2.csv', index_col = 0)
wine_data.head(3)
# Get insights of data shape
# wine_data.describe(include='all')
# Function for summary on DF
def dataSummary(data):
    eda_dict = {
            'record_count': data.count() + data.isnull().sum()
            , 'column_type': data.dtypes
            , 'unique_count': data.nunique()
            , 'nonmissing_count': data.count()
            , 'missing_count': data.isnull().sum()
            , 'missing_perc': round(data.isnull().sum() / (data.count() + data.isnull().sum()) * 100, 2)
            }
    eda_data = pd.concat(eda_dict, axis = 1)
    return eda_data
# Get insights of data
eda_wine_data = dataSummary(wine_data)
eda_wine_data<jupyter_output><empty_output><jupyter_text>Data above points the following:* There are 129971 records in dataset* Only 120975 records contain price - one of our targets* Every record has description, points, title and winery filled; other variables have missingsWill take a look on all variables one by one in order to clean, impute missings and possibly get new features  3.2 Imputation and cleaning[Back to Table of Contents](toc)<jupyter_code># Function for summary on variable
def varCountDistStats(data, var_name, n_return = 10):
    eda_dict = {
            'record_count': data.groupby([var_name])[var_name].count()
            , 'record_perc': round(data.groupby([var_name])[var_name].count() / (data[var_name].count()) * 100, 1)
            }
    eda_data = pd.concat(eda_dict, axis = 1, sort = False)
    return eda_data.sort_values('record_count', ascending = False).head(n_return)<jupyter_output><empty_output><jupyter_text>3.2.1 Price As price is one of our target variables, it is important to have it for all records. Above we've seen, that almost 7% of data has no price filled. Lets discover nature of missing records and decide wheteher they could be dropped for further calculations. Let's take a look on points histogram depending on price availability.<jupyter_code># Make missing price flag
wine_data['has_missing_price'] = np.where(wine_data['price'].isna(), 'No price', 'Has price')
plt.figure(figsize=(10,5))
bins = np.linspace(80, 100, 21)
plt.hist(wine_data[wine_data['has_missing_price'] == 'Has price']['points'], bins, label='Has price', density = True, alpha = 0.5)
plt.hist(wine_data[wine_data['has_missing_price'] == 'No price']['points'], bins, label='No price', density = True, alpha = 0.5)
plt.legend(loc = 'upper right')
plt.title('Distribution of records by points')
plt.show()<jupyter_output><empty_output><jupyter_text>Distributions of records with and without price are very similar. Letš find if there are differencies by country.<jupyter_code>wine_by_country = pd.concat({
                            'record_count': wine_data.groupby(['country'])['country'].count()
                            , 'with_price_percent': wine_data[wine_data['has_missing_price'] == 'Has price'].groupby(['country'])['country'].count() / wine_data.groupby(['country'])['country'].count()
                            , 'no_price_percent': wine_data[wine_data['has_missing_price'] == 'No price'].groupby(['country'])['country'].count() / wine_data.groupby(['country'])['country'].count()
                            }
                        , axis = 1, sort = False).sort_values('record_count', ascending = False)
wine_by_country.fillna(0, inplace = True)
wine_by_country.head()
plt.figure(figsize=(20,5))
plt.bar(wine_by_country.index, wine_by_country['with_price_percent'], label = 'Has price', alpha = 0.5)
plt.bar(wine_by_country.index, wine_by_country['no_price_percent'], label = 'No price', alpha = 0.5
        , bottom = wine_by_country['with_price_percent'])

for i, v in enumerate(wine_by_country['record_count']):
    plt.text(i, 0.76, str(v), rotation = 30, ha = 'center')
for i, v in enumerate(wine_by_country['no_price_percent']):
    plt.text(i, 0.975, str(round(v*100,1))+'%', rotation = 30, ha = 'center')
plt.text(-1.5, 0.76, 'records:', rotation = 30, ha = 'center')
plt.text(-1.38, 0.965, 'no price %:', rotation = 30, ha = 'center')

plt.ylim((0.75, 1))
plt.xticks(rotation = 45)
plt.legend(loc = 'center right')
plt.title('Distribution of records by country (with record count))')
plt.show()<jupyter_output><empty_output><jupyter_text>Data shows, that prices are missing not at random, if divided by country. There are much frequent missings for large represented France, Italy, Portugal and Austria (13-19%) and quite a few for US, Spain, Chile, Argentina, Australia and Germany (2% or less). Reasons for this inconsistency are unknown. It is obvious, that dataset does not contain all ever existed wines all over the world, therefore dropping of records with no price should not cause problems as data does not represent whole population. Let's drop records with no price and recalculate data shape.<jupyter_code>wine_data_with_price = wine_data[wine_data['has_missing_price'] == 'Has price']
# Get insights of data
eda_wine_data_with_price = dataSummary(wine_data_with_price)
eda_wine_data_with_price<jupyter_output><empty_output><jupyter_text>Let's also check if there is dependancy between points and price.As there are only few points with price > 1000 USD, let's see graphs limiting axes to 1000 USD for better understanding.<jupyter_code>print('Wines with price > 1000 USD:', wine_data_with_price[wine_data_with_price['price'] > 1000]['price'].count())
g = sb.pairplot(wine_data_with_price.loc[:,['price', 'points']])
g.axes[0,0].set_xlim((0,1000))
g.axes[0,0].set_ylim((0,1000));<jupyter_output>Wines with price > 1000 USD: 14<jupyter_text>For sure there is corelation between price and points. There are almost no wines with price >500 USD having score <90 points. Nevertheless, Winemag website states, that wines are evaluated with no background on winery, price etc.  3.2.2 Country<jupyter_code># Take a look on most popular countries
varCountDistStats(wine_data_with_price, 'country')<jupyter_output><empty_output><jupyter_text>Wines from US, France and Italy cover 74% of all wine records. Let's take a look on records with missings to understand how to impute.<jupyter_code>print(eda_wine_data_with_price.loc[['country'],['missing_count', 'missing_perc']])
wine_data_with_price[wine_data_with_price['country'].isnull()].describe(include = 'all')<jupyter_output>missing_count  missing_perc
country             59          0.05<jupyter_text>For missing values provinces and regions are also missing. The only field which could help to impute country is winery.<jupyter_code># Get all winery names with missing country
wine_data_with_price[wine_data_with_price['country'].isnull()]['winery'].unique()
# Create dictionary
# winery: country
winery_country_dict = {
    'Gotsa Family Wines': 'Georgia'
    , 'Kakhetia Traditional Winemaking': 'Georgia'
    , 'Tsililis': 'Georgia'
    , 'Ross-idi': 'Bulgaria'
    , 'Orbelus': 'Bulgaria'
    , 'St. Donat': 'Hungary'
    , 'Familia Deicas': 'Uruguay'
    , 'Stone Castle': 'Albania'
    , 'Teliani Valley': 'Georgia'
    , 'Undurraga': 'Chile'
    , 'Mt. Beautiful': 'New Zealand'
    , 'Neumeister': 'Austria'
    , 'Bachelder': 'Canada'
    , 'Chilcas': 'Chile'
    , 'Santa Ema': 'Chile'
    , 'Psagot': 'Israel'
    , 'Ktima Voyatzi': 'Greece'
    , 'Lismore': 'South Africa'
    , 'Logodaj': 'Bulgaria'
    , 'Somlói Vándor': 'Hungary'
    , 'Les Frères Dutruy': 'Switzerland'
    , 'El Capricho': 'Uruguay'
    , 'Amiran Vepkhvadze': 'Georgia'
    , 'Büyülübağ': 'Turkey'
}
# For further possible usage, create is_imputed_country flag
wine_data_with_price['is_imputed_country'] = wine_data_with_price['country'].isnull()
# Impute country
wine_data_with_price['country'] = wine_data_with_price['country'].fillna(wine_data_with_price['winery'].map(winery_country_dict));<jupyter_output>C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  after removing the cwd from sys.path.<jupyter_text>3.2.2 Province<jupyter_code># Take a look on most popular provincies
varCountDistStats(wine_data_with_price, 'province')<jupyter_output><empty_output><jupyter_text>Above (in 'country' section) we saw, that provincies are missing for the same records, where countries did. It is much more time consuming to find manually correct provinces for wineries. Let's take a look, whether there are records in initial dataset, where 'country' = 'province'. Perhaps it could be used for simple imputation.<jupyter_code>specific_countries = set(winery_country_dict.values())
wine_data_spec_c = wine_data_with_price[wine_data_with_price['country'].isin(specific_countries)]
country_spec_eda_dict = {
        'record_count': wine_data_spec_c.groupby(['country'])['country'].count()
        , 'same_province_record_count': wine_data_spec_c[wine_data_spec_c['province'] == wine_data_spec_c['country']].groupby(['country'])['country'].count()
        , 'same_province_record_perc': round(wine_data_spec_c[wine_data_spec_c['province'] == wine_data_spec_c['country']].groupby(['country'])['country'].count() / wine_data_spec_c.groupby(['country'])['country'].count() * 100, 1)
        }
country_spec_eda_data = pd.concat(country_spec_eda_dict, axis = 1, sort = False)
country_spec_eda_data.sort_values('record_count', ascending = False).head(10)<jupyter_output><empty_output><jupyter_text>Data shows, that initial data contains such records. So let`s impute province with country.<jupyter_code># For further possible usage, create is_imputed_province flag
wine_data_with_price['is_imputed_province'] = wine_data_with_price['province'].isnull()
# Impute province
wine_data_with_price['province'] = wine_data_with_price['province'].fillna(wine_data_with_price['winery'].map(winery_country_dict))<jupyter_output>C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  after removing the cwd from sys.path.<jupyter_text>3.2.4 Designation, region_1, region_2 No specific ideas how to impute these columns with relevant data. Will impute 'Unknown' value.<jupyter_code># For further possible usage, create is_imputed flags
wine_data_with_price['is_imputed_designation'] = wine_data_with_price['designation'].isnull()
wine_data_with_price['is_imputed_region_1'] = wine_data_with_price['region_1'].isnull()
wine_data_with_price['is_imputed_region_2'] = wine_data_with_price['region_2'].isnull()
# Impute
wine_data_with_price['designation'] = wine_data_with_price['designation'].fillna('Unknown designation')
wine_data_with_price['region_1'] = wine_data_with_price['region_1'].fillna('Unknown region_1')
wine_data_with_price['region_2'] = wine_data_with_price['region_2'].fillna('Unknown region_2');<jupyter_output>C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  This is separate from the ipykernel package so we can avoid doing imports until
C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = val[...]<jupyter_text>3.2.5 Variety<jupyter_code># Take a look on most popular varieties
varCountDistStats(wine_data_with_price, 'variety')
print(eda_wine_data_with_price.loc[['variety'],['missing_count', 'missing_perc']])
wine_data_with_price[wine_data_with_price['variety'].isnull()]<jupyter_output>missing_count  missing_perc
variety              1           0.0<jupyter_text>There is only 1 missing record, let's impute it manually. [Winemag page](https://www.winemag.com/buying-guide/carmen-1999-maipo-valley/) states it's red wine, but has no variety. Let's impute the most popular variety: Pinot Noir.<jupyter_code># For further possible usage, create is_imputed_variety flag
wine_data_with_price['is_imputed_variety'] = wine_data_with_price['variety'].isnull()
# Impute variety
wine_data_with_price['variety'] = wine_data_with_price['variety'].fillna('Pinot Noir');<jupyter_output>C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  after removing the cwd from sys.path.<jupyter_text>3.2.6 Twitter account From missing records stats before we've seen that there are records with taster names with no twitter account reference. Let's add this missing info.<jupyter_code># Info from https://www.geeksforgeeks.org/performing-google-search-using-python-code/
# Install google package
# !pip install google
from googlesearch import search
import requests
# Find taster names with no twitter accounts
taster_names = list(set(wine_data_with_price[wine_data_with_price['taster_twitter_handle'].isnull() 
                         & (wine_data_with_price['taster_name'].notnull())]['taster_name']))
taster_names
# for i in taster_names[0]:
#     query = i + ' site:winemag.com/contributor'
#     for j in search(query, tld='com', lang='en', start=0, stop=1, pause=2.0): 
#         print(j)
#         req = requests.get(j)
#         if req.status_code != 200:
#             print(f'Unexpected status code {req.status_code}. Stopping parse')
#         soup = BeautifulSoup(req.text, 'lxml')<jupyter_output><empty_output><jupyter_text>No luck, website has block for python requests. Will need to impute manually (luckily, there are only 3 values to impute).<jupyter_code>taster_twitter_accounts_dict = {
    '<NAME>': '@carriedykes1'
    , '<NAME>': '@apatrone23'
}
# For taster '<NAME>' there is no Twiter at all<jupyter_output><empty_output><jupyter_text>The only missing values remaining are some reviewer names and their twitters. As they are not specified on website will fill them with 'Unknown' value.<jupyter_code># For further possible usage, create is_imputed flags
wine_data_with_price['is_imputed_twitter'] = wine_data_with_price['taster_twitter_handle'].isnull()
wine_data_with_price['is_imputed_taster_name'] = wine_data_with_price['taster_name'].isnull()
# Impute
wine_data_with_price['taster_twitter_handle'] = wine_data_with_price['taster_twitter_handle'].fillna('Unknown twitter')
wine_data_with_price['taster_name'] = wine_data_with_price['taster_name'].fillna('Unknown taster_name');
# Get insights of data
eda_wine_data_with_price = dataSummary(wine_data_with_price)
eda_wine_data_with_price<jupyter_output><empty_output><jupyter_text>No more missings, could continue with additional feature engineering.  4. Additional features[Back to Table of Contents](toc)  4.1 Reviewer's count of followers on Twitter[Back to Table of Contents](toc) Idea how to get followers count from https://kaspars.net/blog/twitter-follower-count-without-api<jupyter_code># Find all Twitter accounts in dataset, delete Unknown, remove unicode space, remove @
twitter_accounts = list(set(wine_data_with_price['taster_twitter_handle']))
twitter_accounts = [unicodedata.normalize("NFKD", i)[1:] for i in twitter_accounts if '@' in i]
twitter_accounts
twitter_url_base = 'https://cdn.syndication.twimg.com/widgets/followbutton/info.json?screen_names='
# Check if all Twitter accounts exist
for i in twitter_accounts:
    req = requests.get(twitter_url_base + i)
    if req.status_code != 200:
        print(f'Unexpected status code {req.status_code}. Stopping parse')
    soup = BeautifulSoup(req.text, 'lxml')
    if soup.text == '[]':
        print('Twitter name does not exist, check manually: ', i)<jupyter_output>Twitter name does not exist, check manually:  winewchristina<jupyter_text>Found out non-existing Twitter account - now it's '@ckpickard'. To replace account name and to add follower count to dictionary.<jupyter_code>wine_data_with_price.loc[wine_data_with_price['taster_twitter_handle'] == '@winewchristina', 'taster_twitter_handle'] = '@ckpickard'
# Find all Twitter accounts once again
twitter_accounts = list(set(wine_data_with_price['taster_twitter_handle']))
twitter_accounts = [unicodedata.normalize("NFKD", i)[1:] for i in twitter_accounts if '@' in i]
twitter_accounts
twitter_dict = dict()
for i in twitter_accounts:
    req = requests.get(twitter_url_base + i)
    if req.status_code != 200:
        print(f'Unexpected status code {req.status_code}. Stopping parse')
    soup = BeautifulSoup(req.text, 'lxml')
    if soup.text == '[]':
        print('Twitter name does not exist, check manually: ', i)
        continue
    twitter_dict['@' + i] = json.loads(soup.find('body').string)[0]['followers_count']
twitter_dict
# Add Twitter followers count to data, fill all records with no Twitter accounts with 0
wine_data_with_price['twitter_follower_count'] = np.nan
wine_data_with_price['twitter_follower_count'] = wine_data_with_price['twitter_follower_count'].fillna(wine_data_with_price['taster_twitter_handle'].map(twitter_dict))
wine_data_with_price['twitter_follower_count'] = wine_data_with_price['twitter_follower_count'].fillna(0)<jupyter_output>C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  This is separate from the ipykernel package so we can avoid doing imports until
C:\Users\jekat\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = val[...]<jupyter_text>4.2 Analysis of keywords in descriptions[Back to Table of Contents](toc)<jupyter_code># pip install nltk==3.3
# wordcloud installed through prompt
# conda install -c conda-forge wordcloud
import nltk
import string  
# nltk.download('stopwords')
from nltk.corpus import stopwords
import re

# from nltk.corpus import webtext
# from nltk.probability import FreqDist
from wordcloud import WordCloud
from nltk.stem import PorterStemmer 
# Create column with cleaned divided description
symbols_to_clean = string.punctuation + '¬•–––—–”“´’‘…' + string.digits
wine_data_with_price['description_divided'] = wine_data_with_price['description'].apply(lambda i: i.lower().translate(str.maketrans('', '', symbols_to_clean)))
wine_data_with_price['description_divided'] = wine_data_with_price['description_divided'].apply(lambda i: i.split())
wine_data_with_price.head()
# Create list of descriptions
descriptions = wine_data_with_price['description_divided']
descriptions[:3]
# Make list of words from descriptions
all_words = []
for i in descriptions:
    all_words += i
all_words[:10]
# Remove stopwords (make same punctuation removal from them)
stopWords = [i.lower().translate(str.maketrans('', '', symbols_to_clean)) for i in stopwords.words('english')]
stopWords = set(stopWords)
print('Examples of stopwords:', list(stopWords)[:15])

all_words = [i for i in all_words if not i in stopWords]
print('Examples of left words', all_words[:15])<jupyter_output>Examples of stopwords: ['t', 'what', 'havent', 'doing', 'further', 'dont', 'from', 'and', 's', 'should', 'shant', 'o', 'theirs', 'were', 'he']
Examples of left words ['ripe', 'fruity', 'wine', 'smooth', 'still', 'structured', 'firm', 'tannins', 'filled', 'juicy', 'red', 'berry', 'fruits', 'freshened', 'acidity']<jupyter_text>To improve word matching let's perform stemming of words: convert, for example 'wine' and 'wines' into 'wine'.<jupyter_code># Perform word stemming (idea from https://www.geeksforgeeks.org/python-stemming-words-with-nltk/)
# Create dictionary to make opposite conversion
ps = PorterStemmer() 
all_words_stemmed_dict = dict(zip(set(all_words), [ps.stem(i) for i in list(set(all_words))]))
# all_words_stemmed_dict
# Create frequencies of stemmed words
fdist = nltk.FreqDist([ps.stem(i) for i in all_words])
keyword_freq = pd.DataFrame(fdist.items(), columns=['word_stemmed', 'frequency'])
keyword_freq.sort_values('frequency', ascending = False)
# Take a look on frequencies
keyword_freq['frequency'].describe()
# Take a look on frequency quantiles
keyword_freq['frequency'].quantile(np.array(range(90, 100)) / 100)<jupyter_output><empty_output><jupyter_text>There are 33223 unique stemmed words in descriptions, 90% of them appear up to 42 times in descriptions. Take closer look on words: make frequency wordclouds and histograms.<jupyter_code># Plotting the wordclouds
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,5))
fig.suptitle('Word clouds for words in wine descriptions')

wcloud = WordCloud().generate_from_frequencies(dict([(m, n) for m, n in fdist.items() if n > 5000]))
ax1.imshow(wcloud, interpolation="bilinear")
ax1.axis("off")
ax1.title.set_text('Word in > 5000 descriptions')

wcloud = WordCloud().generate_from_frequencies(dict([(m, n) for m, n in fdist.items() if n == 1]))
ax2.imshow(wcloud, interpolation="bilinear")
ax2.axis("off")
ax2.title.set_text('Word in only 1 description')
# Plotting the histograms
k = 5000

fig, (ax1, ax2) = plt.subplots(1, 2, figsize = (20,5))
fig.suptitle('Histograms of word frequencies')

ax1.hist(keyword_freq[keyword_freq['frequency'] < k]['frequency'], bins = min(k - 1, 100));
ax1.title.set_text('Words with frequencies < {} (total {} words)'.format(k, keyword_freq[keyword_freq['frequency'] < k]['frequency'].count()))

ax2.hist(keyword_freq[keyword_freq['frequency'] >= k]['frequency'], bins = keyword_freq[keyword_freq['frequency'] >= k]['frequency'].count() - 1);
ax2.title.set_text('Words with frequencies >= {} (total {} words)'.format(k, keyword_freq[keyword_freq['frequency'] >= k]['frequency'].count()))<jupyter_output><empty_output><jupyter_text>There are a lot of words appearing in few descriptions. Let's concentrate on more frequent words (expert choice of >=5000 appearances in descriptions) and create dummy variables for them: whether wine description contains the word or not.(Word count is reduced in order to decrease computational time - 111 words selected)<jupyter_code># Filter words of interest
words_set = set(keyword_freq[keyword_freq['frequency'] >= k]['word_stemmed'])
# Create stemmed word dictionary only of words of interest
words_set_dict = dict()
for key, value in all_words_stemmed_dict.items():
    if value in words_set:
        words_set_dict[key] = value
%%time
# Create dummy columns: 0/1
for i in list(words_set):
    wine_data_with_price['dummy_desc_' + i] = 0
    for key, value in words_set_dict.items():
        if value == i:
            wine_data_with_price['dummy_desc_' + i][wine_data_with_price['description_divided'].apply(lambda x: key in x)] = 1
# 31 sec for 111 words
wine_data_with_price.head()<jupyter_output><empty_output><jupyter_text>5. Dummies for categorical variables[Back to Table of Contents](toc) Will try to include only column with less categories - country.<jupyter_code># Choose suitable categorical columns and create for them dummy variables
categorical_columns = wine_data_with_price.select_dtypes(include = 'object').columns.tolist()
to_remove = ['description', 'designation', 'title', 'winery', 'has_missing_price', 'description_divided', 
             'region_1', 'region_2', 'province'
#              ,'variety', 'taster_name', 'taster_twitter_handle'
            ]
for i in to_remove:
    categorical_columns.remove(i)
categorical_columns
for i in categorical_columns:
    temp = pd.get_dummies(wine_data_with_price[i], prefix = 'dummy_' + i)
    temp.columns = temp.columns.str.replace(' ', '')
    wine_data_with_price = pd.merge(wine_data_with_price, temp, left_index = True, right_index = True)
wine_data_with_price.head()<jupyter_output><empty_output><jupyter_text>6. Predictions[Back to Table of Contents](toc)<jupyter_code>from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import SGDRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import StratifiedKFold

random_state = 20191229
# Helper function
def MAPE(y, yhat, digits = 5):
    ''' Calculate mean average percentage error
    digits: count of digits for rounding'''
    return round(np.mean(abs((y - yhat) / y)), digits)

# From https://stackoverflow.com/questions/7941226/how-to-add-line-based-on-slope-and-intercept-in-matplotlib
def abline(slope, intercept):
    """Plot a line from slope and intercept"""
    axes = plt.gca()
    x_vals = np.array(axes.get_xlim())
    y_vals = intercept + slope * x_vals
    plt.plot(x_vals, y_vals, '--')
# Create column name objects
X_columns = ['twitter_follower_count'] + [s for s in wine_data_with_price.columns if 'dummy_' in s]
Y1_columns = 'points'
Y2_columns = 'price'
# Scale columns of interest
scaler = MinMaxScaler()
wine_data_with_price[X_columns] = scaler.fit_transform(wine_data_with_price[X_columns])
# Test/train split: 0.8/0.2
train_df, test_df = train_test_split(wine_data_with_price, test_size = 0.2, random_state = random_state)<jupyter_output><empty_output><jupyter_text>6.1. Points[Back to Table of Contents](toc)<jupyter_code>X_train = train_df[X_columns].values
y_train = train_df[Y1_columns].values

X_test = test_df[X_columns].values
y_test = test_df[Y1_columns].values
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)<jupyter_output>(96780, 888)
(96780,)
(24195, 888)
(24195,)<jupyter_text>6.1.1. Linear regression with default settings<jupyter_code>learner = LinearRegression()
learner.fit(X_train, y_train)
y_test_pred = learner.predict(X_test)
y_train_pred = learner.predict(X_train)
# Check MAPE
MAPE_test = MAPE(y_test, y_test_pred)
MAPE_train = MAPE(y_train, y_train_pred)
print(MAPE_test, MAPE_train)
# Check results visually
print(y_test_pred, y_test)
# Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.show()<jupyter_output><empty_output><jupyter_text>Such a huge MAPE on test data is explained by few points having abnormally high predictions. This needs to be inspected more carefully in case model is chosen for usage.<jupyter_code># Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.xlim(75, 105)
plt.ylim(75, 105)
plt.show()<jupyter_output><empty_output><jupyter_text>6.1.2. Stochastic Gradient Descent Regressor with default settings<jupyter_code># Stochastic Gradient Descent: SGDRegressor
learner = SGDRegressor(random_state = random_state)
learner.fit(X_train, y_train)
y_test_pred = learner.predict(X_test)
y_train_pred = learner.predict(X_train)
# Check results visually
print(y_test_pred, y_test)
# Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.show()
# Check MAPE
MAPE_test = MAPE(y_test, y_test_pred)
MAPE_train = MAPE(y_train, y_train_pred)
print(MAPE_test, MAPE_train)<jupyter_output>0.01958 0.01957<jupyter_text>Valuable result achieved even using default parameters of model: 1.96% mean average percentage error  6.1.3. Stochastic Gradient Descent Regressor: Parameter grid search<jupyter_code># learner = SGDRegressor(random_state = random_state)
# parameter_grid = {'loss': ['squared_loss', 'huber', 'epsilon_insensitive'],
#                   'penalty': ['none', 'l2', 'l1', 'elasticnet']}
# cross_validation = StratifiedKFold(n_splits = 10)
# grid_search = GridSearchCV(learner,
#                            param_grid = parameter_grid,
#                            cv = cross_validation)
# grid_search.fit(X_train, y_train)
# print('Best score: {}'.format(grid_search.best_score_))
# print('Best parameters: {}'.format(grid_search.best_params_))

# grid_search.best_estimator_
# Stochastic Gradient Descent: SGDRegressor
learner = SGDRegressor(loss = 'squared_loss', penalty = 'elasticnet', random_state = random_state)
learner.fit(X_train, y_train)
y_test_pred = learner.predict(X_test)
y_train_pred = learner.predict(X_train)
# Check results visually
print(y_test_pred, y_test)
# Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.show()
# Check MAPE
MAPE_test = MAPE(y_test, y_test_pred)
MAPE_train = MAPE(y_train, y_train_pred)
print(MAPE_test, MAPE_train)<jupyter_output>0.01959 0.01957<jupyter_text>Model chosen by optimisation of grid search shows the same MAPE results as model with default parameters: 1.96% MAPE.  6.2. Price[Back to Table of Contents](toc)<jupyter_code>X_train = train_df[X_columns].values
y_train = train_df[Y2_columns].values

X_test = test_df[X_columns].values
y_test = test_df[Y2_columns].values
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)<jupyter_output>(96780, 888)
(96780,)
(24195, 888)
(24195,)<jupyter_text>6.2.1. Linear regression with default settings<jupyter_code>learner = LinearRegression()
learner.fit(X_train, y_train)
y_test_pred = learner.predict(X_test)
y_train_pred = learner.predict(X_train)
# Check MAPE
MAPE_test = MAPE(y_test, y_test_pred)
MAPE_train = MAPE(y_train, y_train_pred)
print(MAPE_test, MAPE_train)
# Check results visually
print(y_test_pred, y_test)
# Check results visually
plt.scatter(y_test_pred, y_test, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.show()<jupyter_output><empty_output><jupyter_text>Again, such a huge MAPE on test data is explained by few points having abnormally high predictions. This needs to be inspected more carefully in case model is chosen for usage.<jupyter_code># Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.xlim(-10, 200)
plt.ylim(-10, 200)
plt.show()<jupyter_output><empty_output><jupyter_text>6.2.2. Stochastic Gradient Descent Regressor with default settings<jupyter_code># Stochastic Gradient Descent: SGDRegressor
learner = SGDRegressor(random_state = random_state)
learner.fit(X_train, y_train)
y_test_pred = learner.predict(X_test)
y_train_pred = learner.predict(X_train)
# Check results visually
print(y_test_pred, y_test)
# Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.ylim(-10, max(y_test_pred) + 5)
plt.show()
# Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.xlim(-10, 1000)
plt.ylim(-10, max(y_test_pred) + 5)
plt.show()
# Check MAPE
MAPE_test = MAPE(y_test, y_test_pred)
MAPE_train = MAPE(y_train, y_train_pred)
print(MAPE_test, MAPE_train)<jupyter_output>0.56119 0.56029<jupyter_text>In case of price model is very inprecise: MAPE 56%. Let's see if parameter grid search can improve this result.  6.2.3. Stochastic Gradient Descent Regressor: Parameter grid search<jupyter_code># learner = SGDRegressor(random_state = random_state)
# parameter_grid = {'loss': ['squared_loss', 'huber', 'epsilon_insensitive'],
#                   'penalty': ['none', 'l2', 'l1', 'elasticnet']}
# cross_validation = StratifiedKFold(n_splits = 10)
# grid_search = GridSearchCV(learner,
#                            param_grid = parameter_grid,
#                            cv = cross_validation)
# grid_search.fit(X_train, y_train)
# print('Best score: {}'.format(grid_search.best_score_))
# print('Best parameters: {}'.format(grid_search.best_params_))

# grid_search.best_estimator_
# Stochastic Gradient Descent: SGDRegressor
learner = SGDRegressor(loss = 'squared_loss', penalty = 'l2', random_state = random_state)
learner.fit(X_train, y_train)
y_test_pred = learner.predict(X_test)
y_train_pred = learner.predict(X_train)
# Check results visually
print(y_test_pred, y_test)
# Check results visually
plt.scatter(y_test, y_test_pred, c = 'orange', alpha = 0.1)
abline(1,0)
plt.xlabel('Observations')
plt.ylabel('Predictions')
plt.xlim(-10, 1000)
plt.ylim(-10, max(y_test_pred) + 5)
plt.show()
# Check MAPE
MAPE_test = MAPE(y_test, y_test_pred)
MAPE_train = MAPE(y_train, y_train_pred)
print(MAPE_test, MAPE_train)<jupyter_output>0.56119 0.56029
{'fraction_non_alphanumeric': 0.11832958627283768, 'fraction_numerical': 0.016745484146057834, 'mean_word_length': 5.795083406496927, 'pattern_counts': {'":': 0, '<': 136, '<?xml version=': 0, '>': 145, 'https://': 10, 'lorem ipsum': 0, 'www.': 7, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '625825', 'license': 'MIT', 'n_tokens_mistral': 13514, 'n_tokens_neox': 12654, 'n_words': 3258, 'path': 'FinalProject/wine-reviews-analysis.ipynb', 'repo_name': 'ValRCS/RCS_Data_Analysis_Python_04_20', 'size': '38727'}