<jupyter_start><jupyter_text>ADM 2021 Homework 2  <NAME>  Matricola : 1998072 . email : <EMAIL>  ========================================================================= Imports and other housekeeping<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import Markdown as md
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import gc
from statsmodels.stats.weightstats import ztest
from scipy.stats.stats import pearsonr<jupyter_output><empty_output><jupyter_text>Important: Please not that for the sake of simplicity of execution and understanding, each Research Question below is independant and does not depend on data loaded or variables calculated in any previous Research Questions. The import of libraries and any other cells above this text still need to be run.  *RQ1: EDA*  Basic EDABasic EDA is performed on a chunk of main data. The conclusions in basic EDA are assumed to hold for complete main data Load sample dataset:<jupyter_code># load smaller chunk (~4 million rows) of main dataset for basic EDA.

chunksize = 4000000
dataset = pd.read_csv('steam_reviews.csv', nrows = chunksize)<jupyter_output><empty_output><jupyter_text>Check attributes in data:<jupyter_code>dataset.shape[1] #check number of columns
dataset.columns #check columns list<jupyter_output><empty_output><jupyter_text>Check sample rows/content:<jupyter_code>#have a cursory look at the data
dataset.head(3)
#data types. Assume consistent for larger set as well
dataset.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 4000000 entries, 0 to 3999999
Data columns (total 23 columns):
 #   Column                          Dtype  
---  ------                          -----  
 0   Unnamed: 0                      int64  
 1   app_id                          int64  
 2   app_name                        object 
 3   review_id                       int64  
 4   language                        object 
 5   review                          object 
 6   timestamp_created               int64  
 7   timestamp_updated               int64  
 8   recommended                     bool   
 9   votes_helpful                   int64  
 10  votes_funny                     int64  
 11  weighted_vote_score             float64
 12  comment_count                   int64  
 13  steam_purchase                  bool   
 14  received_for_free               bool   
 15  written_during_early_access     bool   
 16  author.steamid                  int64  
 17  author.num_games_owned     [...]<jupyter_text>Basic EDA Report:The facts in the dataset are the **Game** (app), **User**, and **Review** IDs. The dimensions for the game are it's name and whether it was a free purchase or not. The dimensions for the user are their profile statistics like games owned , playtimes and review activity. The dimensions for the reviews themselves are the language, review time and the vote information.There are a total of 4 boolean, 5 float, 3 string and 11 integer fields, counting the index. 3 of the 11 integer fields are actually unix timestamps.  Advanced EDA Due to memory constraints on local machine and free services on the cloud (Sagemaker, Colab) , EDA is performed by looping on dataset multiple times*, gathering statistics for each chunk, and then combining those statistics for final report.*This is less ideal than having higher memory specs, as previous chunk is lost to us and unavailable for further processing if next one is uploaded.<jupyter_code>#overall 

null_check = []
total_records = 0

error_records = 0

#define arrays for each analysis target

app_name = []
language = []
recommendation = []


chunksize = 1000000


#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize):
  total_records += datachunk.shape[0]
  error_records += datachunk[datachunk.steam_purchase == datachunk.received_for_free].shape[0]
  null_check.append(datachunk.isna().sum())
  app_name.append(datachunk.app_name.value_counts())
  language.append(datachunk.language.value_counts())
  recommendation.append(datachunk.recommended.value_counts())<jupyter_output><empty_output><jupyter_text>Bonus data was also tried, but a datatype warning while loading showed problems with data entries (intentional or unintentional) that were taking time to solve in future excercises. (was not simply a datatype conversion). So incorporation of Bonus data was dropped **Total Size of Dataset**<jupyter_code>print('Total records in dataset :', total_records)<jupyter_output>Total records in dataset : 21747371<jupyter_text>**Suspicious Records** Records where the 'Steam Purchase' and 'Received for free' flags were the same, since logically you cannot buy a game and receive it free too, or vice versa; you cannot both not buy a game and not receive it for free<jupyter_code>print('Suspicious records in dataset:', error_records)
print('Percentage of total:', error_records*100/total_records)<jupyter_output>Suspicious records in dataset: 4874918
Percentage of total: 22.416125608929924<jupyter_text>This is a surprisingly high number. Gather more data understanding: Need further discussion with TAs/Professor/Owner of data, as to reason why both these values could be same.For now, for any analysis in Research questions, we cannot discard these as this is too high a percentage. **Percentage of nulls in each column**<jupyter_code>(pd.concat(null_check, axis = 1).sum(axis = 1)*100/total_records).plot.bar(xlabel='Data Columns', ylabel = 'Percentage of Nulls', title= 'Percentage of nulls in each column')

md('Nulls are only present in two columns for negligible percentage of records')<jupyter_output><empty_output><jupyter_text>**Games/Applications**<jupyter_code>app_values = pd.concat(app_name, axis = 1).sum(axis = 1)
print('Unique Games in dataset: ', app_values.shape[0])
print('Game with highest records: ', app_values.idxmax(),'with', int(app_values.max()), 'records')
print('Game with highest records: ', app_values.idxmin(),'with', int(app_values.min()), 'records')

(app_values.sort_values(axis = 0, ascending = False)[:20] * 100 /total_records).plot.bar(xlabel='Games', ylabel = 'Percentage of Records', title ='Percentage distribution of records for top 20 games')
md('The top 20 games make up of about {} percent of total records'.format(round(app_values.sort_values(axis = 0, ascending = False)[:20].sum()*100/total_records),2))<jupyter_output><empty_output><jupyter_text>**Languages**<jupyter_code>langs = pd.concat(language, axis = 1).sum(axis = 1)
print('Unique Languages in dataset: ', langs.shape[0])

print('Top 3 languages used for reviews:')
print(langs.sort_values(axis = 0, ascending = False)[0:3])
print('==========================================================================================')
(langs.sort_values(axis = 0, ascending = False) * 100 /total_records).plot.bar(xlabel='Languages', ylabel = 'Percentage of Records', title ='Percentage distribution of languages')<jupyter_output>Unique Languages in dataset:  28
Top 3 languages used for reviews:
english     9635437
schinese    3764967
russian     2348900
dtype: int64
==========================================================================================<jupyter_text>**Review Recommendation**<jupyter_code>recomm = pd.concat(recommendation, axis = 1).sum(axis = 1).plot.pie(label='Game Recommended?', labels =['Yes', 'No'], shadow = True, autopct='%1.1f%%')

print('Reviews in this dataset are skewed towards recommending a game')
#clean up

del dataset
gc.collect()<jupyter_output><empty_output><jupyter_text>*RQ2: Simple Insights*<jupyter_code>## Loop cell

review_counts = []
max_w_v_score = []
recomm_yes = []
purchase = []

chunksize = 1000000


#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize, 
                             usecols = ['app_name', 'review_id', 'weighted_vote_score', 'recommended','steam_purchase']):
  
  review_counts.append(datachunk.groupby('app_name').review_id.count()) #number of reviews
  max_w_v_score.append(datachunk.groupby('app_name').weighted_vote_score.max()) #maximum weighted score
  recomm_yes.append(datachunk.groupby('app_name').recommended.sum()) #recommendations of purchase
  purchase.append(datachunk.groupby('app_name').steam_purchase.sum()) #purchase_count<jupyter_output><empty_output><jupyter_text>**Plot the number of reviews for each application in descending order.**<jupyter_code>num_reviews = pd.concat(review_counts, axis = 1).sum(axis = 1)
print('double click on plot to zoom in and see')
(num_reviews.sort_values(axis = 0, ascending = False)).plot.bar(figsize=(75,5), xlabel='Applications', ylabel = 'Reviews', title ='No. of Reviews for each application')<jupyter_output>double click on plot to zoom in and see<jupyter_text>**What applications have the best Weighted Vote Score?**<jupyter_code>max_scores = pd.concat(max_w_v_score, axis = 1).max(axis = 1) #maximum score amongst the maximums of subsets
print('The top 5 applications with the best weighted vote scores in the entire dataset are:')
print(max_scores.sort_values(axis = 0, ascending = False)[0:5])<jupyter_output>The top 5 applications with the best weighted vote scores in the entire dataset are:
app_name
Stardew Valley              0.995987
Divinity: Original Sin 2    0.995245
Subnautica                  0.994453
Mirror                      0.994038
Wallpaper Engine            0.993540
dtype: float64<jupyter_text>**Which applications have the most and the least recommendations?**<jupyter_code>yes_recomms = pd.concat(recomm_yes, axis = 1).sum(axis = 1) #total yes recommendations


combined = pd.concat([yes_recomms, num_reviews], axis = 1) #combine review counts and recommendations


print('The top 3 applications with the MOST recommendations are:')
print(yes_recomms.sort_values(axis = 0, ascending = False)[0:3])
print('----------------------------------------------------------------------------------------------------------')
print('The top 3 applications with the MOST recommendations (as a percentage of their review counts)  are:')
print((combined[0]*100/combined[1]).sort_values(axis = 0, ascending = False)[0:3])
print('----------------------------------------------------------------------------------------------------------')
print('The top 3 applications with the LEAST recommendations are:')
print(yes_recomms.sort_values(axis = 0, ascending = True)[0:3])
print('----------------------------------------------------------------------------------------------------------')
print('The top 3 applications with the LEAST recommendations (as a percentage of their review counts)  are:')
print((combined[0]*100/combined[1]).sort_values(axis = 0, ascending = True)[0:3])<jupyter_output>The top 3 applications with the MOST recommendations are:
app_name
PLAYERUNKNOWN'S BATTLEGROUNDS     886417.0
Grand Theft Auto V                831311.0
<NAME>'s Rainbow Six Siege    742397.0
dtype: float64
----------------------------------------------------------------------------------------------------------
The top 3 applications with the MOST recommendations (as a percentage of their review counts)  are:
app_name
ULTRAKILL       99.534384
Senren＊Banka    99.404052
A Short Hike    99.144861
dtype: float64
----------------------------------------------------------------------------------------------------------
The top 3 applications with the LEAST recommendations are:
app_name
Neon Hardcorps              89.0
DRAGON QUEST HEROES™ II    423.0
Identity                   439.0
dtype: float64
----------------------------------------------------------------------------------------------------------
The top 3 applications with the LEAST recommendations (as a percentage of their revi[...]<jupyter_text>**How many of these applications were purchased, and how many were given for free?** For this question, suspicious records  where "purchase" and "free" indicators are both the same, we consider "purchase" indicator as authentic. i.e. If purchase is true, that is considered a purchase, even if received for free is true as well.<jupyter_code>num_purchase = pd.concat(purchase, axis = 1).sum(axis = 1) #total purchased
combined = pd.concat([num_purchase, num_reviews], axis = 1) #combine review counts and purchase counts

print('Total purchased applications:', combined[0].sum(), "({} %)".format(combined[0].sum()*100/combined[1].sum()))
print('Total given for free:', combined[1].sum() - combined[0].sum() , "({} %)".format((combined[1].sum() - combined[0].sum())*100/combined[1].sum()))<jupyter_output>Total purchased applications: 16842432.0 (77.44583011896013 %)
Total given for free: 4904939.0 (22.554169881039872 %)<jupyter_text>*RQ3: Time of Reviews* **What is the most common time that authors review an application?**<jupyter_code>def get_review_frequency():

  #this function reads the steam dataset and 
  #outputs a matrix of review counts for each hour, minute, and second of the day


  def date_conv(unix_d):
    return pd.to_datetime(unix_d, unit = 's')


  t_freq_3 = np.zeros((24,60,60))   #review_frequency_3d_matrix

  chunksize = 2000000

  #calculate statistics
  for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize, usecols = ['review_id', 'timestamp_created'],
                              parse_dates=['timestamp_created'], 
                              date_parser=date_conv ):
    
    for ts in datachunk['timestamp_created'].to_list():
      h = ts.hour
      m = ts.minute
      s = ts.second
      t_freq_3[h, m, s] += 1

  return t_freq_3  



t_freq_3 = get_review_frequency() #Frequency of reviews at hour, min and second grain

t_freq_2 = t_freq_3.sum(axis = 2) #since questions asks for common time in hours and mins, roll up this to a 2d matrix

best_time = np.where(t_freq_2 == np.amax(t_freq_2)) #index of maximum value in our 2d matrix

print('The most common time that authors review an application is at ', best_time[0][0], ':', best_time[1][0])
print('The total reviews seen in the data at that time are', int(np.amax(t_freq_2)))<jupyter_output>The most common time that authors review an application is at  14 : 50
The total reviews seen in the data at that time are 19502<jupyter_text>**Create a function that receives as a parameter a list of time intervals and returns the plot the number of reviews for each of the intervals.**<jupyter_code>def plot_reviews(interval_list):
  #input is an list of intervals. The function assumes that each interval in this list
  #is a tuple of size 2 containing initial time and final time in string format
  
  #===================================================================================
  def get_review_frequency():

    #this function reads the steam dataset and 
    #outputs a matrix of review counts for each hour, minute, and second of the day


    def date_conv(unix_d):
      return pd.to_datetime(unix_d, unit = 's')


    t_freq_3 = np.zeros((24,60,60))   #review_frequency_3d_matrix

    chunksize = 2000000
    
    #calculate statistics
    for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize, usecols = ['review_id', 'timestamp_created'],
                                 parse_dates=['timestamp_created'], date_parser=date_conv ):

      datachunk = datachunk.iloc[:,1:] #removing index
      
      for ts in datachunk['timestamp_created'].to_list():
        h = ts.hour
        m = ts.minute
        s = ts.second
        t_freq_3[h, m, s] += 1
      
    return t_freq_3  


  #============================================================================================

  t_freq_3 = get_review_frequency()

  y_arr = [] #array of num reviews. y axis of plot
  x_arr = [] #array of range in x axis
  for timestamps in interval_list:
    start, end = timestamps[0], timestamps[1]
    start_h, start_m, start_s  = map(int, start.split(':'))
    end_h, end_m, end_s  = map(int, end.split(':'))
    y_arr.append(t_freq_3[start_h:end_h+1, start_m:end_m+1, start_s:end_s+1].sum())
    x_arr.append(start + ' to ' + end)
    
 
  plt.figure(figsize=(len(interval_list),5))
  plt.xticks(rotation=90)
  plt.xlabel('Time Intervals')
  plt.ylabel('Number of Reviews')
  plt.bar(x_arr,y_arr)
  plt.show()<jupyter_output><empty_output><jupyter_text>**Use the function that you created in the previous literal to plot the number of reviews between the following time intervals**<jupyter_code>time_list =[('06:00:00', '10:59:59'), ('11:00:00', '13:59:59'), 
            ('14:00:00', '16:59:59'), ('17:00:00', '19:59:59'),
            ('20:00:00', '23:59:59'), ('00:00:00', '02:59:59'),
            ('03:00:00', '05:59:59')]

plot_reviews(time_list)
#clean up
del t_freq_3
del t_freq_2
gc.collect()<jupyter_output><empty_output><jupyter_text>*RQ4: Languages of Reviews* **What are the top 3 languages used to review applications?** As already determined in advanced EDA in *RQ1* above (Section: Languages), the top 3 languages used to review applications are:1.   English2.   Schinese3.   Russian **Create a function that receives as parameters both the name of a data set and a list of languages’ names and returns a data frame filtered only with the reviews written in the provided languages.** Note: The problem statement asks us to return dataframe filtered only with reviews, but we will also return some additional columns besides the review, like votes_funny and votes_helpful, to make the returned dataframe useful for next parts of the question.<jupyter_code>def filter_on_lang(dataset_name, language_list):

  #this function reads a dataset with the dataset name and 
  #outputs a dataframe of reviews with only languages in language list.

  filtered_df = pd.DataFrame()

  chunksize = 2000000

  #calculate statistics
  for datachunk in pd.read_csv(dataset_name, chunksize=chunksize, 
                               usecols =['review_id', 'review','language', 'votes_funny', 'votes_helpful']):
    
    datachunk = datachunk[datachunk.language.isin(language_list)]
    filtered_df = filtered_df.append(datachunk)
    
  return filtered_df<jupyter_output><empty_output><jupyter_text>**Use the function created in the previous literal to find what percentage of these reviews (associated with the top 3 languages) were voted as funny?** any review with > 0 in "votes_funny" is counted as a funny review<jupyter_code>top_3 = ['english', 'schinese', 'russian']
lang_df = filter_on_lang("steam_reviews.csv", top_3)
result = pd.concat([lang_df[lang_df.votes_funny > 0].groupby(['language']).votes_funny.count(),lang_df.groupby(['language']).review_id.count()], axis = 1)
print('Percentage of reviews (for top 3 languages) voted as funny:')
print((result.votes_funny*100/result.review_id))
#clean up
del lang_df
gc.collect()<jupyter_output><empty_output><jupyter_text>**Use the function created in the literal “a” to find what percentage of these reviews (associated with the top 3 languages) were voted as helpful?** any review with > 0 in "votes_helpful" is counted as a helpful review<jupyter_code>top_3 = ['english', 'schinese', 'russian']
lang_df = filter_on_lang("steam_reviews.csv", top_3)
result = pd.concat([lang_df[lang_df.votes_helpful > 0].groupby(['language']).votes_helpful.count(),lang_df.groupby(['language']).review_id.count()], axis = 1)
print('Percentage of reviews (for top 3 languages) voted as helpful:')
print((result.votes_helpful*100/result.review_id))
#clean up
del lang_df
gc.collect()<jupyter_output><empty_output><jupyter_text>*RQ5: Users* For this question, records where "purchase" and "free" indicators are both the same, we consider "purchase" indicator as source of truth. i.e. If purchase is true, that is considered a purchase, even if received for free is true as well. **Plot the top 10 most popular reviewers and the number of reviews.**<jupyter_code>author_data = []

chunksize = 1000000


#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize,
                             usecols = ['author.steamid', 'review_id']):
  author_data.append(datachunk.groupby(['author.steamid']).count())

author_reviews = pd.concat(author_data, axis = 1).sum(axis = 1)

(author_reviews.sort_values(axis = 0, ascending = False)[0:10]).plot.bar(xlabel='Author Steam ID', ylabel = 'Number of Reviews', title ='No. of Reviews of 10 most popular authors')<jupyter_output><empty_output><jupyter_text>Now we have the steam ID of the most popular author and can deep dive into further questions about them<jupyter_code>pop_user = author_reviews.sort_values(axis = 0, ascending = False)[0:1].index.to_list()[0]  #steam ID for most popular user
#get data for most popular user

chunksize = 1000000

pop_user_df = pd.DataFrame()

#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize,
                             usecols = ['author.steamid', 'app_name', 'received_for_free', 'steam_purchase', 'recommended']):
  pop_user_df = pop_user_df.append(datachunk[datachunk['author.steamid'] == pop_user])<jupyter_output><empty_output><jupyter_text>**What applications did the most popular author review?**<jupyter_code>print('List of applications reviewd by most popular user:', sorted(list(pop_user_df.app_name.unique())))<jupyter_output>List of applications reviewd by most popular user: ['A Hat in Time', 'ARK: Survival Evolved', 'Age of Empires II (2013)', 'Age of Empires: Definitive Edition', 'American Truck Simulator', 'Among Us', 'Ancestors Legacy', 'Axiom Verge', 'Banished', 'BattleBlock Theater', 'BioShock Infinite', 'Black Desert Online', 'Black Mesa', 'Blackwake', 'Bomber Crew', 'Broforce', 'Castle Crashers', 'Cave Story+', 'Celeste', 'Cities: Skylines', 'Conan Exiles', 'Counter-Strike: Source', 'Cuphead', 'DARK SOULS™ III', 'DOOM', 'DOOM Eternal', 'DUSK', 'Darksiders III', 'Day of Infamy', 'Dead Cells', 'Dead Rising 4', 'Dead by Daylight', 'Deep Rock Galactic', 'Devil May Cry HD Collection', 'Dishonored', "Don't Escape: 4 Days to Survive", "Don't Starve", 'Duck Game', 'Dying Light', 'Enter the Gungeon', 'Euro Truck Simulator 2', 'Europa Universalis IV', 'FAR: Lone Sails', 'FTL: Faster Than Light', 'Factorio', 'Fallout 4', 'Finding Paradise', 'Foundation', 'Frostpunk', 'GRIS', "Garry's Mod", 'Getting Over It wi[...]<jupyter_text>**How many applications did he purchase, and how many did he get as free?**<jupyter_code>purchase_dict  = pop_user_df.steam_purchase.value_counts().to_dict()

print('Count of applications purchased:',purchase_dict[True])
print('Percentage of applications purchased:',round(purchase_dict[True]*100/sum(purchase_dict.values()),2))


print('Count of applications gotten free:',purchase_dict[False])
print('Percentage of applications gotten free:',round(purchase_dict[False]*100/sum(purchase_dict.values()),2))<jupyter_output>Count of applications purchased: 110
Percentage of applications purchased: 73.83
Count of applications gotten free: 39
Percentage of applications gotten free: 26.17<jupyter_text>**How many of the applications he purchased reviewed positively, and how many negatively? How about the applications he received for free?**<jupyter_code>pop_user_df.groupby(['steam_purchase','recommended']).app_name.count().to_frame().sort_values(by= ['steam_purchase', 'recommended'], ascending = False).rename(columns = {'app_name':'Number of Applications'})<jupyter_output><empty_output><jupyter_text>*RQ6: Updates to Reviews* The assumption taken in this exercise is that reviews are updated only if timestamps of creation and update are different<jupyter_code>def date_conv(unix_d):
  return pd.to_datetime(unix_d, unit = 's')

total_diff_sec = 0
update_counts = 0

updating_authors = []

chunksize = 2000000

#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize, 
                             usecols = ['author.steamid', 'timestamp_created','timestamp_updated','review_id'],
                            parse_dates=['timestamp_created','timestamp_updated'], 
                            date_parser=date_conv ):
  
  datachunk = datachunk[datachunk.timestamp_updated != datachunk.timestamp_created] #only take data where reviews were updated
  
  updating_authors.append(datachunk.groupby(['author.steamid']).review_id.count())

  for difference in (datachunk['timestamp_updated'] - datachunk['timestamp_created']).to_list():
    total_diff_sec += difference.total_seconds()
    update_counts += 1<jupyter_output><empty_output><jupyter_text>**What is the average time (days and minutes) a user lets pass before he updates a review?**<jupyter_code>avg_sec =  total_diff_sec/update_counts

print('Average time a user lets pass before he updates a review:', int(avg_sec//(24*3600)), 'days and', round(((avg_sec % (24*3600))/60),2), 'minutes')<jupyter_output>Average time a user lets pass before he updates a review: 321 days and 46.41 minutes<jupyter_text>**Plot the top 3 authors that usually update their reviews.**<jupyter_code>author_updates = pd.concat(updating_authors, axis = 1).sum(axis = 1)

(author_updates.sort_values(axis = 0, ascending = False)[0:3]).plot.bar(xlabel='Author Steam ID', ylabel = 'Number of Reviews Updated', title ='No. of Updates of 3 most updating authors')<jupyter_output><empty_output><jupyter_text>*RQ7: Probabilities*<jupyter_code>count_score_big_equal = 0  #count of reviews where weighted score >=0.5
count_score_big = 0  #count of reviews where weighted score > 0.5
count_one_funny = 0 #count of reviews with atleast one funny vote
count_one_funny_score_big_equal = 0 #count of reviews where weighted score >= 0.5 AND atleast one funny vote
count_one_funny_score_big = 0 #count of reviews where weighted score > 0.5 AND atleast one funny vote

count_reviews = 0

chunksize = 2000000

#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize, 
                             usecols = ['review_id', 'votes_funny','weighted_vote_score']):
  
  count_reviews += datachunk.shape[0]
  count_score_big_equal += datachunk[datachunk.weighted_vote_score >= 0.5].shape[0]
  count_score_big += datachunk[datachunk.weighted_vote_score > 0.5].shape[0]
  count_one_funny += datachunk[datachunk.votes_funny >= 1].shape[0]  
  count_one_funny_score_big_equal += datachunk[(datachunk.votes_funny >= 1) & (datachunk.weighted_vote_score >= 0.5) ].shape[0]   
  count_one_funny_score_big += datachunk[(datachunk.votes_funny >= 1) & (datachunk.weighted_vote_score > 0.5) ].shape[0]<jupyter_output><empty_output><jupyter_text>**What’s the probability that a review has a Weighted Vote Score equal to or bigger than 0.5?**<jupyter_code>print('Probability that review has a Weighted Vote Score equal to or bigger than 0.5 =', count_score_big_equal/count_reviews)<jupyter_output>Probability that review has a Weighted Vote Score equal to or bigger than 0.5 = 0.21998484322541792<jupyter_text>**What’s the probability that a review has at least one vote as funny given that the Weighted Vote Score is bigger than 0.5?**<jupyter_code>p_a = count_one_funny/count_reviews
p_b = count_score_big/count_reviews
p_a_and_b = count_one_funny_score_big/count_reviews

p_a_given_b = p_a_and_b/p_b

print('Probability that a review has at least one vote as funny given that the Weighted Vote Score is bigger than 0.5 = ', p_a_given_b)<jupyter_output>Probability that a review has at least one vote as funny given that the Weighted Vote Score is bigger than 0.5 =  0.24978288914523247<jupyter_text>**Is the probability that “a review has at least one vote as funny” independent of the “probability that a review has a Weighted Vote Score equal or bigger than 0.5”?**<jupyter_code>p_a = count_one_funny/count_reviews
p_b = count_score_big_equal/count_reviews
p_a_and_b = count_one_funny_score_big_equal/count_reviews

print('P(A) => Probability that a review has at least one vote as funny =', p_a)
print('P(B) => Probability that a review has a weighted score equal or bigger than 0.5 =', p_b)
print('P(A and B) => Probability that a review has at least one vote as funny AND has a weighted score equal or bigger than 0.5 =', p_a_and_b)
print('Since P(A and B) != P(A)*P(B), these events are not independant')<jupyter_output>P(A) => Probability that a review has at least one vote as funny = 0.1196959393390585
P(B) => Probability that a review has a weighted score equal or bigger than 0.5 = 0.21998484322541792
P(A and B) => Probability that a review has at least one vote as funny AND has a weighted score equal or bigger than 0.5 = 0.05480736039312522
Since P(A and B) != P(A)*P(B), these events are not independant<jupyter_text>*RQ8: Tests and Analysis* **Is there a significant difference in the Weighted Vote Score of reviews made in Chinese vs the ones made in Russian?** In this question , 'Chinese' reviews are considered where reviews are in 'schinese'. Considering that player population of China is huge, logically the larger more significant language (i.e. which has more reviews in data) should be chosen. so 'tchinese' is not considered.Also, the data with weighted vote scores = 0 is excluded, which assumes there was no voting on that review<jupyter_code>filtered_df = pd.DataFrame()



chunksize = 2000000

#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize, 
                           usecols =['weighted_vote_score','language']):

    
    datachunk = datachunk[datachunk.weighted_vote_score > 0] #take data where there is a valid weighted score
    datachunk = datachunk[datachunk.language.isin(['schinese', 'russian'])]
    filtered_df = filtered_df.append(datachunk)<jupyter_output><empty_output><jupyter_text>*Box Plot*<jupyter_code>filtered_df.boxplot(by=['language'], figsize= (5, 6))<jupyter_output><empty_output><jupyter_text>*Z Test:* Null Hypothesis: Weighted vote scores for both are not significantly different<jupyter_code>z_stat, p_val = ztest(filtered_df[filtered_df.language == 'russian']['weighted_vote_score'].to_list(), 
      filtered_df[filtered_df.language == 'schinese']['weighted_vote_score'].to_list(),
      value=0) 
print('Z statistic =', z_stat)
print('P Value =', p_val)
print('Reject Null Hypothesis. The weighted vote scores for both languages ARE significantly different')

#clean up
del filtered_df
gc.collect()<jupyter_output><empty_output><jupyter_text>**Can you find any significant relationship between the time that a user lets pass before he updates the review and the Weighted Vote Score?**<jupyter_code>def date_conv(unix_d):
  return pd.to_datetime(unix_d, unit = 's')

w_v_score = []
update_lag = []

chunksize = 2000000

#calculate statistics
for datachunk in pd.read_csv("steam_reviews.csv", chunksize=chunksize, 
                             usecols = ['weighted_vote_score', 'timestamp_created','timestamp_updated'],
                            parse_dates=['timestamp_created','timestamp_updated'], 
                            date_parser=date_conv ):
  
  datachunk = datachunk[datachunk.weighted_vote_score > 0] #take data where there is a valid weighted score
  datachunk = datachunk[datachunk.timestamp_updated != datachunk.timestamp_created] #only take data where reviews were updated
  for difference in (datachunk['timestamp_updated'] - datachunk['timestamp_created']).to_list():
    update_lag.append(difference.total_seconds())
    
  for score in (datachunk['weighted_vote_score']).to_list():
    w_v_score.append(score)<jupyter_output><empty_output><jupyter_text>*Scatter Plot*<jupyter_code>plt.scatter(w_v_score, update_lag)
plt.show()<jupyter_output><empty_output><jupyter_text>*Pearson Correlation*<jupyter_code>corr_coef, p_val = pearsonr(w_v_score, update_lag)

print('Correlation coefficient:', round(corr_coef, 4))
print('P Value:' , p_val)
print('We can rely on the coefficient since p value < 0.05 and correlation coefficient is statistically significant')
print('The correlation is very low. Hence there is no significant relationship.')
#clean up
del w_v_score
del update_lag
gc.collect()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1329494241362043, 'fraction_numerical': 0.027916875312969453, 'mean_word_length': 4.392911392405063, 'pattern_counts': {'":': 0, '<': 151, '<?xml version=': 0, '>': 173, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '40', 'dir': 'jupyter-structured-clean-dedup', 'id': '259356', 'license': 'MIT', 'n_tokens_mistral': 10352, 'n_tokens_neox': 9538, 'n_words': 3011, 'path': 'main.ipynb', 'repo_name': 'Jethrodeen/uniroma1-ADM-hw2', 'size': '31982'}