<jupyter_start><jupyter_text>Data lookup In this first section of the data exploration chapter, an initial data screening will be conducted with descriptive statistics in order to provide a high level understanding of the dataset.This dataset contains reviews and metadata of video-games sold in Amazon. It includes hundreds of thousands of reviews spanning May 1996 - July 2014.<jupyter_code>import pandas as pd
import numpy as np

#Reproduce the same result every time if the script is kept consistent otherwise each run will produce different results (for classification)
np.random.seed(500)
    
#[1] Read the data
Corpus = pd.read_json(r"C:\Users\Panos\Desktop\Dissert\Code\Video_Games_5.json", lines=True, encoding='latin-1')

#Remove review with blank reviewText
Corpus = Corpus[~Corpus['reviewText'].isnull()]

#Print the first 3 rows
Corpus.iloc[:3]<jupyter_output><empty_output><jupyter_text>As a first step, the data were imported from a JSON file. After that, an initial data screening was conducted in order to understand the data structure by printing out the first three rows of the dataset. It is noticeable that there are a few columns that can be assumed unnecessary for the main goal for this moment (reviewerID, reviewTime, etc.). Although, almost all columns can be useful for extracting information which can help improve the model later. The most important columns for classification, not only in natural language processing (NLP) are, first, the data which the model will be trained on in order to categorize every review into the second most important column, the labels. For this classification problem, those columns are 'reviewText' and 'overall'. For a row in the dataset, 'reviewText' includes a review that a user left for a specific product (video-game) and the 'overall' contains the rating he submitted for that product from one to five. In this case the model will 'decide' in which rating category a given review text would fit best. For the next step, it is important to print out some valuable counts from the dataset.<jupyter_code>import nltk
from nltk.probability import FreqDist
import os
from collections import Counter
import statistics

#Calculate the total number of unique video games
asinString = " ".join(Corpus['asin'])
tokens = nltk.word_tokenize(asinString)
fdist = FreqDist(tokens)
count_asin = len(fdist)

# Calculate the total number of unique reviewers
reviewerIDString = " ".join(Corpus['reviewerID'])
tokens = nltk.word_tokenize(reviewerIDString)
fdist = FreqDist(tokens)
count_reviewerID = len(fdist)

# Set data in variables for the dataframe
totalReviews = int(Corpus.shape[0])
file_size = os.path.getsize(r"C:\Users\Panos\Desktop\Dissert\Code\Video_Games_5.json")
fileSize = int(round((file_size/2**20),2))
avgPerUser = int(round(statistics.mean(Counter(Corpus['reviewerID']).values()),2))
avgPerVG = int(round(statistics.mean(Counter(Corpus['asin']).values()),2))
minReviews = min(Counter(Corpus['asin']).values())
maxReviews = max(Counter(Corpus['asin']).values())

# Create a new dataframe to print a clear table
d1 = {'Description': ["Total reviews",
                      "File size (MB)",
                      "Total number of unique video games",
                      "Total number of unique reviewers",
                      "Average number of reviews per user", 
                      "Average number of reviews per videogame",
                      "Videogame with the minimum reviews",
                      "Videogame with the maximum reviews"],
      'Data': [totalReviews, fileSize, count_asin, count_reviewerID, avgPerUser, avgPerVG, minReviews, maxReviews]}

df1 = pd.DataFrame(data=d1)
df1<jupyter_output><empty_output><jupyter_text>The dataset consists of 231780 records which are not considered alot for a classification problem although it is enough data for building a model with a descent accuracy percent. Also, the dataset contains reviews from 10672 different games reviewed by 24303 different users which means that there is a variety of different personalities to help build a generic model.<jupyter_code>#Save this session for the next notebook
import dill
dill.dump_session('notebook_env.db')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07522543901281442, 'fraction_numerical': 0.01044138585666825, 'mean_word_length': 4.657718120805369, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '480045', 'license': 'MIT', 'n_tokens_mistral': 1162, 'n_tokens_neox': 1115, 'n_words': 490, 'path': '[Data_Exploration]Data_Lookup.ipynb', 'repo_name': 'panayiotiska/Sentiment-Analysis-Reviews', 'size': '4214'}