<jupyter_start><jupyter_text>Steam Games Exploratory data analysis**Author:** <NAME>  IntroductionThis end-to-end data science project started as a final assignment for a Data Science course at Milwaukee School of Engineering. In this project, we will explore real-world data which contains information about video games published on Steam - Digital distribution platform developed by Valve Corporation. We will go through the basic process of data science and explore this particular data in the following sections. References:* Steam Game Data [dataset] by <NAME>* Introduction to Data Science by <NAME>, Ph.D.* A Comprehensive Guide to Data Exploration by Sunil Ray  HypothesisFirst of all, we need to get our hypotheses ready. Here are some questions we will try to answer and problems we will try to solve:* **Do Free-To-Play games have more DLCs?**    * Free games, in order to make a profit, could have more paid DLCs than games that are not free-to-play. Of course, there are other methods (e.g. Microtransactions) which could be more preferable among free-to-play games.    * After a brief search on Steam, we can see that paid DLCs are pretty common among the bigger free-to-play games (extensions packs, collector packs...).    * **Hypothesis 1:** Free-to-Play games have on average more DLCs than paid games.* **Examine increase in games that support VR according to time**    * We could see a rise in popularity of VR games in recent years. How many games with VR support have been released? Can we see a significant increase in this type of games?    * **Hypothesis 2:** Each year the number of games with VR support being released increases.* **What features contribute to number of owners and can we predict it?**Besides these questions, we will also try to take as many observations as we can to understand this data better.  DatasetFor this Data Science project, we will use a dataset created by <NAME>, that contains 13357 steam games (unfortunately with some duplicates). The dataset was created at the end of the year 2016. Information**Name**: Steam Game Data**Author**: <NAME>**Data Source**: Combination of data from publicly available Steam API's and steamspy.com**Links**:* Dataset available on data.world (with details)* Full details on GitHub  Attributes* **QueryID** - (Integer) The original ID* **ResponseID** - (Integer) The ID returned in the Steam response (should equal QueryID)* **QueryName** - (Text) The original name* **ResponseName** - (Text) The name returned in the Steam response (should equal QueryName)* **ReleaseDate** - (Text) Appears to the be the initial release date for the game* **RequiredAge** - (Integer) required age for the game* **DemoCount** - (TextualCount) number of demos* **DeveloperCount** - (TextualCount) number of developers* **DLCCount** - (TextualCount) number of DLCs* **Metacritic** - (Integer) numeric score from metacritic* **MovieCount** - (TextualCount) number of videos on Store Page* **PackageCount** - (TextualCount) number of bundles the game occurs in* **RecommendationCount** - (Integer) number of recomendations* **PublisherCount** - (TextualCount) number of publishers* **ScreenshotCount** - (TextualCount) number of screenshots on Store Page* **AchievementCount** - (Integer) number of achievements* **AchievementHighlightedCount** - (TextualCount) number of highlighted achievements* **ControllerSupport** - (Boolean) True if the game has full controller support* **IsFree** - (Boolean) True if the game is free* **FreeVerAvail** - (Boolean) True the game has a free version* **PurchaseAvail** - (Boolean) True if game cost more than 0 cents with discounts* **SubscriptionAvail** - (Boolean) True if the game has a recurring subscription option* **PlatformWindows** - (Boolean) True if Windows platform is supported* **PlatformLinux** - (Boolean) True if Linux platform is supported* **PlatformMac** - (Boolean) True if Mac platform is supported* **PCReqsHaveMin** - (Boolean) True if PC minimal requirements are available* **PCReqsHaveRec** - (Boolean) True if PC recommended requirements are available* **LinuxReqsHaveMin** - (Boolean) True if Linux minimal requirements are available* **LinuxReqsHaveRec** - (Boolean) True if Linux recommended requirements are available* **MacReqsHaveMin** - (Boolean) True if Mac minimal requirements are available* **MacReqsHaveRec** - (Boolean) True if Mac recommended requirements are available* **CategorySinglePlayer** - (Boolean) True if category is Singleplayer* **CategoryMultiplayer** - (Boolean) True if category is Cross-platform multiplayer/Local multi-player/Multi-player/Online multi-player/Shared screen/Split screen* **CategoryCoop** - (Boolean) True if category is Co-op/Local co-op/Online co-op* **CategoryMMO** - (Boolean) True if category is MMO* **CategoryInAppPurchase** - (Boolean) True if game has in-app purchases* **CategoryIncludeSrcSDK** - (Boolean) True if game includes source sdk* **CategoryIncludeLevelEditor** - (Boolean) True if game includes level editor* **CategoryVRSupport** - (Boolean) True if game has VR support* **GenreIsNonGame** - (Boolean) True if genre is Utilities/Design & illustration/Animation & modeling/Software training/Education/Audio production/Video production/Web publishing/Photo editing/Accounting* **GenreIsIndie** - (Boolean) True if genre is indie* **GenreIsAction** - (Boolean) True if genre is action* **GenreIsAdventure** - (Boolean) True if genre is adventure* **GenreIsCasual** - (Boolean) True if genre is casual* **GenreIsStrategy** - (Boolean) True if genre is strategy* **GenreIsRPG** - (Boolean) True if genre is rpg* **GenreIsSimulation** - (Boolean) True if genre is simulation* **GenreIsEarlyAccess** - (Boolean) True if genre is early access* **GenreIsFreeToPlay** - (Boolean) True if genre is free to play* **GenreIsSports** - (Boolean) True if genre is sports* **GenreIsRacing** - (Boolean) True if genre is racing* **GenreIsMassivelyMultiplayer** - (Boolean) True if genre is massively multiplayer* **PriceCurrency** - (Text) Currency of the price* **PriceInitial** - (Float) Initial price of the game* **PriceFinal** - (Float) Final price of the game* **SteamSpyOwners** - [steamspy.com] total owners, which includes free weekend trials and other possibly spurious numbers* **SteamSpyOwnersVariance** - [teamspy.com] total owners, which includes free weekend trials and other possibly spurious numbers. Note that this is not technically variance: according to steamspy.com, "the real number... lies somewhere on... (value +/- variance)"* **SteamSpyPlayersEstimate** - [steamspy.com] best estimate of total number of people who have played the game since March 2009* **SteamSpyPlayersVariance** - [steamspy.com] errors bounds on SteamSpyPlayersEstimate. Note that this is not technically variance: according to steamspy.com, "the real number... lies somewhere on... (value +/- variance)"* **SupportEmail** - (Textual) Support email address* **SupportURL** - (Textual) Support URL* **AboutText** - (Textual) About text* **Background** - (Textual) Background of the game* **ShortDescrip** - (Textual) Short description of the game* **DetailedDescrip** - (Textual) Detailed description of the game* **DRMNotice** - (Textual) Digital rights management notice* **ExtUserAcctNotice** - (Textual) ext_user_account_notice in JSON* **HeaderImage** - (Textual) Header image* **LegalNotice** - (Textual) Legal notice* **Reviews** - (Textual) Reviews* **SupportedLanguages** - (Textual) Supported langiages in the game* **Website** - (Textual) Game website* **PCMinReqsText** - (Textual) PC minimal requirements* **PCRecReqsText** - (Textual) PC recommended requirements* **LinuxMinReqsText** - (Textual) Linux minimal requirements* **LinuxRecReqsText** - (Textual) Linux recommended requirements* **MacMinReqsText** - (Textual) Mac minimal requirements* **MacRecReqsText** - (Textual) Mac recommended requirements  Data preprocessingIn this section, we will look at the actual data closely and preprocess the data for our analysis and modeling. Let's include some libraries we will need.<jupyter_code>import pandas as pd # powerful data structures for data analysis
import numpy as np # advanced math functionalities
import scipy.stats as stats # statistical functions
import matplotlib.pyplot as plt # a numerical plotting library
import datetime as dt # date and time manipulation module
import seaborn as sns # statistical data visualization

from sklearn import metrics # metrics for evaluating
from sklearn.model_selection import train_test_split # splitting dataset into train and test portions
from sklearn.preprocessing import MinMaxScaler # scaling numeric values
from sklearn.preprocessing import LabelEncoder # scaling categorical values

import tensorflow as tf
from tensorflow import keras

# plots within notebook versus launching a separate window
%matplotlib inline

# our helper functions from utils.py
exec(compile(open("utils/utils.py", "rb").read(), "utils/utils.py", 'exec'))

#sets up pandas table display
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)

# configure seaborn to our liking
sns.set_style('darkgrid', {"axes.facecolor": ".95", "axes.edgecolor": "black", "xtick.bottom": "True", \
                           "ytick.left": "True", 'patch.force_edgecolor': False})
sns.set_context("notebook")
sns.set_palette(["#28abe3", "#db3340", "#e8b71a", "#1fda9a", "#f7eac8"])
blend_palette = sns.diverging_palette(258, 12, sep=20, as_cmap=True)<jupyter_output><empty_output><jupyter_text>We will read the dataset into pandas for easy manipulation.<jupyter_code>df = pd.read_csv("data/games-features.csv") # read the data into panda's data frame
print('The shape of the dataset is {}.'.format(df.shape)) # print out the shape of the dataset
df.head(10) # show first 10 records<jupyter_output>The shape of the dataset is (13357, 78).<jupyter_text>Column SelectionNow let's go through all attributes and determine which attributes we will need for our analysis. We can list all 78 attributes:<jupyter_code>columns = list(df.columns)
print("{} columns:\n{}".format(len(columns),columns))<jupyter_output>78 columns:
['QueryID', 'ResponseID', 'QueryName', 'ResponseName', 'ReleaseDate', 'RequiredAge', 'DemoCount', 'DeveloperCount', 'DLCCount', 'Metacritic', 'MovieCount', 'PackageCount', 'RecommendationCount', 'PublisherCount', 'ScreenshotCount', 'SteamSpyOwners', 'SteamSpyOwnersVariance', 'SteamSpyPlayersEstimate', 'SteamSpyPlayersVariance', 'AchievementCount', 'AchievementHighlightedCount', 'ControllerSupport', 'IsFree', 'FreeVerAvail', 'PurchaseAvail', 'SubscriptionAvail', 'PlatformWindows', 'PlatformLinux', 'PlatformMac', 'PCReqsHaveMin', 'PCReqsHaveRec', 'LinuxReqsHaveMin', 'LinuxReqsHaveRec', 'MacReqsHaveMin', 'MacReqsHaveRec', 'CategorySinglePlayer', 'CategoryMultiplayer', 'CategoryCoop', 'CategoryMMO', 'CategoryInAppPurchase', 'CategoryIncludeSrcSDK', 'CategoryIncludeLevelEditor', 'CategoryVRSupport', 'GenreIsNonGame', 'GenreIsIndie', 'GenreIsAction', 'GenreIsAdventure', 'GenreIsCasual', 'GenreIsStrategy', 'GenreIsRPG', 'GenreIsSimulation', 'GenreIsEarlyAccess', 'GenreIsFreeToPlay[...]<jupyter_text>From the first 4 columns ('QueryID', 'ResponseID', 'QueryName', 'ResponseName'), we are really interested only in the response names. After a quick examination we would be able to see that there are some inconsistencies between 'QueryID' and 'ResponseID'. There are a lot more entries with 'QueryName' and 'ResponseName' mismatch. The reason for this is most likely that the Response Names are more specific and that's what we are interested - the actual names of the games. We can drop the first three columns and rather look for duplicates in the entries (there should be a lot of them).<jupyter_code># only for perspective
print('There are {} entries, where QuerryID != ResponseID.'.format(len(df[df.QueryID != df.ResponseID])))
print('There are {} entries, where QueryName != ResponseName.'.format(len(df[df.QueryName != df.ResponseName])))
# drop first three columns
df.drop(['QueryID', 'ResponseID', 'QueryName'], axis=1, inplace=True)
df.shape
# count duplicates
len(df[df.duplicated(subset=['ResponseName'], keep=False)])<jupyter_output><empty_output><jupyter_text>We can see there are 239 entries that are duplicates (the ResponseName has more than one appearance). We might want to keep the first appearance of the game. But if we go through the duplicates we'll find out that the first appearance is sometimes invalid. We'll have to use a bit more sophisticated duplication deletion. If we examine the duplicates, we can see that some of them do have missing values in attributes 'SteamSpyOwners', 'SteamSpyOwnersVariance', 'SteamSpyPlayersEstimate' and 'SteamSpyPlayersVariance' or the values are significantly lower than for the other duplicates of the same game. Based on this observation, we can decide which entry do we leave in the dataset.<jupyter_code># delete invalid duplicates from the dataset
df = delete_duplicates(df) # function from our helper functions
df.shape
# count duplicates again
print('There are now {} duplicates.'.format(len(df[df.duplicated(subset=['ResponseName'], keep=False)])))<jupyter_output>There are now 0 duplicates.<jupyter_text>Now that we dropped all duplicates, let's get rid of all entries that are not games. We can use 'GenreIsNonGame' column for this (as a mask for dropping entries). After that, we can also drop the column.<jupyter_code>print('There are {} non-game entreis in the dataset.'.format(len(df[df.GenreIsNonGame == True])))
df.drop(df[df.GenreIsNonGame == True].index, axis=0, inplace=True) # drop all non-Game entries
print('Now there are are {} non-game entreis in the dataset.'.format(len(df[df.GenreIsNonGame == True])))
df.drop(['GenreIsNonGame'], axis=1, inplace=True) # drop the column
df.shape # check how the shape of the dataset changed<jupyter_output>There are 318 non-game entreis in the dataset.
Now there are are 0 non-game entreis in the dataset.<jupyter_text>Apparently, there were some non-game entries that has 'GenreIsNonGame' == False. We will have to remove them manually, but we most likely won't be able to clear all the data.<jupyter_code># Manual cleaning of the data based on manual search on Steam
# videos or movies:
df.drop(df[df.ResponseName.str.contains('From Bedrooms to Billions')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('PAYDAY: The Web Series')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('The Making of Grim Fandango Remastered')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Mortal Kombat: Legacy')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Mortal Kombat: Legacy II')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Double Fine Adventure')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Naruto Shippuden Uncut')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Complete Figure Drawing Course')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Kalen Chock Presents')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Now You See Me')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Robotpencil Presents')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Raiders! : The Story of the Greatest Fan Film')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Man vs Snake: The Long and Twisted Tale')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('CS:GO Player Profiles')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Lets Play:')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Dota 2 Player Profiles')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Ghost in the Shell')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Meet the Blacks')].index, axis=0, inplace=True)
df.drop(df[df.ResponseName.str.contains('Kindred Spirits on the Roof Drama')].index, axis=0, inplace=True)

# game extensions with missing data
df.drop(df[(df.ResponseName.str.contains('Strong Bads Cool Game for Attractive People')) & (df.Metacritic == 0)].index, axis=0, inplace=True)
df.drop(df[(df.ResponseName.str.contains('Back to the Future')) & (df.Metacritic == 0)].index, axis=0, inplace=True)

# mods/servers/SW with missing values
df.drop(df[(df.ResponseName.str.contains('EuroTruck Simulator - Mod Tools'))].index, axis=0, inplace=True)
df.drop(df[(df.ResponseName.str.contains('IS Defense Editor'))].index, axis=0, inplace=True)
df.drop(df[(df.ResponseName.str.contains('Day of Infamy Dedicated Server'))].index, axis=0, inplace=True)

df.shape<jupyter_output><empty_output><jupyter_text>Now we can proceed with other attributes.We won need 'AchievementHighlightedCount' attribute. This attribute contains a number of highlighted achievements, which is a number from 0 - 10 depending on the number of achievements. If a game has 10 or more achievements, the 'AchievementHighlightedCount' is equal to 10. Otherwise, it is equal to 'AchievementCount'. We can drop this column<jupyter_code>df.drop(['AchievementHighlightedCount'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>We also wont use minimal and maximal recommendations for these games. Also a big portion of the dataset miss these values. We're dropping 'PCReqsHaveMin', 'PCReqsHaveRec', 'LinuxReqsHaveMin', 'LinuxReqsHaveRec', 'MacReqsHaveMin', 'MacReqsHaveRec', 'PCMinReqsText', 'PCRecReqsText', 'LinuxMinReqsText', 'LinuxRecReqsText', 'MacMinReqsText' and 'MacRecReqsText'.<jupyter_code>columns_to_drop = ['PCReqsHaveMin', 'PCReqsHaveRec', 'LinuxReqsHaveMin', 'LinuxReqsHaveRec', 'MacReqsHaveMin',
                   'MacReqsHaveRec', 'PCMinReqsText', 'PCRecReqsText', 'LinuxMinReqsText', 'LinuxRecReqsText',
                   'MacMinReqsText', 'MacRecReqsText']
df.drop(columns_to_drop, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>There are more columns we won't need. For example all the support contact information, legal notices and a website. So we can drop 'SupportEmail', 'SupportURL', 'DRMNotice', 'ExtUserAcctNotice', 'LegalNotice' and 'Website'.<jupyter_code>columns_to_drop = ['SupportEmail', 'SupportURL', 'DRMNotice', 'ExtUserAcctNotice', 'LegalNotice', 'Website']
df.drop(columns_to_drop, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Also, some descriptions aren't very helpful for our purposes. Though, it might be useful to have the information if certain games have or do not have these descriptions. The same goes with images (background and header). Let's have a look at the 'AboutText', 'ShortDescrip' and 'DetailedDescrip'. The 'ShortDescrip' is a short description of the game that can be found right under the header image on the Store Page of the game. Some games have it, some don't and sometimes the short description is a part of 'AboutText'. We will change this into a Boolean attribute with True value if the game has a short description and False if it has no short description.<jupyter_code>print('{} of our entries have no short description.'.format(len(df[df.ShortDescrip == ' '])))
df.replace({'ShortDescrip': {' ': ''}}, inplace=True) # replace with empty string in order to get False when retyped
df.ShortDescrip = df.ShortDescrip.astype(bool) # retype to Boolean
# check the result
print(df.ShortDescrip.dtype)
len(df[df.ShortDescrip == False])<jupyter_output>bool<jupyter_text>Now the 'AboutText' and 'DetailedDescrip'. 'AboutText' is a text on the game's Store Page with the title "ABOUT THIS GAME", so it makes sense for us to note if a game has this text available. On the other hand, 'DetailedDescrip' is nowhere to be found on the Store Page. If we look at the dataset, we can observe, that a big portion of the 'DetailedDescrip' is identical to 'AboutText' and others seem to be some Update descriptions or are simply missing. We will drop this column and retype 'AboutText' into boolean.<jupyter_code>p = len(df[df.AboutText == df.DetailedDescrip]) / len(df) * 100
print("{:0.2f}% of the 'DetailedDescrip' is identical to 'AboutText'.".format(p))

df.drop(['DetailedDescrip'], axis=1, inplace=True) # drop 'DetailedDescrip'<jupyter_output>90.49% of the 'DetailedDescrip' is identical to 'AboutText'.<jupyter_text>Now we only have to take care of the 'AboutText'. Let's have a look at it.<jupyter_code>print('{} of our entries have no "about the game" text.'.format(len(df[df.AboutText == ' '])))<jupyter_output>34 of our entries have no "about the game" text.<jupyter_text>Note that some of these entries are missing the 'AboutText' value because there is no text in the "ABOUT THIS GAME" section on Store Page of the game. If we search these games on steam, we'll find out that some of the games have an image version of the text (with other graphics) instead of text. Here are some exceptions we'll fix manually:<jupyter_code>df.loc[df[df.ResponseName == 'iREC'].index, 'AboutText'] = 'True'
df.loc[df[df.ResponseName == 'Crypt of the NecroDancer'].index, 'AboutText'] = 'True'
df.loc[df[df.ResponseName == 'Monochroma'].index, 'AboutText'] = 'True'
df.loc[df[df.ResponseName == 'No Turning Back: The Pixel Art Action-Adventure Roguelike'].index, 'AboutText'] = 'True'
df.loc[df[df.ResponseName == 'Crazy Pixel Streaker'].index, 'AboutText'] = 'True'
df.loc[df[df.ResponseName == 'One Thousand Lies'].index, 'AboutText'] = 'True'
df.loc[df[df.ResponseName == 'Azurea Juncture'].index, 'AboutText'] = 'True'

# NOTE: value 'True' can be any non-empty string at this point...

print('Now, {} of our entries have no "about the game" text.'.format(len(df[df.AboutText == ' '])))<jupyter_output>Now, 27 of our entries have no "about the game" text.<jupyter_text>We can now store only a boolean value if a game has "About the game" section or not.<jupyter_code>df.replace({'AboutText': {' ': ''}}, inplace=True) # replace with empty string in order to get False when retyped
df.AboutText = df.AboutText.astype(bool) # retype to Boolean<jupyter_output><empty_output><jupyter_text>We will change 'Background' and 'HeaderImage' in a similar fashion.<jupyter_code>print("{} of games don't have a background image and {} of games don't have a header image."\
      .format(len(df[df.Background==' ']),len(df[df.HeaderImage==' '])))<jupyter_output>73 of games don't have a background image and 0 of games don't have a header image.<jupyter_text>Apparently, there are no entries that are missing 'HeaderImage' what makes this attribute useless for our purposes and we can drop it.<jupyter_code>df.drop(['HeaderImage'], axis=1, inplace=True) # drop 'HeaderImage'

# retype 'Background'
df.replace({'Background': {' ': ''}}, inplace=True) # replace with empty string in order to get False when retyped
df.Background = df.Background.astype(bool) # retype to Boolean
columns = list(df.columns)
print("{} columns:\n{}".format(len(columns),columns))<jupyter_output>53 columns:
['ResponseName', 'ReleaseDate', 'RequiredAge', 'DemoCount', 'DeveloperCount', 'DLCCount', 'Metacritic', 'MovieCount', 'PackageCount', 'RecommendationCount', 'PublisherCount', 'ScreenshotCount', 'SteamSpyOwners', 'SteamSpyOwnersVariance', 'SteamSpyPlayersEstimate', 'SteamSpyPlayersVariance', 'AchievementCount', 'ControllerSupport', 'IsFree', 'FreeVerAvail', 'PurchaseAvail', 'SubscriptionAvail', 'PlatformWindows', 'PlatformLinux', 'PlatformMac', 'CategorySinglePlayer', 'CategoryMultiplayer', 'CategoryCoop', 'CategoryMMO', 'CategoryInAppPurchase', 'CategoryIncludeSrcSDK', 'CategoryIncludeLevelEditor', 'CategoryVRSupport', 'GenreIsIndie', 'GenreIsAction', 'GenreIsAdventure', 'GenreIsCasual', 'GenreIsStrategy', 'GenreIsRPG', 'GenreIsSimulation', 'GenreIsEarlyAccess', 'GenreIsFreeToPlay', 'GenreIsSports', 'GenreIsRacing', 'GenreIsMassivelyMultiplayer', 'PriceCurrency', 'PriceInitial', 'PriceFinal', 'AboutText', 'Background', 'ShortDescrip', 'Reviews', 'SupportedLanguages']<jupyter_text>Let's look at the 'Reviews' attribute. Apparently, a majority of the dataset miss this values. There might have been a problem with the extraction of the data during the creation of the dataset. Nevertheless, the text form of reviews wouldn't be very helpful for our purposes. We assume, that 'RecommendationCount' is the number of Reviews (or positive Reviews only?). Therefore, we'll drop this column.<jupyter_code>p = len(df[df.Reviews == ' ']) / len(df) * 100 
print("{:0.2f}% of the 'Reviews' values are empty.".format(p))

df.drop(['Reviews'], axis=1, inplace=True) # drop 'Reviews'<jupyter_output>73.60% of the 'Reviews' values are empty.<jupyter_text>Now for languages, this is a very messy column. There is also no use for us to keep this text format of the data.<jupyter_code>print("{} entries are missing the 'SupportedLanguages' data.".format(len(df[df.SupportedLanguages == ' '])))
print("There are {} unique values in 'SupportedLanguages' column.".format(len(df.SupportedLanguages.unique())))<jupyter_output>9 entries are missing the 'SupportedLanguages' data.
There are 2321 unique values in 'SupportedLanguages' column.<jupyter_text>We could be more interested in the number of supported languages and if a game supports English language or not. Let's parse this data and save our results in new columns.<jupyter_code># get all unique values, to construct an array of all supported languages and understand the data
supported_languages = list(df.SupportedLanguages.unique()) 
# print(supported_languages)

#function for the exploration of these values had to be destroyed for the greater good of programming community (sorry)<jupyter_output><empty_output><jupyter_text>**Values format:** Most of the data contains a list of languages separated with space. Some of these languages are suffixed with '*' and the list contains '*languages with full audio support' at the end. A few entries end with '(all with full audio support)' or '(text only)' instead of '*languages with full audio support'. Some entries use '[b]*[/b]' instead of simple '*'. Three entries are missing spaces between the languages and two entries contain semicolon after 'German' language.**Conclusion:** To determine if a game supports the English language, we can simply search if the value contains "English". If we want to count how many languages a game supports, we will search the value for all possible languages and count each occurrence. We will need a set of these languages for this.<jupyter_code># array of all languages from the data
languages = set(['English','French','German','Italian','Spanish','Russian','Korean','Danish','Dutch','Finnish',\
                 'Italian','Japanese','Norwegian','Polish','Portuguese','Swedish','Traditional Chinese',\
                 'Simplified Chinese','Czech','Hungarian','Bulgarian','Portuguese-Brazil','Ukrainian','Turkish','Greek',\
                 'Romanian','Thai','Slovakian','Arabic'])
# create a new column with Boolean value True: game supports Eng / False: game does not support Eng
df['SupportsEnglish'] = df.apply(lambda row: supports_english(row), axis=1)
print('There are {} games that do not support the English language at all.'.format(len(df[df.SupportsEnglish==False])))
# replace the column values with number of supported languages
df['SupportedLanguages'] = df.apply(lambda row: count_languages(row,languages), axis=1)
print("Now {} entries (missing values) should have a value 0.".format(len(df[df.SupportedLanguages == 0])))<jupyter_output>Now 9 entries (missing values) should have a value 0.<jupyter_text>Let's have a look at attributes 'IsFree', 'FreeVerAvail', 'PurchaseAvail', 'PriceCurrency', 'PriceInitial' and 'PriceFinal' to understand them better. We might not need all of these attributes.<jupyter_code>df.PriceCurrency.unique()<jupyter_output><empty_output><jupyter_text>Price currency is only in USD or has a missing value. This could be a sign, that all prices are in USD and we can drop this column. But let's investigate further.<jupyter_code>print("Unique values for entries with missing currency values:\n- 'PriceInitial': {}\n- 'PriceFinal': {}"\
      .format(df[df.PriceCurrency == ' '].PriceInitial.unique(),df[df.PriceCurrency == ' '].PriceFinal.unique()))<jupyter_output>Unique values for entries with missing currency values:
- 'PriceInitial': [0.]
- 'PriceFinal': [0.]<jupyter_text>Apparently, games with a missing value for 'PriceCurrency' attribute are all free. We can drop this column.<jupyter_code>df.drop(['PriceCurrency'], axis=1, inplace=True) # drop 'PriceCurrency' column<jupyter_output><empty_output><jupyter_text>The 'PriceInitial' and the 'PriceFinal' are our only numerical price features so we will keep them. However, we might use only one of them for our analysis and modeling, because the percentage of entries that have different Initial and final price is insignificant.<jupyter_code>drop = len(df[df.PriceInitial > df.PriceFinal]) # price drop
increase = len(df[df.PriceInitial < df.PriceFinal]) # price increase
free = len(df[(df.PriceInitial != 0) & (df.PriceFinal == 0)]) # made free

print("{} of games had a price drop, {} of game's prices increased and {} of games became free."\
     .format(drop,increase,free))
print("Only {:0.2f}% of games changed the price.".format((drop+increase)/len(df)*100))<jupyter_output>686 of games had a price drop, 0 of game's prices increased and 0 of games became free.
Only 5.60% of games changed the price.<jupyter_text>Our 'PurchaseAvail' should be pretty straightforward. This boolean attribute should be true if the price is bigger than $0.00.<jupyter_code>len(df[(df.PurchaseAvail  == True) & (df.PriceFinal == 0.0)]) # this sould be close to 0
# NOTE: we don't need to test this for 'PriceFinal' because we've already shown that no price dopped to $0.00<jupyter_output><empty_output><jupyter_text>The 'IsFree' attributes 'FreeVerAvail' shouldn't represent the same feature of the data, though it's not that clear what's the information they hold. Let's try to determine this.<jupyter_code>print('There are approximately(!!) {} free games in the dataset.'.format(len(df[df.PurchaseAvail == False])))
# NOTE: Remmember that also missing values are FALSE values - therefore it is only approximation!

print("'IsFree' == True : {}".format(len(df[df.IsFree == True])))
print("'FreeVerAvail' == True : {}\n".format(len(df[df.FreeVerAvail == True])))

print("'IsFree' == True && 'FreeVerAvail' == False: {}"\
      .format(len(df[(df.IsFree == True) & (df.FreeVerAvail  == False)])))
print("'IsFree' == False && 'FreeVerAvail' == True: {}\n"\
      .format(len(df[(df.IsFree == False) & (df.FreeVerAvail  == True)])))

print("'IsFree' == True && 'PriceInitial' > $0.00: {}"\
      .format(len(df[(df.IsFree == True) & (df.PriceInitial  > 0)])))
print("'FreeVerAvail' == True && 'PriceInitial' > $0.00: {}\n"\
      .format(len(df[(df.FreeVerAvail == True) & (df.PriceInitial  > 0)])))<jupyter_output>There are approximately(!!) 2055 free games in the dataset.
'IsFree' == True : 884
'FreeVerAvail' == True : 26

'IsFree' == True && 'FreeVerAvail' == False: 860
'IsFree' == False && 'FreeVerAvail' == True: 2

'IsFree' == True && 'PriceInitial' > $0.00: 42
'FreeVerAvail' == True && 'PriceInitial' > $0.00: 4<jupyter_text>The approximation of free games would be most likely closer to true if we would have deleted invalid entries previously. Though, from these observations (and after a manual search on steam), we can assume, that 'IsFree' attribute represents a Free game. The 'FreeVerAvail' attribute represents a game that has a free version along with a paid one (for example one of the episodes is free).Attribute 'FreeVerAvail' is useless for our purposes and we will drop it. Now we practically have two attributes with a Free/Paid information. We might want to drop one of these after we're done with the data cleaning. We'll decide which one is more accurate afterward. In the best-case scenario we'll get 'IsFree' == !'PurchaseAvail' and we'll be able to drop either of them.<jupyter_code>df.drop(['FreeVerAvail'], axis=1, inplace=True) # drop 'FreeVerAvail' column<jupyter_output><empty_output><jupyter_text>We will keep all other attributes. Let's rename the columns we are left with and finish the Column selection process. After all, we've changed the meaning of some columns.<jupyter_code>df.rename(columns={'ResponseName':'Name','MovieCount':'StoreVideoCount','PackageCount':'BundleCount'\
                  ,'ScreenshotCount':'StoreScreenshotCount','PurchaseAvail':'IsPaid'\
                  ,'SubscriptionAvail':'SubscriptionOption','SupportedLanguages':'LanguageCount'}, inplace=True)<jupyter_output><empty_output><jupyter_text>Here are all the attributes we now have in our attribute:<jupyter_code>columns = list(df.columns)
print("{} columns:\n{}".format(len(columns),columns))<jupyter_output>51 columns:
['Name', 'ReleaseDate', 'RequiredAge', 'DemoCount', 'DeveloperCount', 'DLCCount', 'Metacritic', 'StoreVideoCount', 'BundleCount', 'RecommendationCount', 'PublisherCount', 'StoreScreenshotCount', 'SteamSpyOwners', 'SteamSpyOwnersVariance', 'SteamSpyPlayersEstimate', 'SteamSpyPlayersVariance', 'AchievementCount', 'ControllerSupport', 'IsFree', 'IsPaid', 'SubscriptionOption', 'PlatformWindows', 'PlatformLinux', 'PlatformMac', 'CategorySinglePlayer', 'CategoryMultiplayer', 'CategoryCoop', 'CategoryMMO', 'CategoryInAppPurchase', 'CategoryIncludeSrcSDK', 'CategoryIncludeLevelEditor', 'CategoryVRSupport', 'GenreIsIndie', 'GenreIsAction', 'GenreIsAdventure', 'GenreIsCasual', 'GenreIsStrategy', 'GenreIsRPG', 'GenreIsSimulation', 'GenreIsEarlyAccess', 'GenreIsFreeToPlay', 'GenreIsSports', 'GenreIsRacing', 'GenreIsMassivelyMultiplayer', 'PriceInitial', 'PriceFinal', 'AboutText', 'Background', 'ShortDescrip', 'LanguageCount', 'SupportsEnglish']<jupyter_text>Missing/Invalid values handlingNow we can proceed with handling missing and invalid values. Let's go through all of our columns one by one and review the values.Column **Name** does not have any missing values. Let's have a look at **ReleaseDate**.<jupyter_code>df.ReleaseDate.head()<jupyter_output><empty_output><jupyter_text>We can see what our default Date format looks like. Of course, these are all strings and we would like to convert it into something more useful. But first, let's check if there are any invalid values and how do they look like, so as we can fix all the problems before our conversion.<jupyter_code># set the format string
format_str = "%b %d %Y"

# we will go through all the values and put the invalid ones into a set
invalid_values = find_invalid_dates(df,format_str)

# count entries with invalid values
cnt = 0
for i in invalid_values:
    cnt += len(df[df.ReleaseDate == i])

print('There are {} unique invalid values and {} entries containing invalid value (including {} missing values).'\
      .format(len(invalid_values),cnt,len(df[df.ReleaseDate == ' '])))<jupyter_output>There are 205 unique invalid values and 845 entries containing invalid value (including 74 missing values).<jupyter_text>**Few examples of invalid values:**    - Differen foramt: '07.11.2016', '12th December 2016', 'December 16 2016', '10/3/16'    - Missing parts: 'Dec 2015', '2017', 'September'    - Inaccurate: 'Summer 2017', 'Q2 2017', 'Mid-2017', 'Late Summer - 2016', 'Late 2016', 'Holiday 2017', 'Early October', 'Harvest 2016', 'Q4 (ish) 2016'    - Other: 'Before the Apocalypse', 'COMING SOON!', 'League of Evil is preparing its evil plans...', 'Not yet available', 'Someday', 'TBA', 'TBD', 'When its done', 'When you least expect it'    And many others.Of course, we should try to fix this values.<jupyter_code># we use our function from utilities to replace invalid dates with valid format if possible
df.ReleaseDate = df.ReleaseDate.map(lambda s: fix_invalid_date(s, format_str))<jupyter_output><empty_output><jupyter_text>We can review the invalid values once more in order to check our progress.<jupyter_code>invalid_values = find_invalid_dates(df, format_str)
# count entries with invalid values
cnt = 0
for i in invalid_values:
    cnt += len(df[df.ReleaseDate == i])

print('There are {} unique invalid values and {} entries containing invalid value (including {} missing values).'\
      .format(len(invalid_values),cnt,len(df[df.ReleaseDate == ' '])))<jupyter_output>There are 78 unique invalid values and 396 entries containing invalid value (including 74 missing values).<jupyter_text>We will drop all remaining invalid values.<jupyter_code>print('{:0.2f}% of the Release Dates are invalid or missing.'.format((len(df[df.ReleaseDate==' '])+cnt)/len(df)*100))

invalid_values.add(' ')
mask = []
mask = df.ReleaseDate.map(lambda s: True if s in invalid_values else False)

df.drop(df[mask].index, axis=0, inplace=True) # drop entries with invalid or missing ReleaseDate<jupyter_output>3.84% of the Release Dates are invalid or missing.<jupyter_text>Now, we can change the strings to datetime:<jupyter_code>df.ReleaseDate = df.ReleaseDate.map(lambda s: dt.datetime.strptime(s, format_str))<jupyter_output><empty_output><jupyter_text>This is how our Release Dates looks like now:<jupyter_code>df.ReleaseDate.head()<jupyter_output><empty_output><jupyter_text>In most of the other columns, we are not able to distinguish missing values that well. In boolean columns, the missing value is False, which is also a valid value. In columns with numerical values, the missing value is 0, which is also a valid value.For example: If 'Metacritic' == 0, the game has not yet been rated. If 'PriceInitial' == 0.00, the game is free. If 'DemoCount' == 0, the game has no demos available.But there are of course some columns for which this methodology doesn't make sense. How can 'DeveloperCount' or 'PublisherCount' be 0? Based on a quick manual search on Steam, not all games have their developer or publisher stated, therefore it is 0.There are also some 0 values in columns 'SteamSpyOwners', 'SteamSpyOwnersVariance' and 'SteamSpyPlayersEstimate', 'SteamSpyPlayersVariance' and there shouldn't be. Some of these games do not exist anymore, are not on SteamSpy or were not on SteamSpy when the dataset was created.For now, we will only check if there are not some invalid values (e.g. negative values) and in case some of the columns would be really relevant to our analysis, we'll make a sample of the data that based on this column.<jupyter_code># check datatypes
df.dtypes
# check unique values
#for col in columns:
#    print(df[col].unique())<jupyter_output><empty_output><jupyter_text>Everything looks fine and we can proceed with Data Exploration and visualization. This is how our data looks like now:<jupyter_code>print('The shape of the dataset is {}.'.format(df.shape)) # print out the shape of the dataset
df.head(10) # show first 10 records<jupyter_output>The shape of the dataset is (11853, 51).<jupyter_text>Data analysis and visualization In this section, we will review and visualize our data in order to understand the data better. Our data exploration and analysis will help us accept or reject some of our hypothesis.Let pandas describe the dataset for more insight.<jupyter_code>pd.options.display.float_format = '{:,.2f}'.format
df.describe()<jupyter_output><empty_output><jupyter_text>To make it as systematic as possible, we'll try to review the data column by column. Sometimes, we'll be also interested in relationship/correlation with other columns. Therefore more columns might be explored at once. Firstly, let's only review the values and distribution of the data and we'll leave the more complicated questions for later. ReleaseDateThis is our only time-related column in the whole dataset. It is also one of the most interesting and relevant piece of data we have. Some of our hypotheses are also related to this feature and we will get back to it later in order to answer some of our questions from the 'Hypothesis' section of this notebook. Let's find out when the games have been released.<jupyter_code>print('Our first game was released on {} (min value).'.format(df.ReleaseDate.min().strftime("%d. %B %Y")))
print('Our last game was released on {} (max value).'.format(df.ReleaseDate.max().strftime("%d. %B %Y")))
plt.hist(df['ReleaseDate'], bins=100)
plt.xlabel('Time')
plt.ylabel('Count')
plt.title('Time distribution')

plt.show()<jupyter_output><empty_output><jupyter_text>We can clearly see, that the number of games being released on Steam is rising exponentially. As the dataset was created at the end of 2016, the drop at the end of this histogram is because these games were only announced to be released at that time. Let's plot a similar histogram, that will show us exactly how many games were released each year. We can also ask questions as: Are some month more preferred for releasing games than others? Are games being released at the end of the week or rather at the beginning?<jupyter_code>years = df['ReleaseDate'].map(lambda t: t.year)

plt.hist(years, bins=15, range=(2005,2019))
plt.xlabel('Year')
plt.ylabel('Count')
plt.title('Year distribution')

plt.show()
months = df['ReleaseDate'].map(lambda t: t.month)

plt.hist(months, bins=12)
plt.xlabel('Month of the year')
plt.ylabel('Count')
plt.title('Month distribution')

plt.show()
# 0: Monday, 6: Sunday
week_days = df['ReleaseDate'].map(lambda t: t.weekday())

plt.hist(week_days, bins=7)
plt.xlabel('Day of the week')
plt.ylabel('Count')
plt.title('Week day distribution')

plt.show()<jupyter_output><empty_output><jupyter_text>We can see, that the least games were released in January and December. Most of the games are being released during the July-November period. Approximately twice as many games are being released in November in comparison to January.The more significant differences are visible in the histogram for days in a week. Bare minimum of games was released during the weekends. The reason for this might be, that the release process is closely overwatched by the development team, therefore the releases are happening during the working days. We can also see a slightly higher release rate at the end of the week (Thursday, Friday).  RequiredAgeThis feature won't be that relevant for us. A great portion of games on steam with mature content does not have an age restriction set. This is especially true for older games. Therefore we can expect a lot of our entries to have 'RequiredAge' == 0.<jupyter_code>print('{:0.2f}% of our games have no age restriction.'.format(len(df[df.RequiredAge==0])/len(df)*100))
df.RequiredAge.describe()<jupyter_output>94.84% of our games have no age restriction.<jupyter_text>Not surprisingly, the minimum value is 0 and the maximum is 18. Though, this description is not helpful at all since almost 95% of our data has no age restriction. Let's create a subset of games that do have this value and review it.<jupyter_code>age_sample = df[df.RequiredAge > 0]
print('Our sample of games with age restriction has {} entries'.format(len(age_sample)))
print('The mean of our sample is {:0.2f} with a standard deviation of {:0.2f}.'\
      .format(age_sample.RequiredAge.mean(),age_sample.RequiredAge.std()))
print('We can clearly see, that the majority of games with age restriction have age restricted to 17 years.')
age_sample.RequiredAge.describe()
plt.hist(age_sample.RequiredAge,bins=18)
plt.xlabel('Age')
plt.ylabel('Count')
plt.title('Age restrictions distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>DemoCountOnce again, it is not very common for games to have a demo. We can expect that a big percentage of games in our dataset does not have a demo.<jupyter_code>print('{:0.2f}% of our games have no demos available.'.format(len(df[df.DemoCount==0])/len(df)*100))
df.DemoCount.describe()<jupyter_output>90.64% of our games have no demos available.<jupyter_text>We can also create a sample of games that do have at least one demo and we'll found out that the majority of games with demos have only one demo.<jupyter_code>demo_sample = df[df.DemoCount > 0]
print('Our sample of games with demos has {} entries'.format(len(demo_sample)))
demo_sample.DemoCount.describe()<jupyter_output>Our sample of games with demos has 1110 entries<jupyter_text>DeveloperCount<jupyter_code>print('{:0.2f}% of our games have no developers stated.'.format(len(df[df.DeveloperCount==0])/len(df)*100))
print('Unique values: {}'.format(set(df.DeveloperCount.unique())))
df.DeveloperCount.describe()
print('As we would probably expect, the mean is {:0.2f} with a standard deviation of {:0.2f}.'\
      .format(df.DeveloperCount.mean(),df.DeveloperCount.std()))<jupyter_output>As we would probably expect, the mean is 1.08 with a standard deviation of 0.39.<jupyter_text>Let's look at the distribution of outliers, which shouldn't be very surprising.<jupyter_code>plt.hist(df[df.DeveloperCount > 1].DeveloperCount,bins=11)
plt.xlabel('Number of developers')
plt.ylabel('Count')
plt.title('Developer count outliers distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>DLCCountThis feature will be used to accept or reject our Hypothesis 1. Therefore, we'll closely analyze it later. For now, let's just overview it. Some games are really packed with DLCs. We can expect some games to have an enormous number of DLCs. However, most of the game will have no DLCs at all.<jupyter_code>print('{:0.2f}% of our games have no DLCs.'.format(len(df[df.DLCCount==0])/len(df)*100))
df.DLCCount.describe()<jupyter_output>82.81% of our games have no DLCs.<jupyter_text>We can clearly see, that the majority of the games have no DLCs. Let's create a sample of games that have at least one DLC and look at the distribution of the data in that sample. After all, we'll need this sample later.<jupyter_code>dlc_sample = df[df.DLCCount > 0]
print('Our sample of games with DLCs has {} entries'.format(len(dlc_sample)))
dlc_sample.DLCCount.describe()
plt.hist(dlc_sample.DLCCount,bins=11,range=(0,10)) # withouth outliers 
plt.xlabel('Number of DLCs')
plt.ylabel('Count')
plt.title('DLC distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>Again, it should be no surprise that most of the games with DLCs have one or two DLCs.  MetacriticAnother one of the most important columns in the whole dataset. This feature represents the rating on metacritic.com. Not all of the games are yet rated (and a lot of them won't ever be).<jupyter_code>print('{:0.2f}% of our games have not been rated yet.'.format(len(df[df.Metacritic==0])/len(df)*100))
df.Metacritic.describe()<jupyter_output>81.59% of our games have not been rated yet.<jupyter_text>The majority of games has not been rated and it will trouble us later when we'll analyze the ratings in time.<jupyter_code>rating_sample = df[df.Metacritic> 0]
print('Our sample of games that have been rated has {} entries'.format(len(rating_sample)))
rating_sample.Metacritic.describe()
plt.hist(rating_sample.Metacritic,bins=75)
plt.xlabel('Rating')
plt.ylabel('Count')
plt.title('Metacritic rating distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>StoreVideoCount<jupyter_code>print('{:0.2f}% of our games have no Video on Store Page.'.format(len(df[df.StoreVideoCount==0])/len(df)*100))
df.StoreVideoCount.describe()
plt.hist(df.StoreVideoCount, bins=20)
plt.xlabel('Number of videos')
plt.ylabel('Count')
plt.title('Video distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>Most of the games represent themselves with a trailer or a gameplay video on the Store Page. Most of the games have 1 video on the Store Page.  BundleCount<jupyter_code>print('{:0.2f}% of our games have no appearance in any bundle.'.format(len(df[df.BundleCount==0])/len(df)*100))
df.BundleCount.describe()
plt.hist(df.BundleCount, bins=18)
plt.xlabel('Number of bundles')
plt.ylabel('Count')
plt.title('Bundle appearance distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>Majority of the games are included in one bundle.  RecommendationCount<jupyter_code>print('{:0.2f}% of our games have no recommendations from users.'.format(len(df[df.RecommendationCount==0])/len(df)*100))
df.RecommendationCount.describe()<jupyter_output>60.19% of our games have no recommendations from users.<jupyter_text>The range of values is very big but the majority of data is on the lower side of this range.<jupyter_code>plt.hist(df.RecommendationCount,bins=500,range=(1,10000))
plt.xlabel('Number of recommendations')
plt.ylabel('Count')
plt.title('Recommendation distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>PublisherCountThis column should be similar to 'DeveloperCount', though we can expect the range of unique values to be smaller.<jupyter_code>print('{:0.2f}% of our games have no publisher stated.'.format(len(df[df.PublisherCount==0])/len(df)*100))
print('Unique values: {}'.format(set(df.PublisherCount.unique())))
df.PublisherCount.describe()
plt.hist(df.PublisherCount, bins=5)
plt.xlabel('Number of publishers')
plt.ylabel('Count')
plt.title('Publisher count distribution')
plt.show()
print('{:0.2f}% of our games have only one publisher.'.format(len(df[df.PublisherCount==1])/len(df)*100))<jupyter_output>94.40% of our games have only one publisher.<jupyter_text>StoreScreenshotCountThis feature is similar to 'StoreVideoCount'. However, we expect most of the games that had no video on the Store Page to have at least one Screenshot. Moreover, it is common for games to have more than one screenshot.<jupyter_code>print('{:0.2f}% of our games have no Screenshots on Store Page.'.format(len(df[df.StoreScreenshotCount==0])/len(df)*100))
df.StoreScreenshotCount.describe()<jupyter_output>0.57% of our games have no Screenshots on Store Page.<jupyter_text>Only a fraction of the games have no Screenshot. Let's have a look at the data distribution.<jupyter_code>plt.hist(df.StoreScreenshotCount,bins=50, range=(0,50))
plt.xlabel('Number of Screenshots')
plt.ylabel('Count')
plt.title('Screenshot distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>SteamSpyOwners and SteamSpyOwnersVarianceThese two attributes are of course strictly correlative. However, some of our data does not have this value as we have mentioned earlier.<jupyter_code>print('{:0.2f}% of our games are missing SteamSpyOwners feature.'.format(len(df[df.SteamSpyOwners==0])/len(df)*100))
print('{:0.2f}% of our games are missing SteamSpyOwnersVariance feature.'\
      .format(len(df[df.SteamSpyOwnersVariance==0])/len(df)*100))
df.SteamSpyOwners.describe()
df.SteamSpyOwnersVariance.describe()<jupyter_output><empty_output><jupyter_text>There are a lot of small games in our dataset and we expect the distribution to be tilted towards the lower values. Let's have look at the correlation between 'SteamSpyOwners' and 'SteamSpyOwnersVariance'.<jupyter_code># df.plot.scatter('SteamSpyOwners','SteamSpyOwnersVariance');
sns.scatterplot(x='SteamSpyOwners', y='SteamSpyOwnersVariance', data=df)<jupyter_output><empty_output><jupyter_text>SteamSpyPlayersEstimate and SteamSpyPlayersVarianceSimilarly to the previous two attributes, these two are strictly correlative.<jupyter_code>print('{:0.2f}% of our games are missing SteamSpyOwners feature.'.format(len(df[df.SteamSpyPlayersEstimate==0])/len(df)*100))
print('{:0.2f}% of our games are missing SteamSpyOwnersVariance feature.'\
      .format(len(df[df.SteamSpyPlayersVariance==0])/len(df)*100))
df.SteamSpyPlayersEstimate.describe()
df.SteamSpyPlayersVariance.describe()<jupyter_output><empty_output><jupyter_text>AchievementCount<jupyter_code>print('{:0.2f}% of our games have no Achievements.'.format(len(df[df.AchievementCount==0])/len(df)*100))
df.AchievementCount.describe()<jupyter_output>51.17% of our games have no Achievements.<jupyter_text>Almost a half of the games in our dataset have no achievements. Let's also examine the sample of games which have achievements separately.<jupyter_code>achievement_sample = df[df.AchievementCount> 0]
print('Our sample of games that have achievements has {} entries'.format(len(achievement_sample)))
achievement_sample.AchievementCount.describe()
plt.hist(achievement_sample.AchievementCount,bins=150,range=(0,300))
plt.xlabel('Number of achievements')
plt.ylabel('Count')
plt.title('Achievements distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>Now, we would explore some of our boolean attributes. However, we can group some of these in relevant groups. Let's finish our numerical attributes first before we get into the boolean values. PriceInitial and PriceFinalHere our two price features.<jupyter_code>print('{:0.2f}% of our games have PriceInitial == 0.'.format(len(df[df.PriceInitial==0])/len(df)*100))
print('{:0.2f}% of our games have PriceFinal == 0.'.format(len(df[df.PriceFinal==0])/len(df)*100))
df.PriceInitial.describe()
df.PriceFinal.describe()<jupyter_output><empty_output><jupyter_text>Let's have a look at the price distribution throughout our data.<jupyter_code>plt.hist(df.PriceInitial, bins=60, range=(0,60))
plt.xlabel('Price')
plt.ylabel('Count')
plt.title('Price distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>Note the trend of pricing games. The peeks are at prices $X4.99 and $X9.99. We can compare the distributions of Initial and Final praces.<jupyter_code>plt.hist([df.PriceInitial, df.PriceFinal], bins=30, range=(0,30), label=['PriceInitial', 'PriceFinal'])
plt.legend(loc='upper right')
plt.xlabel('Price')
plt.ylabel('Count')
plt.title('Price comparison distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>We can also have a look at the price drop distribution among the games that dropped their prices.<jupyter_code>changed_price = df.apply(lambda row: row.PriceInitial - row.PriceFinal, axis=1)
plt.hist(changed_price,bins=50, range=(1,45))
plt.xlabel('Price Drop')
plt.ylabel('Count')
plt.title('Price drop distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>LanguageCountThis is a column that we have modified previously.<jupyter_code>print('{:0.2f}% of our games are missing the supported languages attribute.'\
      .format(len(df[df.LanguageCount==0])/len(df)*100))
df.LanguageCount.describe()
print('Most of the games support {} language(s).'.format(df.LanguageCount.mode()[0]))
plt.hist(df.LanguageCount, bins=28)
plt.xlabel('Number of supported languages')
plt.ylabel('Count')
plt.title('Language support distribution')
plt.show()<jupyter_output><empty_output><jupyter_text>CONTINUE WITH THIS OVERVIEW OF ATTRIBUTES **[TODO]**  SupportsEnglishThis is a column that we have created. We can expect that a majority of games do support English language.<jupyter_code>print('{:0.2f}% of our games don\'t support English language.'\
      .format(len(df[df.SupportsEnglish==0])/len(df)*100))
print('{} games don\'t support English language and {} do.'\
      .format(len(df[df.SupportsEnglish==0]),len(df[df.SupportsEnglish==1])))<jupyter_output>0.39% of our games don't support English language.
46 games don't support English language and 11807 do.<jupyter_text>Correlation Matrix of numerical attributesLet's have a look at the correlation between all our numerical attributes.<jupyter_code>numerical_columns = ['RequiredAge', 'DemoCount', 'DeveloperCount', 'DLCCount', 'Metacritic', 'StoreVideoCount',\
                     'BundleCount', 'RecommendationCount', 'PublisherCount', 'StoreScreenshotCount',\
                     'SteamSpyOwners', 'SteamSpyOwnersVariance', 'SteamSpyPlayersEstimate',\
                     'SteamSpyPlayersVariance', 'AchievementCount', 'PriceInitial', 'PriceFinal','LanguageCount']
cm = sns.light_palette("green", as_cmap=True)
corr = df[numerical_columns].corr()
corr.style.background_gradient(cmap=cm)<jupyter_output><empty_output><jupyter_text>As we have mentioned earlier, there is a strong correlatioan between in the goups ['SteamSpyOwners', 'SteamSpyOwnersVariance', 'SteamSpyPlayersEstimate', 'SteamSpyPlayersVariance'] and ['PriceInitial', 'PriceFinal']. Other than that, there is a very little correlation in the data. **[TODO]** More interesting visualizations  Number of DLCs for Free/Paid gamesWe have hypothesized that free games do have on average more DLCs than the paid games. Let's create two samples from our data. One sample will contain all of the free games and the second one all of the paid games.<jupyter_code>free_games = df[df.IsFree == True]
paid_games = df[df.IsFree == False]
print('There are {} free games and {} paid games in our dataset.'.format(len(free_games),len(paid_games)))<jupyter_output>There are 854 free games and 10999 paid games in our dataset.<jupyter_text>If we describe the sampled data, we'll be able to see the mean and standard deviation.<jupyter_code>free_games.DLCCount.describe()
paid_games.DLCCount.describe()<jupyter_output><empty_output><jupyter_text>Apparently, the average number of DLCs for free games is higher than for paid games. However, there are a majority of games that have no DLCs at all and they are lowering the averages. Let's make the same comparison for games that have at least one DLC.<jupyter_code>print('There are {} free games and {} paid games that have some DLCs in our dataset.'\
      .format(len(free_games[free_games.DLCCount>0]),len(paid_games[paid_games.DLCCount>0])))
free_games[free_games.DLCCount>0].DLCCount.describe()
paid_games[paid_games.DLCCount>0].DLCCount.describe()<jupyter_output><empty_output><jupyter_text>Even though, there are some outliers in the paid games sample (e.g. the game that has 630 DLCs), our data supports our hypothesis. **[TODO]** 'CategoryInAppPurchase' more in paid or free games?   VR support growthAs we've observed previously, the number of games being released every year is rising exponentially. We can expect games with VR support to follow this trend.<jupyter_code>vr_games = df[df.CategoryVRSupport == True]
print('There are {} games with VR support in our dataset'.format(len(vr_games)))
plt.hist(vr_games['ReleaseDate'], bins=100)
plt.xlabel('Time')
plt.ylabel('Count')
plt.title('VR games time distribution')

plt.show()<jupyter_output><empty_output><jupyter_text>As we can see, there is a slight rise in the numbers of VR games being released. Though it is not as extreme as with all the games. We have an interesting peak somewhere around April 2016.<jupyter_code>vr_peak = vr_games[(vr_games.ReleaseDate > dt.datetime(2016,3,25)) \
                  & (vr_games.ReleaseDate < dt.datetime(2016,5,1))]
plt.hist(vr_peak['ReleaseDate'], bins=30)
plt.title('VR games 2016 peak')
plt.xlabel('Time')
plt.ylabel('Count')
plt.show()
# vr_peak.sort_values('ReleaseDate')
print("{} games released on April 4th, 2016.".format(len(vr_peak[vr_peak.ReleaseDate == dt.datetime(2016,4,4)])))
print("{} games released on April 5th, 2016.".format(len(vr_peak[vr_peak.ReleaseDate == dt.datetime(2016,4,5)])))<jupyter_output>14 games released on April 4th, 2016.
33 games released on April 5th, 2016.<jupyter_text>This peak was most likely fueld by the release of HTC Vive (VR headset) on April 5th, 2016.  Data modeling and prediction From the data we have, one logical thing to predict is the number of owners. In this section, we will scale our features and train a neural network for predicting the target variable, in our case 'SteamSpyOwners'. We can't use the 'SteamSpyOwnersVariance' feature due to its correlation with 'SteamSpyOwners'. We also shouldn't use 'SteamSpyPlayersEstimate' and 'SteamSpyPlayersVariance' for the same reason.  ScalingTo get the best results, we'll need to scale all of our atributes. Fortunately, we have only numerical and boolean values in our data. For our numerical values, we'll use a Min-Max scaler.<jupyter_code># Encode True/False values into 1/0
# from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()

bool_columns = ['ControllerSupport', 'IsFree', 'IsPaid', 'SubscriptionOption', 'PlatformWindows', 'PlatformLinux',\
                'PlatformMac', 'CategorySinglePlayer', 'CategoryMultiplayer', 'CategoryCoop', 'CategoryMMO',\
                'CategoryInAppPurchase', 'CategoryIncludeSrcSDK', 'CategoryIncludeLevelEditor', 'CategoryVRSupport',\
                'GenreIsIndie', 'GenreIsAction', 'GenreIsAdventure', 'GenreIsCasual', 'GenreIsStrategy',\
                'GenreIsRPG', 'GenreIsSimulation', 'GenreIsEarlyAccess', 'GenreIsFreeToPlay', 'GenreIsSports',\
                'GenreIsRacing', 'GenreIsMassivelyMultiplayer', 'AboutText', 'Background', 'ShortDescrip',\
                'SupportsEnglish']

for col in bool_columns:
    df[col] = le.fit_transform(df[col])
    
df.head(10)
# use a numeric representation of dates
t_0 = dt.datetime(1990,1,1)
df['ReleaseDate'] = df.ReleaseDate.map(lambda x: (x-t_0).days)
# scale numeric values with min/max scaler (scales to range 0-1)
# from sklearn.preprocessing import MinMaxScaler
minmax = MinMaxScaler()
num_colums = ['ReleaseDate', 'RequiredAge', 'DemoCount', 'DeveloperCount', 'DLCCount', 'Metacritic',\
              'StoreVideoCount', 'BundleCount', 'RecommendationCount', 'PublisherCount', 'StoreScreenshotCount',\
#               'SteamSpyOwners', 'SteamSpyOwnersVariance', 'SteamSpyPlayersEstimate', 'SteamSpyPlayersVariance',\
              'AchievementCount', 'PriceInitial', 'PriceFinal', 'LanguageCount']

df[num_colums] = minmax.fit_transform(df[num_colums].astype('float64'))
df.head()<jupyter_output><empty_output><jupyter_text>Training a modelWhen the data is properly scaled and ready, we can create split the data for training and testing. Let's train our model on 70% of the data and test it on 30%.<jupyter_code># create targets and inputs
X = df.copy()
# y = df[['SteamSpyOwners','SteamSpyOwnersVariance','SteamSpyPlayersEstimate','SteamSpyPlayersVariance']]
y = df.SteamSpyOwners
X.drop(['Name','SteamSpyOwners','SteamSpyOwnersVariance','SteamSpyPlayersEstimate', \
        'SteamSpyPlayersVariance'], axis=1, inplace=True)
# split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
print (X_train.shape, X_test.shape, y_train.shape, y_test.shape)<jupyter_output>(8297, 46) (3556, 46) (8297,) (3556,)<jupyter_text>We can now build and train our neural network.<jupyter_code># training networks with various parameters
hid_layer_cnts = [16,32,64,128,256]
epochs = [16,32,64,128,200,256,512,1024]
batches = [16,32,64,128]
def train_one_layer_models(hid_layer_cnts,epochs,batches):
    '''
    Function takes arrays of parameters for a one layer neural network and trains a neural network for every
    combination of these parameters. Function creates a table (data frame) with RMSE on testing data 
    for every neural network.
    '''
    table_cols = ['# of units per layer', '# of epochs', 'Size of batches', 'RMSE']
    table = pd.DataFrame(columns = table_cols)
    
    # model all combinations of parameters
    for hid in hid_layer_cnts:
        for e in epochs:
            for b in batches:
                # create model
                model = keras.Sequential([
                    keras.layers.Dense(128, input_dim=46, activation='relu'),
                    keras.layers.Dropout(0.4),
                    keras.layers.Dense(hid, activation='relu'),
                    keras.layers.Dense(1, activation='relu')
                ])

                # Compile model
                model.compile(loss='mean_squared_error', optimizer='adam')

                # Fit the model
                model.fit(X_train, y_train, epochs=e, batch_size=b, verbose=0, validation_split=0.1)

                # predict targets for testing data
                y_pred = model.predict(X_test)

                # compute RMSE
                rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))

                run = pd.Series( \
                        {'# of units per layer':hid,'# of epochs':e,'Size of batches':b,'RMSE':rmse} \
                      )
                table = table.append(run, ignore_index=True)
    return table
# this will take a lot of time

# start = dt.datetime.now()
# table = train_one_layer_models(hid_layer_cnts,epochs,batches)
# end = dt.datetime.now()

# duration = end - start
duration = end - start
print("{} neural networks trained and it took {}.".format(len(table),pprint_duration(duration)))
table<jupyter_output>160 neural networks trained and it took 10h 20m 52s.<jupyter_text>Let's now pick four of these (inclooding the best two) for one more training and their comparison.<jupyter_code>table_cols = ['# of units per layer', '# of epochs', 'Size of batches', 'Training time', 'RMSE']
table2 = pd.DataFrame(columns = table_cols)

param_sets = [[256,1024,16],[16,200,16],[16,128,16],[16,32,64]]
# create and train a model for each parameter set
for ps in param_sets:
    # create model
    model = keras.Sequential([
        keras.layers.Dense(128, input_dim=46, activation='relu'),
        keras.layers.Dropout(0.4),
        keras.layers.Dense(ps[0], activation='relu'),
        keras.layers.Dense(1, activation='relu')
    ])

    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')

    # Fit the model
    train_start = dt.datetime.now()
    model.fit(X_train, y_train, epochs=ps[1], batch_size=ps[2], verbose=0, validation_split=0.1)
    train_end = dt.datetime.now()
    train_duration = train_end - train_start
    
    # predict targets for testing data
    y_pred = model.predict(X_test)

    # compute RMSE
    rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))

    run = pd.Series( \
            {'# of units per layer':ps[0],'# of epochs':ps[1],'Size of batches':ps[2], \
             'Training time': pprint_duration(train_duration),'RMSE':rmse} \
          )
    table2 = table2.append(run, ignore_index=True)
# results:
table2<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11544658600352647, 'fraction_numerical': 0.01279868669058187, 'mean_word_length': 5.751744663382595, 'pattern_counts': {'":': 4, '<': 292, '<?xml version=': 0, '>': 308, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '80', 'dir': 'jupyter-structured-clean-dedup', 'id': '625762', 'license': 'MIT', 'n_tokens_mistral': 20683, 'n_tokens_neox': 19295, 'n_words': 6782, 'path': 'eda/SteamGamesEDA.ipynb', 'repo_name': 'Emejzon/SteamGamesEDA', 'size': '65811'}