<jupyter_start><jupyter_text>Last week I scrapped a bunch of data from the Steam API using my [Steam Graph Project](https://github.com/jrtechs/SteamFriendsGraph).This project captures steam users, their friends, and the games that they own.Using the Janus-Graph traversal object, we are able to use the Gremlin graph query language to pull this data.Since I am storing the hours played in a game as a property on the relationship between a player and a game node, I had to do a join statement to get the hours property with the game information in a single query.   ```javaObject o = graph.con.getTraversal()    .V()    .hasLabel(Game.KEY_DB)    .match(            __.as("c").values(Game.KEY_STEAM_GAME_ID).as("gameID"),            __.as("c").values(Game.KEY_GAME_NAME).as("gameName"),            __.as("c").inE(Game.KEY_RELATIONSHIP).values(Game.KEY_PLAY_TIME).as("time")    ).select("gameID", "time", "gameName").toList();WrappedFileWriter.writeToFile(new Gson().toJson(o).toLowerCase(), "games.json");``` Using the game indexing property on the players, I noted that I only ended up fully indexing the games of 481 games.  ```javagraph.con.getTraversal()    .V()    .hasLabel(SteamGraph.KEY_PLAYER)    .has(SteamGraph.KEY_CRAWLED_GAME_STATUS, 1)    .count().next()``` We now transition to python and Matlptlib to visualize the data exported from our JanusGraph Query as a json object.The dependencies for this [notebook](https://github.com/jrtechs/RandomScripts/tree/master/notebooks) can be installed using pip.<jupyter_code>!pip install pandas
!pip install matplotlib<jupyter_output>Collecting pandas
  Downloading pandas-1.0.5-cp38-cp38-manylinux1_x86_64.whl (10.0 MB)
[K     |████████████████████████████████| 10.0 MB 4.3 MB/s eta 0:00:01
[?25hCollecting pytz>=2017.2
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
[K     |████████████████████████████████| 510 kB 2.9 MB/s eta 0:00:01
[?25hRequirement already satisfied: numpy>=1.13.3 in /home/jeff/Documents/python/ml/lib/python3.8/site-packages (from pandas) (1.18.5)
Requirement already satisfied: python-dateutil>=2.6.1 in /home/jeff/Documents/python/ml/lib/python3.8/site-packages (from pandas) (2.8.1)
Requirement already satisfied: six>=1.5 in /home/jeff/Documents/python/ml/lib/python3.8/site-packages (from python-dateutil>=2.6.1->pandas) (1.15.0)
Installing collected packages: pytz, pandas
Successfully installed pandas-1.0.5 pytz-2020.1<jupyter_text>The first step we are doing is importing our JSON data as a pandas dataframe.Pandas is a popular open-source data analysis and manipulation tool.I enjoy pandas because it has native integration with matplotlib and supports operations like aggregations and groupings.<jupyter_code>import matplotlib.pyplot as plt
import pandas as pd

games_df = pd.read_json('games.json')
games_df<jupyter_output><empty_output><jupyter_text>Using the built-in matplotlib wrapper function, we can graph a historgram of the number of hours played in a game.<jupyter_code>ax = games_df.hist(column='time', bins=20, range=(0, 4000))
ax=ax[0][0]
ax.set_title("Game Play Distribution")
ax.set_xlabel("Minutes Played")
ax.set_ylabel("Frequency")<jupyter_output><empty_output><jupyter_text>Notice that the vast majority of the games are rarely ever played, however, it is skewek to the right with a lot of outliers.We can change the scale to make it easier to view using the range parameter.<jupyter_code>ax = games_df.hist(column='time', bins=20, range=(0, 100))
ax=ax[0][0]
ax.set_title("Game Play Distribution")
ax.set_xlabel("Minutes Played")
ax.set_ylabel("Frequency")<jupyter_output><empty_output><jupyter_text>If we remove games that have never been played, the distribution looks more normal.<jupyter_code>ax = games_df.hist(column='time', bins=20, range=(2, 100))
ax=ax[0][0]
ax.set_title("Game Play Distribution")
ax.set_xlabel("Minutes Played")
ax.set_ylabel("Frequency")<jupyter_output><empty_output><jupyter_text>Although historgrams are useful, viewing the CDF is often more useful since it is easier to extract numerical information from.<jupyter_code>ax = games_df.hist(column='time',density=True, range=(0, 2000),  histtype='step',cumulative=True)
ax=ax[0][0]
ax.set_title("Game Play Distribution")
ax.set_xlabel("Minutes Played")
ax.set_ylabel("Frequency")<jupyter_output><empty_output><jupyter_text>80% of people who owns a game plays it under 4 hours. In fact, nearly half of all downloaded or purchased steam games go un-played. This is a cool example of famous 80/20 principle -- aka Pareto principle. The Pareto principle states that for most events, roughly 80% of the effects from 20% of the causes.As mentioned earlier, the time in owned game distribution is heavily skewed to the right.<jupyter_code>ax = plt.gca()
ax.set_title('Game Play Distribution')
ax.boxplot(games_df['time'], vert=False,manage_ticks=False, notch=True)
plt.xlabel("Game Play in Minutes")
ax.set_yticks([])
plt.show()<jupyter_output><empty_output><jupyter_text>Zooming in on the distribution we see that nearly half of all the purchased games go un-opened.<jupyter_code>ax = plt.gca()
ax.set_title('Game Play Distribution')
ax.boxplot(games_df['time']/60, vert=False,manage_ticks=False, notch=True)
plt.xlabel("Game Play in Hours")
ax.set_yticks([])
ax.set_xlim([0, 10])
plt.show()<jupyter_output><empty_output><jupyter_text>Viewing the aggregate pool of hours in particular game data is insightful, however, comparing different games against each other is more interesting.In pandas, after we create a grouping on a column, we can aggregate it into metrics such as max, min, mean, etc.I am also sorting the data I get by count since we are more interested in "popular" games.<jupyter_code>stats_df = (games_df.groupby("gamename")
                    .agg({'time': ['count', "min", 'max', 'mean']})
                    .sort_values(by=('time', 'count')))
stats_df<jupyter_output><empty_output><jupyter_text>To prevent one off essoteric games that I don't have a lot of data for, throwing off my metrics, I am disregarding any games that I have less than 10 values for.<jupyter_code>stats_df = stats_df[stats_df[('time', 'count')] > 10]
stats_df<jupyter_output><empty_output><jupyter_text>We see that games on average have a play time of 5-hours in a box-plot.<jupyter_code>ax = plt.gca()
ax.set_title('Game Play Distribution')
ax.boxplot(stats_df[('time', 'mean')]/60, vert=False,manage_ticks=False, notch=True)
plt.xlabel("Mean Game Play in Hours")
ax.set_xlim([0, 40])
ax.set_yticks([])
plt.show()<jupyter_output><empty_output><jupyter_text>I had a hunch that more popular games got played more, however, I feel like this data-set is still too small the verify this hunch.<jupyter_code>stats_df.plot.scatter(x=('time', 'count'), y=('time', 'mean'))
We can create a new filtered dataframe that only contains the result of a single game. 
cc_df = games_df[games_df['gamename'] == "counter-strike: global offensive"]
cc_df<jupyter_output><empty_output><jupyter_text>It is supprizing how many hours certain people play in Counter-Strike. The highest number in the dataset was 8,444 hours or 352 days!<jupyter_code>ax = plt.gca()
ax.set_title('Game Play Distribution for Counter-Strike')
ax.boxplot(cc_df['time']/60, vert=False,manage_ticks=False, notch=True)
plt.xlabel("Game Play in Hours")
ax.set_yticks([])
plt.show()<jupyter_output><empty_output><jupyter_text>Viewing the distribution for a different game like Unturned, yields a vastly different distribution than Counter-Strike. I believe the key difference is that Counter-Strike is played competitively where Unturned is a more leasurely game. It is likely that competative gamers skew the distribution of Counter-Strike to be very high.<jupyter_code>u_df = games_df[games_df['gamename'] == "unturned"]
u_df
ax = plt.gca()
ax.set_title('Game Play Distribution for Unturned')
ax.boxplot(u_df['time']/60, vert=False,manage_ticks=False, notch=True)
plt.xlabel("Game Play in Hours")
ax.set_yticks([])
plt.show()<jupyter_output><empty_output><jupyter_text>Next, I made a dataframe just containing the raw data points of games that had a aggregate count of over 80. For the sample size of the crawl that I did, having a count of 80, would make the game "popular". Since we only have 485 players indexed, having over 80 entries means that over 17% of people scrapped had the game. Glancing at the results, it is easy to verify that the games returned were very popular.<jupyter_code>df1 = games_df[games_df['gamename'].map(games_df['gamename'].value_counts()) > 80]
df1['time'] = df1['time']/60
df1
ax = plt.gca()
ax.set_title('Steps Distribution for July\n')
df1.boxplot(column=["time"], by='gamename',ax=ax, notch=True, vert=False)
plt.xlabel("Hours Played")
plt.ylabel("Games")
plt.show()
ax = df1.boxplot(column=["time"], by='gamename', notch=True, vert=False)
fig = ax.get_figure()
fig.suptitle('')
ax.set_title('Play-time Distribution')
plt.xlabel("Hours Played")
ax.set_xlim([0, 2000])
plt.ylabel("Game")
plt.savefig("playTimes.png", dpi=300, bbox_inches = "tight")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12422703180212014, 'fraction_numerical': 0.021311837455830387, 'mean_word_length': 5.876993166287016, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 68, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '513899', 'license': 'MIT', 'n_tokens_mistral': 3209, 'n_tokens_neox': 2954, 'n_words': 921, 'path': 'notebooks/steam/steam.ipynb', 'repo_name': 'sunnythepatel/Python-Scripts', 'size': '9056'}