<jupyter_start><jupyter_text>Week 3 - Clustering with Spark MLlib Assignment: Recommending Actions from Clustering Analysis<jupyter_code>import pandas as pd
from pyspark.mllib.clustering import KMeans, KMeansModel
from numpy import array
pip install pyspark<jupyter_output><empty_output><jupyter_text>Step 1: Attribute Selection Import DataFirst let us read the contents of the file ad-clicks.csv. The following commands read in the CSV file in a table format and removes any extra whitespaces. So, if the CSV contained ' userid  ' it becomes 'userid'.<jupyter_code>from google.colab import files
def getLocalFiles():
    _files = files.upload()
    if len(_files) >0:
       for k,v in _files.items():
         open(k,'wb').write(v)
getLocalFiles()
import pandas as pd
adclicks_df = pd.read_csv('ad-clicks.csv')
adclicks_df = adclicks_df.rename(columns=lambda x: x.strip()) #remove whitespaces from headers<jupyter_output><empty_output><jupyter_text>Let us display the first 5 lines of adclicks_df:<jupyter_code>adclicks_df.head(5)<jupyter_output><empty_output><jupyter_text>Next, We are going to add an extra column to the ad-clicks table and make it equal to 1. We do so to record the fact that each ROW is 1 ad-click. You will see how this will become useful when we sum up this column to find how many ads did a user click.<jupyter_code>adclicks_df['adCount'] = 1<jupyter_output><empty_output><jupyter_text>Let us display the first 5 lines of adclicks_df and see if a new column has been added:<jupyter_code>adclicks_df.head(5)
from google.colab import files
def getLocalFiles():
    _files = files.upload()
    if len(_files) >0:
       for k,v in _files.items():
         open(k,'wb').write(v)
getLocalFiles()<jupyter_output><empty_output><jupyter_text>Then let us read the contents of the file game-clicks.csv. The following commands read in the CSV file in a table format and removes any extra whitespaces. So, if the CSV contained ' userid ' it becomes 'userid'.<jupyter_code>gameclicks_df = pd.read_csv('game-clicks.csv')
gameclicks_df = gameclicks_df.rename(columns=lambda x: x.strip()) #remove whitespaces from headers<jupyter_output><empty_output><jupyter_text>Let us display the first 5 lines of gameclicks_df:<jupyter_code>gameclicks_df.head(5)<jupyter_output><empty_output><jupyter_text>Next, We are going to add an extra column to the game-clicks table and make it equal to 1. We do so to record the fact that each ROW is 1 game-click. You will see how this will become useful when we sum up this column to find how many times did a user click.<jupyter_code>gameclicks_df['clickCount'] = 1<jupyter_output><empty_output><jupyter_text>Let us display the first 5 lines of gameclicks_df and see if a new column has been added:<jupyter_code>gameclicks_df.head(5)<jupyter_output><empty_output><jupyter_text>Next, let us read the contents of the file buy-clicks.csv. As before, the following commands read in the CSV file in a table format and removes any extra whitespaces. So, if the CSV contained ' userid  ' it becomes 'userid'.<jupyter_code>from google.colab import files
def getLocalFiles():
    _files = files.upload()
    if len(_files) >0:
       for k,v in _files.items():
         open(k,'wb').write(v)
getLocalFiles()
buyclicks_df = pd.read_csv('buy-clicks.csv')
buyclicks_df = buyclicks_df.rename(columns=lambda x: x.strip()) #removes whitespaces from headers<jupyter_output><empty_output><jupyter_text>Let us display the first 5 lines of buyclicks_df:<jupyter_code>buyclicks_df.head(5)<jupyter_output><empty_output><jupyter_text>Feature SelectionFor this exercise, we can choose from buyclicks_df,  the 'price' of each app that a user purchases as an attribute that captures user's purchasing behavior. The following command selects 'userid' and 'price' and drops all other columns that we do not want to use at this stage.<jupyter_code>user_purchases = buyclicks_df[['userId','price']] #select only userid and price
user_purchases.head(5)<jupyter_output><empty_output><jupyter_text>From the adclicks_df, we will use the 'adCount' as an attribute that captures user's inclination to click on ads. The following command selects 'userId' and 'adCount' and drops all other columns that we do not want to use at this stage.<jupyter_code>user_adclicks = adclicks_df[['userId','adCount']]
user_adclicks.head(5)<jupyter_output><empty_output><jupyter_text>Similarly, from the gameclicks_df,  we will use the 'clickCount' as an attribute that captures number of times of user's clicking on game. The following command selects 'userid' and 'clickCount' and drops all other columns that we do not want to use at this stage.<jupyter_code>user_gameclicks = gameclicks_df[['userId','clickCount']]
user_gameclicks.head(5) #as we saw before, this line displays first five lines<jupyter_output><empty_output><jupyter_text>Step 2: Training Data Set Creation Create the first aggregate feature for clusteringFrom each of these single ad-clicks per row, we can now generate total ad clicks per user. Let's pick a user with userid = 3. To find out how many ads this user has clicked overall, we have to find each row that contains userid = 3, and report the total number of such rows.The following commands sum the total number of ads per user and rename the columns to be called 'userId' and 'totalAdClicks'. In the end, we want to get one row per user, if we are performing clustering over users.<jupyter_code>ads_per_user = user_adclicks.groupby('userId').sum()
ads_per_user = ads_per_user.reset_index()
ads_per_user.columns = ['userId', 'totalAdClicks'] #rename the columns<jupyter_output><empty_output><jupyter_text>Let us display the first 5 lines of 'ads_per_user' to see if there is a column named 'totalAdClicks' containing total adclicks per user.<jupyter_code>ads_per_user.head(5)<jupyter_output><empty_output><jupyter_text>Create the second aggregate feature for clusteringThe following commands sum the total number of clicking on game per user and rename the columns to be called 'userId' and 'totalGameClicks'. In the end, we want to get one row per user, if we are performing clustering over users.<jupyter_code>game_clicks_per_user = user_gameclicks.groupby('userId').sum()
game_clicks_per_user = game_clicks_per_user.reset_index()
game_clicks_per_user.columns = ['userId', 'totalGameClicks'] #rename the columns<jupyter_output><empty_output><jupyter_text>Let us display the first 5 lines of 'game_clicks_per_user' to see if there is a column named 'totalGameClicks' containing total gameclicks per user.<jupyter_code>game_clicks_per_user.head(5)<jupyter_output><empty_output><jupyter_text>Create the third aggregate feature for clusteringSimilar to what we did for adclicks and gameclicks, here we find out how much money in total did each user spend on buying in-app purchases. As an example, let's pick a user with userId = 9. To find out the total money spent by this user, we have to find each row that contains userId = 9, and report the sum of the column'price' of each product they purchased.The following commands sum the total money spent by each user and rename the columns to be called 'userId' and 'revenue'.<jupyter_code>revenue_per_user = user_purchases.groupby('userId').sum()
revenue_per_user = revenue_per_user.reset_index()
revenue_per_user.columns = ['userId', 'revenue'] #rename the columns
revenue_per_user.head(5)<jupyter_output><empty_output><jupyter_text>Merge the three tablesLet's see what we have so far. We have a table called revenue_per_user, where each row contains total money a user (with that 'userId') has spent. We have a table called ads_per_user where each row contains total number of ads a user has clicked. We also have another table called game_clicks_per_user where each row contains total number of clicking of a user in the game. We will use revenue_per_user, ads_per_user and game_clicks_per_user as features / attributes to capture our users' behavior.Let us combine these three attributes (features) so that each row contains both attributes per user. Let's merge these three tables to get one single table we can use for K-Means clustering.<jupyter_code>combined_df = ads_per_user.merge(game_clicks_per_user, on='userId') #userId, adCount, clickCount
combined_df = combined_df.merge(revenue_per_user, on='userId') #userId, adCount, clickCount, price
combined_df.head(5) #display how the merged table looks<jupyter_output><empty_output><jupyter_text>Create the final training datasetOur training data set is almost ready. At this stage we can remove the 'userId' from each row, since 'userId' is a computer generated random number assigned to each user. It does not capture any behavioral aspect of a user. One way to drop the 'userId', is to select the other two columns.<jupyter_code>cluster_df = combined_df[['totalAdClicks', 'totalGameClicks','revenue']]
cluster_df.head(5)<jupyter_output><empty_output><jupyter_text>Display the dimensions of the training datasetDisplay the dimension of the training data set. To display the dimensions of the training_df, simply add `.shape` as a suffix and hit enter.<jupyter_code>cluster_df.shape<jupyter_output><empty_output><jupyter_text>The following two commands convert the tables we created into a format that can be understood by the `KMeans.train` function.line[0] refers to the first column. line[1] refers to the second column. If you have more than 2 columns in your training table, modify this command by adding line[2], line[3], line[4] ...<jupyter_code>pip install pyspark
from pyspark import SparkContext

sc =SparkContext.getOrCreate()

sqlContext = SQLContext(sc)
p_df = sqlContext.createDataFrame(cluster_df)
parsed_data = p_df.rdd.map(lambda line: array([line[0], line[1], line[2]])) #totalAdClicks, totalGameClicks, revenue<jupyter_output><empty_output><jupyter_text>Step 3: Train to Create Cluster Centers Train KMeans model Here we are creating two clusters as denoted in the second argument.<jupyter_code>import pandas as pd
from pyspark.mllib.clustering import KMeans, KMeansModel
from numpy import array
km_model = KMeans.train(parsed_data, 3, maxIterations=10, runs=10, initializationMode="random")<jupyter_output>/usr/local/lib/python3.6/dist-packages/pyspark/mllib/clustering.py:347: UserWarning: The param `runs` has no effect since Spark 2.0.0.
  warnings.warn("The param `runs` has no effect since Spark 2.0.0.")<jupyter_text>Display the centers of three clusters formed<jupyter_code>print(km_model.centers)<jupyter_output>[array([   32.05,  2393.95,    41.2 ]), array([  36.47486034,  953.82122905,   46.16201117]), array([  25.12037037,  362.50308642,   35.35802469])]
{'fraction_non_alphanumeric': 0.0788729739917075, 'fraction_numerical': 0.01290991330569167, 'mean_word_length': 5.986833443054641, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 0, '>': 102, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '456630', 'license': 'MIT', 'n_tokens_mistral': 3376, 'n_tokens_neox': 3261, 'n_words': 1219, 'path': 'pyspark_mllib_clustering.ipynb', 'repo_name': 'javedmatrah/A-to-Z-Resources-for-Students', 'size': '10612'}