<jupyter_start><jupyter_text>Machine Learning - Clustering<jupyter_code>from pyspark.sql import SparkSession

from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.clustering import KMeans
spark = SparkSession.builder.getOrCreate()
data_path = '../Data'
file_path = data_path + '/utilization.csv'
df = spark.read.format('csv').options(header=False, inferSchema=True).load(file_path)

# rename columns
df = df.withColumnRenamed('_c0', 'event_datetime')\
    .withColumnRenamed('_c1', 'server_id')\
    .withColumnRenamed('_c2', 'cpu_utilization')\
    .withColumnRenamed('_c3', 'free_memory')\
    .withColumnRenamed('_c4', 'session_count')
df.show(5)<jupyter_output>+-------------------+---------+---------------+-----------+-------------+
|     event_datetime|server_id|cpu_utilization|free_memory|session_count|
+-------------------+---------+---------------+-----------+-------------+
|03/05/2019 08:06:14|      100|           0.57|       0.51|           47|
|03/05/2019 08:11:14|      100|           0.47|       0.62|           43|
|03/05/2019 08:16:14|      100|           0.56|       0.57|           62|
|03/05/2019 08:21:14|      100|           0.57|       0.56|           50|
|03/05/2019 08:26:14|      100|           0.35|       0.46|           43|
+-------------------+---------+---------------+-----------+-------------+
only showing top 5 rows<jupyter_text>---------  Vectorizing Features, tranform using VectorAssembler- Spark ML algorithm expects the inputs in vector format.<jupyter_code>vector_assembler = VectorAssembler(inputCols=['cpu_utilization', 'free_memory', 'session_count'], outputCol='features')
# transform by passing the original dataframe
vcluster_df = vector_assembler.transform(df)
vcluster_df.show(5)<jupyter_output>+-------------------+---------+---------------+-----------+-------------+----------------+
|     event_datetime|server_id|cpu_utilization|free_memory|session_count|        features|
+-------------------+---------+---------------+-----------+-------------+----------------+
|03/05/2019 08:06:14|      100|           0.57|       0.51|           47|[0.57,0.51,47.0]|
|03/05/2019 08:11:14|      100|           0.47|       0.62|           43|[0.47,0.62,43.0]|
|03/05/2019 08:16:14|      100|           0.56|       0.57|           62|[0.56,0.57,62.0]|
|03/05/2019 08:21:14|      100|           0.57|       0.56|           50|[0.57,0.56,50.0]|
|03/05/2019 08:26:14|      100|           0.35|       0.46|           43|[0.35,0.46,43.0]|
+-------------------+---------+---------------+-----------+-------------+----------------+
only showing top 5 rows<jupyter_text>We can see that there is a new column 'features' in the newly created DataFrame.<jupyter_code># type(vcluster_df)<jupyter_output><empty_output><jupyter_text>Model Creation<jupyter_code>kmeans = KMeans().setK(3)   # set clusters as 3
kmeans = kmeans.setSeed(1)  # set random seed as 1
k_model = kmeans.fit(vcluster_df)  #fit the model with transformed df<jupyter_output><empty_output><jupyter_text>Get Cluster Centers<jupyter_code>k_model.clusterCenters()         # get the clusters center<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2697160883280757, 'fraction_numerical': 0.0971608832807571, 'mean_word_length': 1.8439461883408073, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '393155', 'license': 'MIT', 'n_tokens_mistral': 1212, 'n_tokens_neox': 1018, 'n_words': 184, 'path': 'Spark - Spark SQL and Data Frames/03.Data Analysis with Spark/05.03.Machine Learning - Clustering.ipynb', 'repo_name': 'ptyadana/probability-and-statistics-for-business-and-data-science', 'size': '3170'}