<jupyter_start><jupyter_text>Spark Initializing<jupyter_code>import findspark
findspark.init()
import pyspark
import pandas as pd
import seaborn as sns # To get graphs
import matplotlib.pyplot as plt # To get graphs
#### First we have create a spark session
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[1]") \
                    .appName('Clustering') \
                    .getOrCreate()
spark<jupyter_output><empty_output><jupyter_text>Importing Data Set<jupyter_code>df = spark.read.option("header" , True).csv("/Users/Asus/Documents/InternCsv/combine_final.csv")
df.show()
from pyspark.sql.functions import percent_rank
from pyspark.sql import Window
df_dev = df.withColumn("rank", percent_rank().over(Window.partitionBy().orderBy("Date","Time","Plant")))
train_df = df_dev.where("rank <= .8").drop("rank")
train_df.show()
train_df = train_df.withColumn("Difference",train_df.Difference.cast('double'))
train_df = train_df.withColumn("Actual_Dispatch",train_df.Actual_Dispatch.cast('double'))
train_df = train_df.withColumn("Forecasted_Dispatch",train_df.Forecasted_Dispatch.cast('double'))
train_df.dtypes
from pyspark.sql.functions import col
sumtot = train_df.groupBy("Plant").sum("Actual_Dispatch","Forecasted_Dispatch","Difference").sort(col("Plant").asc())
sumtot.show(5)
from pyspark.sql.functions import *
newColumns = ["Plant","Actual_Dispatch","Forecasted_Dispatch","DispatchDiff"]
sumtot=sumtot.toDF(*newColumns)
df1 = sumtot.toPandas()
df1.head()
# df description

df1.describe()<jupyter_output><empty_output><jupyter_text>Clustering Data Set<jupyter_code># import required libraries for clustering
import sklearn
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from scipy.cluster.hierarchy import linkage
from scipy.cluster.hierarchy import dendrogram
from scipy.cluster.hierarchy import cut_tree<jupyter_output><empty_output><jupyter_text>Finding Outliers **There are 2 types of outliers and we will treat outliers as it can skew our dataset**StatisticalDomain specific<jupyter_code>attributes = ['Actual_Dispatch','Forecasted_Dispatch']
plt.rcParams['figure.figsize'] = [10,8]
sns.boxplot(data = df1[attributes], orient="v", palette="Set2" ,whis=1.5,saturation=1, width=0.7)
plt.title("Outliers Variable Distribution", fontsize = 14, fontweight = 'bold')
plt.ylabel("Range", fontweight = 'bold')
plt.xlabel("Attributes", fontweight = 'bold')
# Removing (statistical) outliers for Actual_Dispatch
Q1 = df1.Actual_Dispatch.quantile(0.05)
Q3 = df1.Actual_Dispatch.quantile(0.95)
IQR = Q3 - Q1
df1 = df1[(df1.Actual_Dispatch >= Q1 - 1.5*IQR) & (df1.Actual_Dispatch <= Q3 + 1.5*IQR)]

# Removing (statistical) outliers for 
Q1 = df1.Forecasted_Dispatch.quantile(0.05)
Q3 = df1.Forecasted_Dispatch.quantile(0.95)
IQR = Q3 - Q1
df1 = df1[(df1.Forecasted_Dispatch >= Q1 - 1.5*IQR) & (df1.Forecasted_Dispatch <= Q3 + 1.5*IQR)]<jupyter_output><empty_output><jupyter_text>Rescaling the Attributes**It is extremely important to rescale the variables so that they have a comparable scale.| There are two common ways of rescaling:**1) Min-Max scaling2) Standardisation (mean-0, sigma-1)Here, we will use Standardisation Scaling.<jupyter_code># Rescaling the attributes

rfm_df = df1[['Actual_Dispatch','Forecasted_Dispatch']]

# Instantiate
scaler = StandardScaler()

# fit_transform
rfm_df_scaled = scaler.fit_transform(rfm_df)
rfm_df_scaled.shape
rfm_df_scaled = pd.DataFrame(rfm_df_scaled)
rfm_df_scaled.columns = ['Actual_Dispatch','Forecasted_Dispatch']
rfm_df_scaled.head()<jupyter_output><empty_output><jupyter_text>**K-means clustering is one of the simplest and popular unsupervised machine learning algorithms.****The algorithm works as follows:**First we initialize k points, called means, randomly.We categorize each item to its closest mean and we update the mean’s coordinates, which are the averages of the items categorized in that mean so far.We repeat the process for a given number of iterations and at the end, we have our clusters.<jupyter_code># k-means with some arbitrary k

kmeans = KMeans(n_clusters=4, max_iter=50)
kmeans.fit(rfm_df_scaled)
kmeans.labels_<jupyter_output><empty_output><jupyter_text>**Finding the Optimal Number of Clusters****Elbow Curve to get the right number of Clusters**A fundamental step for any unsupervised algorithm is to determine the optimal number of clusters into which the data may be clustered. The Elbow Method is one of the most popular methods to determine this optimal value of k.<jupyter_code># Elbow-curve/SSD

ssd = []
range_n_clusters = [2, 3, 4, 5, 6, 7, 8]
for num_clusters in range_n_clusters:
    kmeans = KMeans(n_clusters=num_clusters, max_iter=50)
    kmeans.fit(rfm_df_scaled)
    
    ssd.append(kmeans.inertia_)
    
# plot the SSDs for each n_clusters
plt.plot(ssd)<jupyter_output><empty_output><jupyter_text>Silhouette Analysis**silhouette score=(p−q) / max(p,q)**p is the mean distance to the points in the nearest cluster that the data point is not a part ofq is the mean intra-cluster distance to all the points in its own cluster.1) The value of the silhouette score range lies between -1 to 1.2) A score closer to 1 indicates that the data point is very similar to other data points in the cluster,3) A score closer to -1 indicates that the data point is not similar to the data points in its cluster.<jupyter_code># Silhouette analysis
range_n_clusters = [2, 3, 4, 5, 6, 7, 8]

for num_clusters in range_n_clusters:
    
    # intialise kmeans
    kmeans = KMeans(n_clusters=num_clusters, max_iter=50)
    kmeans.fit(rfm_df_scaled)
    
    cluster_labels = kmeans.labels_
    
    # silhouette score
    silhouette_avg = silhouette_score(rfm_df_scaled, cluster_labels)
    print("For n_clusters={0}, the silhouette score is {1}".format(num_clusters, silhouette_avg))<jupyter_output>For n_clusters=2, the silhouette score is 0.902772587775885
For n_clusters=3, the silhouette score is 0.6768552682465468
For n_clusters=4, the silhouette score is 0.6579666274159333
For n_clusters=5, the silhouette score is 0.5961496017511657
For n_clusters=6, the silhouette score is 0.5684812032008631
For n_clusters=7, the silhouette score is 0.6137418201011827
For n_clusters=8, the silhouette score is 0.6075209419084975<jupyter_text>**According to silhouette score & Elbow Curve Ideal Clusters for the Plants will be 4**<jupyter_code># Final model with k=4
kmeans = KMeans(n_clusters=4, max_iter=50)
kmeans.fit(rfm_df_scaled)
kmeans.labels_
# assign the label
df1['Cluster_Id'] = kmeans.labels_
df1.head()
# Box plot to visualize Cluster Id vs Plant

sns.boxplot(x='Cluster_Id', y='Plant', data=df1)
# Box plot to visualize Cluster Id vs Frequency

sns.boxplot(x='Cluster_Id', y='Actual_Dispatch', data=df1)<jupyter_output><empty_output><jupyter_text>Hierarchical Clustering**Hierarchical clustering** involves creating clusters that have a predetermined ordering from top to bottom. For example, all files and folders on the hard disk are organized in a hierarchy. There are two types of hierarchical clustering,1) Divisive2) Agglomerative.<jupyter_code># Single linkage: 

mergings = linkage(rfm_df_scaled, method="single", metric='euclidean')
dendrogram(mergings)
plt.show()<jupyter_output><empty_output><jupyter_text>**Complete Linkage**In complete linkage hierarchical clustering, the distance between two clusters is defined as the longest distance between two points in each cluster. For example, the distance between clusters “r” and “s” to the left is equal to the length of the arrow between their two furthest points.<jupyter_code># Complete linkage

mergings = linkage(rfm_df_scaled, method="complete", metric='euclidean')
dendrogram(mergings)
plt.show()<jupyter_output><empty_output><jupyter_text>**Average Linkage:**In average linkage hierarchical clustering, the distance between two clusters is defined as the average distance between each point in one cluster to every point in the other cluster. For example, the distance between clusters “r” and “s” to the left is equal to the average length each arrow between connecting the points of one cluster to the other<jupyter_code># Average linkage

mergings = linkage(rfm_df_scaled, method="average", metric='euclidean')
dendrogram(mergings)
plt.show()<jupyter_output><empty_output><jupyter_text>Cutting the Dendrogram based on K<jupyter_code># 3 clusters
cluster_labels = cut_tree(mergings, n_clusters=3).reshape(-1, )
cluster_labels
# Assign cluster labels

df1['Cluster_Labels'] = cluster_labels
df1.head()
# Plot Cluster Id vs Amount

sns.boxplot(x='Cluster_Labels', y='Plant', data=df1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09791498675267826, 'fraction_numerical': 0.027646584494873862, 'mean_word_length': 6.395229982964225, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '464785', 'license': 'Apache-2.0', 'n_tokens_mistral': 3076, 'n_tokens_neox': 2822, 'n_words': 814, 'path': 'Unsupervised Learning/Dispatch Data Clustering.ipynb', 'repo_name': 'Pandula1234/Data-Analysis-Intern', 'size': '8681'}