<jupyter_start><jupyter_text>Lab8 - Cluster Analysis with RClustering a set of    objects into    groups is usually moved by the aim of identifying internally homogenous groups according to a specific set of variables. In order to accomplish this objective, the starting point is computing a matrix, called dissimilarity matrix, which contains information about the dissimilarity of the observed units. According to the nature of the observed variables (quantitative, qualitative, binary or mixed type variables), we can define and use different measures of dissimilarity. 1) Repeat the following examples2) Apply complete clustering analysis to Lab8_priorities_perifereion dataset  Example 1Data Longley contained in the R-package AER consists of the number of people employed from 1947-1962.<jupyter_code># Libraries & Preparation
install.packages('AER') #uncomment this if you want to install AER
library(AER)
# Dataset Init
data("Longley")
longley<-as.data.frame(Longley)
x<-longley$employment/1000
x
label.x<-as.character(c(1947:1962))

# Calculate the dissimilarity between observations using the Euclidean distance
dist.l<-dist(x,method="euclidean")

# Compute a hierarchical cluster analysis on the distance matrix using the complete linkage method
h<-hclust(dist.l, method="complete")
print(h)

# In order to see all the steps of the clustering type:
h$merge

print("Note that the minus in front of the unit number indicates that this is a single observation being merged;
whereas numbers alone indicate the step at which the considered clusters were built.")

# Create a plot of the clustering tree
plot(h,labels=label.x) # 'plclust' is deprecated
title("Dendrogram of employment figures: Complete linkage")<jupyter_output><empty_output><jupyter_text>What is an appropriate number of clusters according to this plot?A common choice is to cut the tree by the largest difference of  heights between two nodes. The height values are contained in the output of hclust function:<jupyter_code>h.cl<-h$height # height values
h.cl
h.cl2<-c(0,h.cl[-length(h.cl)]) # vector that has to be substracted from h.cl
h.cl2
round(h.cl-h.cl2,3) # differences in height, rounded at the 3rd digit
max(round(h.cl-h.cl2,3)) # the largest increase
which.max(round(h.cl-h.cl2,3)) # the step of the largest increase<jupyter_output><empty_output><jupyter_text>According to this approach, the appropriate number of cluster is two, because the largest difference is at the last step of the merging process.<jupyter_code># Compute a hierarchical cluster analysis on the distance matrix using the average linkage method:

h<-hclust(dist.l,method="average")
print(h)

plot(h,labels=label.x) # 'plclust' is deprecated
title("Dendrogram of employment figures: Average linkage")<jupyter_output>Call:
hclust(d = dist.l, method = "average")

Cluster method   : average 
Distance         : euclidean 
Number of objects: 16<jupyter_text>In order to choose where to cut the tree, the differences in the height values are evaluated:<jupyter_code>h.cl<-h$height # height values
h.cl
h.cl2<-c(0,h.cl[-length(h.cl)]) # vector that has to be substracted from h.cl
round(h.cl-h.cl2,3) # differences in height, rounded at the 3rd digit
max(round(h.cl-h.cl2,3)) # the largest increase
which.max(round(h.cl-h.cl2,3)) # the step of the largest increase
plot(h)<jupyter_output><empty_output><jupyter_text>Example 2Consider data set `lifeexp.dat`. It contains life expectancy in the 1960s distinguished by country, age and sex.Perform single, complete and average linkage agglomerative hierarchical cluster based on the Euclidean distance measure and produce suitable plots.<jupyter_code>life<-read.table("./lifeexp.dat",header=TRUE, fill = TRUE) #row.names = NULL)
#life = life[complete.cases(life),]
#life[life==""] <- NA
#life
#rownames(life)<-life[,1]
country <-rownames(life)

par(mfrow=c(1,3))
plot(hclust(dist(life), method="single"), labels=country , ylab="Distance")
title("Single Linkage")
plot(hclust(dist(life), method="complete"), labels=country, ylab="Distance")
title("Complete Linkage")
plot(hclust(dist(life), method="average"), labels=country , ylab="Distance")
title("Average Linkage")<jupyter_output><empty_output><jupyter_text>The singe linkage method shows a high degree of asymmetry – an example of 'chaining', i.e. result of'prematurely'  combining  individuals/clusters  using  the  minimum  distance  criteria  which  defines  single linkage.The complete linkage method is much more balanced in the way it forms clusters, producing four or five clear clusters.The average linkage method returns clusters which again show asymmetry, but not as pronounced as single linkage.Suppose to cut the complete linkage dendrogram at height 8: it would yield 4 clusters. Try to construct four clusters using the k-means algorithm as follows:<jupyter_code>clusters.km <- kmeans(na.omit(life), 4) 
country.clus.km <- lapply(1:4, function(nc) {
    country[clusters.km$cluster==nc]
}) 

country.clus.km<jupyter_output><empty_output><jupyter_text>Example 3Consider again the `sparrowElphick` dataset. Now try to use the k-means cluster analysis to check whether there are two clusters,one containing the sparrows which survived and the other the sparrows which died.Using the R-function kmeans(), select two clusters and obtain the following output:<jupyter_code>sparrows<-read.table("./SparrowsElphick.txt",header=TRUE)
kmeans(sparrows, 2)<jupyter_output><empty_output><jupyter_text>Do the two clusters contain dead/alive sparrows respectively? No.Run now the k-means clustering allowing the number of clusters to vary from 2 to 5 and compare the results in terms of ASW and PG indexes.ASW and PG indexes can be obtained with the function cluster.stats, in package fpc.<jupyter_code># Libraries 
install.packages('fpc') #uncomment this if you want to install fpc
library(fpc)
#?cluster.stats uncomment this if you want more details about cluster.stats.
dist.data<-dist(sparrows,"euclidean")
cl2<-kmeans(sparrows, 2)
out.cl2<-cluster.stats(dist.data,cl2$cluster)
out.cl2$avg.silwidth<jupyter_output><empty_output><jupyter_text>According to the values of the two indexes, the k-means with two clusters produces more homogenous groups; the second best choice would be fixing the number k of clusters equal to 3.A meaningful representation of the clustering outcome is the so called 'silhouette plot'. On the x-axis it shows the silhouette width for each observation in the corresponding cluster; units in the same cluster are plotted in decreasing order according to their silhouette value. Different clusters are separately plotted. Furthermore, it reports the number of observations in each cluster and the average silhouette width of the classification.In  order  to  produce  a  silhouette  plot  we  need  to  load  the  cluster library  and  to  use  functionsilhouette:<jupyter_code>library(cluster)
sil<-silhouette(cl2$cluster,dist.data)
plot(sil,cex.names=0.6,main="Silhouette Plot for 'Sparrows' dataset, k=2",border=NA)<jupyter_output><empty_output><jupyter_text>Or, with coloured clusters:<jupyter_code>plot(sil,cex.names=0.6,nmax=98,main="Silhouette Plot for 'Sparrows' dataset, k=2", col=c("red","green"), border=NA)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08731147087311471, 'fraction_numerical': 0.010792860107928601, 'mean_word_length': 5.95, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '302826', 'license': 'MIT', 'n_tokens_mistral': 2234, 'n_tokens_neox': 2064, 'n_words': 805, 'path': 'labs/lab8/Clustering Examples Notebook.ipynb', 'repo_name': 'eloukas/uth-machine-learning', 'size': '7227'}