<jupyter_start><jupyter_text>Do you need to reduce the number of columns?<jupyter_code>data.head()
# We need to reduce the number of columns because some columns does not have impact in our analyses like
# Organization Group Code  and others basically is a "duplicate of columns" like for example the total salary is a sum
# of all salaries paid to City employees.So we can use the parcials salaries in our analyses 
# variable like Union, Employee Identifier... is just for get idea about our data and these will not have a big
# impact in our analyses<jupyter_output><empty_output><jupyter_text>Is your data clean?<jupyter_code>data.isnull().sum()
# as we can see Our data is not cleaning , we have null value in Department Code ,Department,Union Code,Union  and job
# visualize the null column.
data[data['Department Code'].isnull()]
data1=data.dropna()
data1
print (data.shape)
print (data1.shape)
# see the info about my new data
data1.info()
data1.isnull().sum()
# see the unique value for each column 
col_name=data1.columns
for i in col_name:
    print('column',i,' has :',data1[i].nunique(),"Unique values")
data1.Job.unique()
data1.dtypes<jupyter_output><empty_output><jupyter_text>What all parameters do we need to give to our clustering algorithm? We need parameters that will have an impact on our analyses so for the clustering algorithm I will choose 4 parameters that are: Salaries 'Overtime', 'Other Salaries', 'Retirement','Health and Dental', 'Other Benefits']]As we can see in the described problem: Normal salaries paid to permanent or temporary City employee so we have to include it because is important for analyses Overtime is the fact a kind of salary, but I include it in our analyses because we have an employee who would be temporary or not work 40h. Those employees are not include in the overtime category Other Salaries like the problem described, it may include SC, irregular payments made to City employees including premium pay,  drop all variable that we do not need<jupyter_code>data1.info()
features_to_drop=['Organization Group Code','Job Family Code','Job Code','Year Type','Organization Group','Department Code',
                  'Department','Union','Job Family','Job',
       'Year','Union Code','Employee Identifier', 'Total Salary', 'Total Benefits',
       'Total Compensation']
final_data=data1.drop(features_to_drop,axis=1)
final_data.info()
sns.distplot(data1['Health and Dental'])
plt.show()<jupyter_output>C:\Users\marce\anaconda3\lib\site-packages\seaborn\distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>Skewness is a measure of symmetry, or more precisely, the lack of symmetry. A distribution, or data set, is symmetric if it looks the same to the left and right of the center point.Kurtosis is a measure of whether the data are heavy-tailed or light-tailed relative to a normal distribution. That is, data sets with high kurtosis tend to have heavy tails, or outliers. Data sets with low kurtosis tend to have light tails, or a lack of outliers. A uniform distribution would be the extreme case.==============================================================https://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm:~:text=Skewness%20is%20a%20measure%20of,precisely%2C%20the%20lack%20of%20symmetry.&text=Kurtosis%20is%20a%20measure%20of,have%20heavy%20tails%2C%20or%20outlier<jupyter_code>print("Skewness: %f" % data1['Salaries'].skew())
print("Kurtosis: %f" % data1['Salaries'].kurt())

# let's see the range salary , overtime and all others informatiom for those variable that i considered by ploting the bar plot
fig_dims = (12, 5)
fig, ax = plt.subplots(figsize=fig_dims)
sns.countplot(data=final_data)
from sklearn.cluster import KMeans
kmeans=KMeans(n_clusters=3)
kmeans<jupyter_output><empty_output><jupyter_text>fit the data to a clustering model.<jupyter_code>y_predict=kmeans.fit_predict(data1[[ 'Salaries', 'Overtime', 'Other Salaries', 'Retirement',
       'Health and Dental', 'Other Benefits']])
print(kmeans.cluster_centers_)
y=kmeans.labels_
print(y)
# appedent my y_predict in my data frame to see how it look like
final_data['cluster']=y_predict

final_data.head(20)
# see if my data is balanced or not by using the countplot 
sns.countplot(x=kmeans.labels_, palette='Oranges')
plt.show()
# As we can see in the counterplot, we can say that our data are balanced because we do not have disproportionality between clustering 0,1 and 2
# even now that we have more number 2 than 0 or 1<jupyter_output><empty_output><jupyter_text>Find the number of clusters for the data.<jupyter_code>from scipy.spatial.distance import cdist, pdist
from sklearn.cluster import KMeans
k_rang=range(1,10)
sse=[] # my some of squere error is an array
for k in range(1,10):
    km=KMeans(n_clusters=k) # for each iteration i create a new model which cluster =k
    km.fit(final_data[['Salaries','Overtime','Other Salaries','Retirement','Health and Dental','Other Benefits']])
    # to get the sum of square error i will call the inertia parameter
    sse.append(km.inertia_)
sse 
plt.xlabel('k')
plt.ylabel('Sum of squere')
plt.plot(k_rang,sse)<jupyter_output><empty_output><jupyter_text>The number of clustering is 3<jupyter_code>kIdx = 2

# elbow curve
# Set the size of the plot
plt.figure(figsize=(10,4))
plt.subplot(1, 2, 1)
plt.plot(K, avgWithinSS, 'b*-')
plt.plot(K[kIdx], avgWithinSS[kIdx], marker='o', markersize=12,
markeredgewidth=2, markeredgecolor='r', markerfacecolor='None')
plt.grid(True)
plt.xlabel('Number of clusters')
plt.ylabel('Average within-cluster sum of squares')
plt.title('Elbow for KMeans clustering')
plt.subplot(1, 2, 2)
plt.plot(K, varExplained, 'b*-')
plt.plot(K[kIdx], varExplained[kIdx], marker='o', markersize=12,
markeredgewidth=2, markeredgecolor='r', markerfacecolor='None')
plt.grid(True)
plt.xlabel('Number of clusters')
plt.ylabel('Percentage of variance explained')
plt.title('Elbow for KMeans clustering')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Try fitting the model to both the clustering algorithms you have learnt,K-Means and Hierarchical clustering.<jupyter_code>plt.scatter(final_data.iloc[:,0].values,final_data.iloc[:,3].values, c=kmeans.labels_, cmap="rainbow")
plt.show()
centers = np.array(kmeans.cluster_centers_)
plt.scatter(centers[:,0], centers[:,1], marker="x", color='k')
plt.scatter(final_data.iloc[:,0].values,final_data.iloc[:,3].values, c=kmeans.labels_, cmap="rainbow")
centers = np.array(kmeans.cluster_centers_)
plt.scatter(centers[:,0], centers[:,1], marker="X", color='k')
plt.show()<jupyter_output><empty_output><jupyter_text>We can see the 3 clustering with the centroid   Now let's try fitting the model using the Hierarchical clustering model<jupyter_code>import scipy
from scipy.cluster.hierarchy import dendrogram,linkage
from scipy.cluster.hierarchy import fcluster
from scipy.cluster.hierarchy import cophenet
from scipy.spatial.distance import pdist

from pylab import rcParams
from sklearn.cluster import AgglomerativeClustering
import sklearn.metrics as sm
plt.style.available
#final_data = final_data.drop('cluster')
final_data = final_data.drop(final_data.columns[['cluster']], axis=1)
final_data
plt.figure(figsize=(15,10))
Z=linkage(final_data,method='single')
print("Z-Shape:",Z.shape)
plt.title("Single",size=30)
dendrogram(Z,orientation='top',truncate_mode='lastp',p=12,get_leaves=False,leaf_rotation=45,leaf_font_size=15,show_contracted=True,)
plt.xlabel("Cluster Size",fontsize=30)
plt.ylabel("Distances",fontsize=30)
plt.show()
""" dtypes = {
        'ip'            : 'uint32',
        'app'           : 'uint16',
        'device'        : 'uint16',
        'os'            : 'uint16',
        'channel'       : 'uint16',
        'is_attributed' : 'uint8',
        }
"""
# I had problem with the memory, so i will reduce the reduce element in my data by specify number of rows to import. For exampe:   nrows=10000) will only read the first 10000 rows (including the heading)..
Hierarchical_data=pd.read_csv("C:\\Users\\marce\\OneDrive\\Ambiente de Trabalho\\DataScience\\Unsupervisor_learning\\Clustering\\employee-compensation.csv",nrows=10000)

Hierarchical_data.head()
Hierarchical_data.shape
Hierarchical_data.info
features_to_drop=['Organization Group Code','Job Family Code','Job Code','Year Type','Organization Group','Department Code',
                  'Department','Union','Job Family','Job',
       'Year','Union Code','Employee Identifier', 'Total Salary', 'Total Benefits',
       'Total Compensation']
Hierarchical_final_data=Hierarchical_data.drop(features_to_drop,axis=1)
Hierarchical_final_data
plt.figure(figsize=(15,10))
Z=linkage(Hierarchical_final_data,method='single')
print("Z-Shape:",Z.shape)
plt.title("Single",size=30)
dendrogram(Z,orientation='top',truncate_mode='lastp',p=12,get_leaves=False,leaf_rotation=45,leaf_font_size=15,show_contracted=True,)
plt.xlabel("Cluster Size",fontsize=30)
plt.ylabel("Distances",fontsize=30)
plt.show()
from sklearn.cluster import AgglomerativeClustering
hc = AgglomerativeClustering(n_clusters = 5, affinity = 'euclidean', linkage = 'single')
y_hc = hc.fit_predict(Hierarchical_final_data)

# Visualising the clusters
plt.scatter(X[y_hc == 0, 0], X[y_hc == 0, 1], s = 50, c = 'red', label = 'Careful')
plt.scatter(X[y_hc == 1, 0], X[y_hc == 1, 1], s = 50, c = 'blue', label = 'Standard')
plt.scatter(X[y_hc == 2, 0], X[y_hc == 2, 1], s = 50, c = 'green', label = 'Target')
plt.scatter(X[y_hc == 3, 0], X[y_hc == 3, 1], s = 50, c = 'cyan', label = 'Careless')
plt.scatter(X[y_hc == 4, 0], X[y_hc == 4, 1], s = 50, c = 'magenta', label = 'Sensible')
plt.title('Clusters of customers')
plt.legend()
plt.show()
# method complete
plt.figure(figsize=(15,10))
Z=linkage(Hierarchical_final_data,method='complete')
print("Z-Shape:",Z.shape)
plt.title("COMPLETE",size=30)
dendrogram(Z,orientation='top',truncate_mode='lastp',p=12,get_leaves=False,leaf_rotation=45,leaf_font_size=15,show_contracted=True,)
plt.xlabel("Cluster Size",fontsize=30)
plt.ylabel("Distances",fontsize=30)
plt.show()
from sklearn.cluster import AgglomerativeClustering
hc = AgglomerativeClustering(n_clusters = 5, affinity = 'euclidean', linkage = 'complete')
y_hc = hc.fit_predict(Hierarchical_final_data)
X = Hierarchical_final_data.iloc[:, :].values

print (X)
# Visualising the clusters
plt.scatter(X[y_hc == 0, 0], X[y_hc == 0, 1], s = 50, c = 'red', label = 'Careful')
plt.scatter(X[y_hc == 1, 0], X[y_hc == 1, 1], s = 50, c = 'blue', label = 'Standard')
plt.scatter(X[y_hc == 2, 0], X[y_hc == 2, 1], s = 50, c = 'green', label = 'Target')
plt.scatter(X[y_hc == 3, 0], X[y_hc == 3, 1], s = 50, c = 'cyan', label = 'Careless')
plt.scatter(X[y_hc == 4, 0], X[y_hc == 4, 1], s = 50, c = 'magenta', label = 'Sensible')
plt.title('Clusters of customers')

plt.legend()
plt.show()
# average method 

plt.figure(figsize=(15,10))
Z=linkage(Hierarchical_final_data,method='average')
print("Z-Shape:",Z.shape)
plt.title("Average",loc='center',size=30)
dendrogram(Z,orientation='top',truncate_mode='lastp',p=12,get_leaves=False,leaf_rotation=45,leaf_font_size=15,show_contracted=True,)
plt.xlabel("Cluster Size",fontsize=30)
plt.ylabel("Distances",fontsize=30)
plt.show()
#weighted method 
plt.figure(figsize=(15,10))
Z=linkage(Hierarchical_final_data,method='weighted')
print("Z-Shape:",Z.shape)
plt.title("Weighted",loc='center',size=30)
dendrogram(Z,orientation='top',truncate_mode='lastp',p=12,get_leaves=False,leaf_rotation=45,leaf_font_size=15,show_contracted=True,)
plt.xlabel("Cluster Size",fontsize=30)
plt.ylabel("Distances",fontsize=30)
plt.show()
# centroid method
plt.figure(figsize=(15,10))
Z=linkage(Hierarchical_final_data,method='centroid')
print("Z-Shape:",Z.shape)
plt.title("CENTROID",loc='center',size=30)
dendrogram(Z,orientation='top',truncate_mode='lastp',p=12,get_leaves=False,leaf_rotation=45,leaf_font_size=15,show_contracted=True,)
plt.xlabel("Cluster Size",fontsize=30)
plt.ylabel("Distances",fontsize=30)
plt.show()
# ward method
plt.figure(figsize=(15,10))
Z=linkage(Hierarchical_final_data,method='ward')
print("Z-Shape:",Z.shape)
plt.title("WARD",size=30)
dendrogram(Z,orientation='top',truncate_mode='lastp',p=12,get_leaves=False,leaf_rotation=45,leaf_font_size=15,show_contracted=True,)
plt.xlabel("Cluster Size",fontsize=30)
plt.ylabel("Distances",fontsize=30)
plt.axhline(y=500)
plt.axhline(y=1500)
plt.show()
from sklearn.cluster import AgglomerativeClustering
hc = AgglomerativeClustering(n_clusters = 5, affinity = 'euclidean', linkage = 'ward')
y_hc = hc.fit_predict(Hierarchical_final_data)
X = Hierarchical_final_data.iloc[:, :].values
# X = Hierarchical_final_data.iloc[:, [0, 5]].values
print (X)
# Visualising the clusters
plt.scatter(X[y_hc == 0, 0], X[y_hc == 0, 1], s = 50, c = 'red', label = 'Careful')
plt.scatter(X[y_hc == 1, 0], X[y_hc == 1, 1], s = 50, c = 'blue', label = 'Standard')
plt.scatter(X[y_hc == 2, 0], X[y_hc == 2, 1], s = 50, c = 'green', label = 'Target')
plt.scatter(X[y_hc == 3, 0], X[y_hc == 3, 1], s = 50, c = 'cyan', label = 'Careless')
plt.scatter(X[y_hc == 4, 0], X[y_hc == 4, 1], s = 50, c = 'magenta', label = 'Sensible')
plt.title('Clusters of customers')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14536657982880535, 'fraction_numerical': 0.026051358392259024, 'mean_word_length': 6.472747497219133, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '283283', 'license': 'Unlicense', 'n_tokens_mistral': 5085, 'n_tokens_neox': 4613, 'n_words': 1188, 'path': 'projects/Kmean_project.ipynb', 'repo_name': 'MarceloD02/Marcelo-Goncalves-Portfolio', 'size': '13435'}