<jupyter_start><jupyter_text>Lecture 2 - Management SolutionsAuthors:- <NAME>- <NAME><jupyter_code># Math
import numpy as np
from scipy.spatial.distance import cdist

# Dataframe
import pandas as pd 

# Representation
import seaborn as sns
import matplotlib.pyplot as plt

# Machine Learning

from sklearn.cluster import KMeans

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read dataset<jupyter_code>df = pd.read_csv('Mall_Customers.csv')

# Rename columns
df.columns = ['ID', 'Gender', 'Age', 'Annual Income', 'Score']<jupyter_output><empty_output><jupyter_text>Dataset info<jupyter_code>df.info
df.sample(5)<jupyter_output><empty_output><jupyter_text>Statistical values of the variablesPandas has a very useful function called describe which automatically gives to us the number of rows, the mean, the standard deviation, the minimun value, the percentiles and the maximun value of each row<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Missing valuesThere are no missing values in any variable<jupyter_code>df.isna().sum()<jupyter_output><empty_output><jupyter_text>Dataset preprocess One Hot EncodingAs we seen before, we have 5 variables. The Gender variable is a categorical variable so the first thing we have to do is one hot encode this column.<jupyter_code>df['Gender'] = df['Gender'].apply(lambda x: 1 if x == 'Male' else 0)<jupyter_output><empty_output><jupyter_text>Missing valuesAs we seen before there are no missing values, but if there had been any missing values we could drop the rows with missing values or take the mean of the column.<jupyter_code>df.fillna(df.mean())<jupyter_output><empty_output><jupyter_text>Data representation Histogram Histogram of ageAs we can see in the following histogram, there are 3 peaks of age. Around the tweenties we can see the first peak, the next peak is around the thirties and the last one on the fifties.<jupyter_code>df['Age'].hist(bins=70)
plt.xlabel('Age')
plt.ylabel('Number')<jupyter_output><empty_output><jupyter_text>Gender count<jupyter_code>sns.countplot(data=df, x='Gender')<jupyter_output><empty_output><jupyter_text>Scatter plot of Age, Annual Income, Gender and ScoreAs we can see in the scatter plot of Annual Income and Score we can identify 5 groups (which will 5 cluster in KMeans). People around 50k of Annual Income are concentrated, in other words, the std is very low. It's worth to say, that we can see two groups of people, both in high and low Annual Income, the ones which spend a lot of money and the others which doesn't spend much money. The maximum spending score values are for women. If we see the relation between Annual Income and Gender, we can see that there a lot of Males and Females whose income is low and there are a few whose income is very high. The person whose annual income is the highest is a women.The annual income in male gender follows a normal distribution, and in the female gender is almost a normal distribution.<jupyter_code>sns.pairplot(df, vars=['Gender', 'Age', 'Annual Income', 'Score'], hue='Gender')<jupyter_output>/usr/lib/python3.8/site-packages/seaborn/distributions.py:288: UserWarning: Data must have variance to compute a kernel density estimate.
  warnings.warn(msg, UserWarning)
/usr/lib/python3.8/site-packages/seaborn/distributions.py:288: UserWarning: Data must have variance to compute a kernel density estimate.
  warnings.warn(msg, UserWarning)<jupyter_text>Distribution plot AgeWe can see that we don't have values of underage population, and we have a lot of samples of people in their thirties.<jupyter_code>sns.distplot(df['Age'])
sns.swarmplot(df['Age'])<jupyter_output><empty_output><jupyter_text>GenderThere a few more womens than mens, but the population is balanced.<jupyter_code>sns.distplot(df['Gender'])
sns.swarmplot(df['Gender'])<jupyter_output><empty_output><jupyter_text>Annual IncomeThe annual incomes rose in a lineal fashion until it reach the mean anual incommes and then it decrease fastly until it reach the maximum income.<jupyter_code>sns.distplot(df['Annual Income'])
sns.swarmplot(df['Annual Income'])<jupyter_output><empty_output><jupyter_text>ScoreThe spending score almost follows a normal distribution.<jupyter_code>sns.distplot(df['Score'])
sns.swarmplot(df['Score'])<jupyter_output><empty_output><jupyter_text>KMeans modelIn order to find the best number of cluster we are going to use the Elbow method. In order to train the KMeans model, we drop the ID column because it doesn't provide any information.<jupyter_code>X = df.drop(columns=['ID'])
kmeans_models = []
distortions = []
inertias = []
scores = []
cluster_range = range(1, 11)

for i in cluster_range:
    print(f"Training KMeans models with n-clusters: {i}")
    kmeans_models.append(KMeans(n_clusters=i))
    kmeans_models[-1].fit(X)
    distortions.append(sum(np.min(cdist(X, kmeans_models[-1].cluster_centers_, 'euclidean'), axis=1)) / X.shape[0])
    inertias.append(kmeans_models[-1].inertia_)
    scores.append(kmeans_models[-1].score(X))<jupyter_output>Training KMeans models with n-clusters: 1
Training KMeans models with n-clusters: 2
Training KMeans models with n-clusters: 3
Training KMeans models with n-clusters: 4
Training KMeans models with n-clusters: 5
Training KMeans models with n-clusters: 6
Training KMeans models with n-clusters: 7
Training KMeans models with n-clusters: 8
Training KMeans models with n-clusters: 9
Training KMeans models with n-clusters: 10<jupyter_text>Elbow method with Distortion<jupyter_code>plt.plot(cluster_range, distortions, 'bo-')
plt.xlabel('Cluster number')
plt.ylabel('Distortion')
plt.xticks(cluster_range)
plt.title('Elbow method')<jupyter_output><empty_output><jupyter_text>Elbow method with Inertia<jupyter_code>plt.plot(cluster_range, inertias, 'bo-')
plt.xlabel('Cluster number')
plt.ylabel('Inertia')
plt.xticks(cluster_range)
plt.title('Elbow method')<jupyter_output><empty_output><jupyter_text>Elbow method with Score<jupyter_code>plt.plot(cluster_range, scores, 'bo-')
plt.xlabel('Cluster number')
plt.ylabel('Score')
plt.xticks(cluster_range)
plt.title('Elbow method')<jupyter_output><empty_output><jupyter_text>As we can see, the improvement after 6 clusters is minimal and not signifcant, so our final model is a KMeans with 6 clustes<jupyter_code># The index of the array goes from 0 to n - 1, 
# so if we want the model with 6 cluster, index = 5.
final_model = kmeans_models[5]
final_model.cluster_centers_
cluster_centers_annual_income = final_model.cluster_centers_[:, 2]
cluster_centers_score = final_model.cluster_centers_[:, 3]<jupyter_output><empty_output><jupyter_text>Cluster centroids for Annual Income and Score<jupyter_code>plt.figure(figsize=(10, 8))
plt.title('Annual Income vs Score and cluster centroids')
ax = sns.scatterplot(data=df, x='Score', y='Annual Income')
sns.scatterplot(x=cluster_centers_score, y=cluster_centers_annual_income, marker='x')
ax.legend(['Samples', 'Cluster Centroids'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09645979647412928, 'fraction_numerical': 0.007739716210405618, 'mean_word_length': 6.7792642140468224, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '637489', 'license': 'MIT', 'n_tokens_mistral': 2355, 'n_tokens_neox': 2222, 'n_words': 707, 'path': 'Lecture 2 - Management Solutions/.ipynb_checkpoints/Lecture2-checkpoint.ipynb', 'repo_name': 'GandalFran/tie-lectures', 'size': '7005'}