<jupyter_start><jupyter_text>**<NAME> SUBMISSION FOR TECHNICAL TEST ****M2 DATA SCIENCE ECOLE POLYTECHNIQUE/INSTITUT POLYTECHNIQUE DE PARIS** **This notebook will be in english for accessibility puroposes**. Let us start by importing some libraries that we'll use<jupyter_code>import pandas as pd 
from sklearn.decomposition import PCA 
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.preprocessing import normalize,StandardScaler<jupyter_output><empty_output><jupyter_text>Now we load our data<jupyter_code>df = pd.read_csv("https://github.com/BechirTr/Some_Projects/raw/main/Dataset.csv", sep=",")
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2164 entries, 0 to 2163
Data columns (total 57 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   ID                          2164 non-null   object 
 1   URL                         2164 non-null   object 
 2   CRAWL_SOURCE                2164 non-null   object 
 3   PROPERTY_TYPE               2164 non-null   object 
 4   NEW_BUILD                   1973 non-null   object 
 5   DESCRIPTION                 2160 non-null   object 
 6   IMAGES                      2164 non-null   object 
 7   SURFACE                     2050 non-null   float64
 8   LAND_SURFACE                3 non-null      float64
 9   BALCONY_SURFACE             0 non-null      float64
 10  TERRACE_SURFACE             25 non-null     float64
 11  ROOM_COUNT                  1835 non-null   float64
 12  BEDROOM_COUNT               696 non-null    float64
 13  BATHROOM_COUNT              0 non[...]<jupyter_text>We can observe the data is not clean and with lots of nan values, we will proceed to some preprocessing and cleaning tasks.First we will drop all columns with a total of nan values greater than 50% of the total number of observations.<jupyter_code>df = df.dropna(axis=1,thresh=1200)
id=df["ID"].tolist()<jupyter_output><empty_output><jupyter_text>Next will will create dummy columns to encode the categorical variables in our data frame so that we can feed it to an estimator later.<jupyter_code>obj_df = df.select_dtypes(include=['object']).copy()
obj_df.head()
obj_df=pd.get_dummies(obj_df, columns=['CRAWL_SOURCE', 'PROPERTY_TYPE',
       'MARKETING_TYPE',
      'OCCUPIED', 'DEALER_TYPE'])

obj_df=obj_df.select_dtypes(exclude=['object'])<jupyter_output><empty_output><jupyter_text>Now we merge the two dataframes and we drop all the remaining categorical variables, and we convert the booleans to integers.Moreover since all the ads are located in the same city we can drop the 'CITY_ID', 'ZIP_CODE', 'DEPT_CODE' columns as well, as they are not significative in our case.<jupyter_code>df=pd.concat([df,obj_df],axis=1)
for (name,data) in df.iteritems():
    if  df[name].dtype == "bool":
        df[name] = df[name].astype(int)
df=df.select_dtypes(exclude=['object']) 
df=df.drop(['CITY_ID', 'ZIP_CODE', 'DEPT_CODE'],axis='columns')
old_df=df.copy()<jupyter_output><empty_output><jupyter_text>Now lets observe our new dataframe.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>We can see that the statistical dispersion of the data is pretty high, which will surely affect the quality of any estimator we use, as such we proceed to standardize our data frame.<jupyter_code>scaler=StandardScaler()
df_scaled = pd.DataFrame(scaler.fit_transform(old_df), columns=old_df.columns)
df_scaled.describe()<jupyter_output><empty_output><jupyter_text>Now we can assume that we are in a case of unsupervised learning, because we do not have acess to a label vector mapping each ad to a given real estate. Our first intuition at this point is finding a way to visualize our data somehow and see if we can observe any clusters, for this purpose we will rely on Principal Component Analysis ( PCA ), but first we have to deal with all the nan values still present within our dataframe and for that we will use an imputer strategy.<jupyter_code>imp=SimpleImputer(strategy="median")
imp.fit(df_scaled)
df_scaled=imp.transform(df_scaled)<jupyter_output><empty_output><jupyter_text>Now this part is optional, but in order to avoid any variables being misleading in terms of statistical significance we can plot the correlation matrix and retrieve variables whom correlation is very high and drop one of them to avoid being misled by their signficance.<jupyter_code>cor=df.corr()
sns.set(font_scale=1.5)
hm = sns.heatmap(cor.to_numpy(),
                 cbar=True,
                 fmt='.2f',
                 annot_kws={'size': 24},
                )
plt.title('Covariance matrix showing correlation coefficients')
plt.show()
indices = np.argwhere(np.abs(cor.to_numpy())>0.85)
indices.shape
temp_list=[]
for i,j in zip(indices[:,0],indices[:,1]):
  if i != j :
    temp_list.append((i,j))
temp_list

    
''' 
This part is optional and requires further investigating
'''
#df_scaled = np.delete(df_scaled,24,axis=1)
#df_scaled.shape<jupyter_output><empty_output><jupyter_text>We can now proceed to apply PCA<jupyter_code>pca=PCA()
pca.fit(df_scaled)
df=pca.transform(df_scaled)
plt.scatter(df[:,0],df[:,1],alpha=0.4)
plt.show()

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(df[:,0],df[:,1],df[:,1])
plt.show()<jupyter_output><empty_output><jupyter_text>We can now see the clusters ( dense blue ), We can apply KMeans for instance to cluster them. We can use the elbow rule to determine the most probable number of significant clusters.<jupyter_code>eigenvalues = pca.explained_variance_
plt.bar(np.arange(0,31,1), eigenvalues)
plt.plot(eigenvalues, "r")
plt.plot(eigenvalues, "ro")
plt.show()<jupyter_output><empty_output><jupyter_text>Using the elbow rule we have two possible choices, either 21 clusters or 25 clusters, we will try both.<jupyter_code>kmeans = KMeans(n_clusters = 21)
kmeans.fit(df)
clusters = kmeans.predict(df)

plt.scatter(df[:,0],df[:,1], c = clusters,alpha=0.4, cmap = 'viridis')
centers = kmeans.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c = 'black', s = 200, alpha = 0.5)

        
kmeans = KMeans(n_clusters = 25)
kmeans.fit(df)
clusters = kmeans.predict(df)

plt.scatter(df[:,0],df[:,1], c = clusters,alpha=0.4, cmap = 'viridis')
centers = kmeans.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c = 'black', s = 200, alpha = 0.5)<jupyter_output><empty_output><jupyter_text>Both choices give pretty close results. Now we want to retreive the clustered ads so we'll create a dataframe mapping each ad to its respective cluster.<jupyter_code>id=np.array(id)

labels = kmeans.labels_

Labeled_IDs=pd.DataFrame({"ID":id,"Real estate label":labels})

Labeled_IDs.head()

Labeled_IDs.loc[Labeled_IDs["Real estate label"]==10]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10533889768311987, 'fraction_numerical': 0.022161462080874945, 'mean_word_length': 4.114054451802796, 'pattern_counts': {'":': 3, '<': 54, '<?xml version=': 0, '>': 55, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '289220', 'license': 'Apache-2.0', 'n_tokens_mistral': 2408, 'n_tokens_neox': 2251, 'n_words': 665, 'path': 'Python/Clustering of real estate ads PCA and KMeans/Bechir_TRABELSI_Submission.ipynb', 'repo_name': 'MohamedMkaouar/Some_Projects', 'size': '6958'}