<jupyter_start><jupyter_text>External Libraries<jupyter_code>import numpy as np # for linear algebra
import pandas as pd # for dataframes
import matplotlib.pyplot as plt # for data visiualization
import seaborn as sns # for data visualization
from sklearn import preprocessing # for one hot encoding
from sklearn.pipeline import Pipeline # for pipelines
from sklearn.model_selection import train_test_split # for tts
from sklearn import svm # classification model
from sklearn.metrics import mean_absolute_error, jaccard_score, f1_score, log_loss, accuracy_score # metrics
from sklearn import metrics # for accuracy score metric
from sklearn.tree import DecisionTreeClassifier # Decision Trees
from tensorflow import keras # for deep learning
from tensorflow.keras import layers # for deep learning
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read data<jupyter_code>df = pd.read_csv('/content/drive/MyDrive/WRI\DSSG - EDA/DHS-PROCESSED-CLEAN.csv',
                 #index_col = 'DHSCLUST'	
                 )
df.shape
# glimpse data
df.head()<jupyter_output><empty_output><jupyter_text>Since wealth is the target, need to ensure the data is balanced.  Data Balancing<jupyter_code>df.Wealth.value_counts().plot.bar(title = 'Distribution of Wealth Classes',
                                  xlabel = 'Wealth Class',
                                  ylabel = 'Number of Records')
#Using matplotlib
pie, ax = plt.subplots(figsize=[10,6])
labels = df.Wealth.unique()
plt.pie(x=df.Wealth.value_counts(), autopct="%.1f%%", labels=labels, explode=[0.05]*5, pctdistance=0.5)
plt.title("Wealth Label Distribution", fontsize=14);
#pie.savefig("DeliveryPieChart.png")
wealth_records = df.Wealth.value_counts()
wealth_records_unbalance = wealth_records - df.Wealth.value_counts().mean()
wealth_records_unbalance.plot.bar( title = 'Class Records Imbalance',
                                  xlabel = 'Wealth Class',
                                  ylabel = 'Variance (Number of Records)')
print('Number of Poorest class records: ', wealth_records[0])
print('Number of Richer class records: ', wealth_records[-1])
print('Poorest to Richer Ratio: ', round(wealth_records[0]/wealth_records[-1],3))<jupyter_output>Number of Poorest class records:  7472
Number of Richer class records:  4724
Poorest to Richer Ratio:  1.582<jupyter_text>Looks like data is highly unbalanced towards the poorest class.Removing bias via random sampling<jupyter_code>'''# undersampling poorest class
delta = wealth_records[0] - wealth_records[-1]
to_remove = np.random.choice(df[df['Wealth'] == 'poorest'].index, delta ,replace=False)
df.drop(to_remove, inplace = True)

# undersampling poorer class
delta = wealth_records[1] - wealth_records[-1]
to_remove = np.random.choice(df[df['Wealth'] == 'poorer'].index, delta ,replace=False)
df.drop(to_remove, inplace = True)

# undersampling richest class
delta = wealth_records[2] - wealth_records[-1]
to_remove = np.random.choice(df[df['Wealth'] == 'richest'].index, delta ,replace=False)
df.drop(to_remove, inplace = True)

# undersampling poorer class
delta = wealth_records[3] - wealth_records[-1]
to_remove = np.random.choice(df[df['Wealth'] == 'middle'].index, delta ,replace=False)
df.drop(to_remove, inplace = True)

df.Wealth.value_counts().plot.bar(title = 'Distribution of Wealth Classes (random undersampled)',
                                  xlabel = 'Wealth Class',
                                  ylabel = 'Number of Records')

'''<jupyter_output><empty_output><jupyter_text>EDA For proper visualization, understanding data types<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>Quantitative Analysis<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Questions:1. Does Wealth Index has outliers? (Box and whiskers will tell).<jupyter_code>df.boxplot(column = ['Wealth Index'])<jupyter_output><empty_output><jupyter_text>Looks okay. What about correlations?<jupyter_code>df.corr()<jupyter_output><empty_output><jupyter_text>*   Latitude and Longitude are irrelevant. *   Population density has a weak correlation with Wealth Index. Makes sense.   Descriptive Analysis<jupyter_code>df.describe(include = 'object')<jupyter_output><empty_output><jupyter_text>Observations:*   No missing dataQuestions:*   Relation between Toilet Facilitty & Wealth*   Relation between Roof Material & Wealth*   Relation between Electricity & Wealth*   Relation between Drinking Water & Wealth*   Relation between Drinking Population Density & WealthThe columns I think will not help:*   ~Urban/Rural~*   DHSREGNA	*   ADM1NAME	*   geometry*   LATNUM*   LONGNUM<jupyter_code>df.drop(columns = ['DHSREGNA', 'ADM1NAME', 'LATNUM', 'LONGNUM', 'geometry'],
        inplace = True)
df.shape
df.columns<jupyter_output><empty_output><jupyter_text>Without Rural/Urban Flag Relation between Toilet Facilitty & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})
sns.boxplot(x="Toilet Facility", 
            y="Wealth Index", 
            data=df)<jupyter_output><empty_output><jupyter_text>Relation between Roof Material & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})

sns.boxplot(x="Roof Material", 
            y="Wealth Index",  
            data=df)<jupyter_output><empty_output><jupyter_text>Relation between Electricity & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})

sns.boxplot(x="Electricity", 
            y="Wealth Index",  
            data=df)<jupyter_output><empty_output><jupyter_text>Relation between Drinking Water & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})

sns.boxplot(x="Drinking Water", 
            y="Wealth Index",  
            data=df)<jupyter_output><empty_output><jupyter_text>I think differentiating between Rurual and Urban will help. Relation between Drinking Population Density & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})

sns.boxplot(x="Population_Density", 
            y="Wealth Index",  
            data=df)<jupyter_output><empty_output><jupyter_text>Grouped by Rural/Urban Relation between Toilet Facilitty & Wealth<jupyter_code>sns.boxplot(x="Toilet Facility", 
            y="Wealth Index",
            hue='URBAN_RURA', 
            data=df)<jupyter_output><empty_output><jupyter_text>Relation between Roof Material & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})

sns.boxplot(x="Roof Material", 
            y="Wealth Index",
            hue='URBAN_RURA',  
            data=df)<jupyter_output><empty_output><jupyter_text>Relation between Electricity & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})

sns.boxplot(x="Electricity", 
            y="Wealth Index",
            hue='URBAN_RURA',  
            data=df)<jupyter_output><empty_output><jupyter_text>Relation between Drinking Water & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(20, 10)})

sns.boxplot(x="Drinking Water", 
            y="Wealth Index",
            hue='URBAN_RURA',  
            data=df)<jupyter_output><empty_output><jupyter_text>Relation between Drinking Population Density & Wealth<jupyter_code>sns.set(rc={'figure.figsize':(5, 4)})

sns.boxplot(x="Population_Density", 
            y="Wealth Index",  
            data=df)<jupyter_output><empty_output><jupyter_text>Modelling & Evaluation This is a classification problem. Classification algorithms that might work:*   SVM*   Logistic Regression*   Decision Trees  Preprocessing<jupyter_code>df.columns
X = df[['Toilet Facility', 
       'Roof Material', 
       'Electricity', 
       'Cooking Fuel', 
       'Drinking Water', 
       'URBAN_RURA', 
       'Population_Density']].values
    
y = df['Wealth']

y[0:5]
le_toilet_facility = preprocessing.LabelEncoder()
le_toilet_facility.fit(['Flush', 'No Facility', 'Pit Latrine', 'Other'])
X[:,0] = le_toilet_facility.transform(X[:,0]) 

le_roof_material = preprocessing.LabelEncoder()
le_roof_material.fit(['Metal', 'Cement', 'Stones', 'Thatch/Palm', 'Tiles', 'Others', 'Mud', 'Bricks', 'Mat/Sheets', 'Wood', 'No Roof'])
X[:,1] = le_roof_material.transform(X[:,1])

le_electricity = preprocessing.LabelEncoder()
le_electricity.fit(['yes', 'no'])
X[:,2] = le_electricity.transform(X[:,2]) 

le_cooking_fuel = preprocessing.LabelEncoder()
le_cooking_fuel.fit(['Petroleum', 'Wood/Straw', 'Electricity', 'Crops', 'Dung', 'Biogas', 'Coal'])
X[:,3] = le_cooking_fuel.transform(X[:,3])

le_drinking_water = preprocessing.LabelEncoder()
le_drinking_water.fit(['Piped Water', 'Public Tap', 'Tube Well/Borehole', 'Well', 'Bottled', 'River/Dam/Lake', 'Ro Plant', 'Tanker', 'Spring', 'Other', 'Small Tank', 'Rainwater'])
X[:,4] = le_drinking_water.transform(X[:,4])

le_urban_rural = preprocessing.LabelEncoder()
le_urban_rural.fit(['U', 'R'])
X[:,5] = le_urban_rural.transform(X[:,5])

le_population_density = preprocessing.LabelEncoder()
le_population_density.fit(['Average', 'Very Low', 'Low', 'High', 'Very High'])
X[:,6] = le_population_density.transform(X[:,6])

X[:5]
y.unique()
le_wealth = preprocessing.LabelEncoder()
le_wealth.fit(['richer', 'middle', 'poorest', 'poorer', 'richest'])
y = le_wealth.transform(y)
y[:5]
#le_wealth
# test-train split
X_trainset, X_testset, y_trainset, y_testset = train_test_split(X, y, test_size=0.3, random_state=3)
print('Shape of X_trainset :',X_trainset.shape,'\nShape of y_trainset :',y_trainset.shape)
X_trainset
# to measure performance
def evaluate_ai (y_testset, y_hat):
    print("AI's Accuracy: ", round(metrics.accuracy_score(y_testset, y_hat),2))
    print("AI's Jaccard Similarity Index: ", round(jaccard_score(y_testset, y_hat, average='weighted'),2))
    print("AI's F1 Score : ", round(f1_score(y_testset, y_hat, average='weighted'),2))
    #print("AI's Log Loss: ", round(log_loss(y_testset, y_hat),2))
    print("AI's MAE: ", round(mean_absolute_error(y_testset, y_hat),2))<jupyter_output><empty_output><jupyter_text>SVM<jupyter_code>clf = svm.SVC(kernel='linear', gamma = 'auto')
clf.fit(X_trainset,y_trainset)
y_hat = clf.predict(X_testset)  
evaluate_ai(y_testset, y_hat)
mean_absolute_error(y_hat, y_testset)<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code>Ks = 12
mean_acc= np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))
ConfustionMx = [];
for n in range(1,Ks):
    
    #Train Model and Predict  
    k_hunter = DecisionTreeClassifier(criterion="entropy", max_depth = n).fit(X_trainset,y_trainset)
    y_hat2=k_hunter.predict(X_testset)
    mean_acc[n-1] = accuracy_score(y_testset, y_hat2)
    std_acc[n-1]=np.std(y_hat2==y_testset)/np.sqrt(y_hat2.shape[0])

plt.plot(range(1,Ks),mean_acc,'g')
plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)
plt.legend(('Accuracy Score ', '+/- 3xstd'))
plt.ylabel('Accuracy Score')
plt.xlabel('Depth')
plt.tight_layout()
plt.show()

Tree_optimizers = []
for i in range(0,len(mean_acc)):
    if mean_acc[i] == mean_acc.max():
        Tree_optimizers.append(i)
        print('Depth With Max Accuracy :',i,'\nMax accuracy :',mean_acc.max())

print('\n')
depth_optimized = Tree_optimizers[0]
Tree_optimized = DecisionTreeClassifier(criterion="entropy", max_depth = depth_optimized)
Tree_optimized.fit(X_trainset,y_trainset)
y_hat_optimized = Tree_optimized.predict(X_testset)
# Build the model
DTree = DecisionTreeClassifier(criterion="entropy", max_depth = 9)
DTree # it shows the default parameters

# Fit the model
DTree.fit(X_trainset,y_trainset)

# Make predictions
predTree = DTree.predict(X_testset)
evaluate_ai (y_testset, predTree)<jupyter_output><empty_output><jupyter_text>Classification via Neural Network<jupyter_code><jupyter_output><empty_output><jupyter_text>Best Performing Models:| Approach | Mean Absolute Error ||:--------:|:-------------------:|| Linear Regression With One Hot Encoding | 27638.585 || Linear Lasso Regression With Catboost Encoder | 28267.662 || Decision Tree Regression (depth = 5)  With CatBoost Encoder | 28096.050 || Decision Tree Regression (depth = 8)  With One Hot Encoding | 26523.034 | | CatBoost Regression With CatBoost Encoder | 26392.809 |<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13662742497098326, 'fraction_numerical': 0.013928038467915769, 'mean_word_length': 5.202056555269923, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 100, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '212673', 'license': 'MIT', 'n_tokens_mistral': 4404, 'n_tokens_neox': 4181, 'n_words': 807, 'path': '(DSSG_WRI)_DHS_Analysis_.ipynb', 'repo_name': 'sunayana/WRI_WellBeing_Data_Layer', 'size': '12062'}