<jupyter_start><jupyter_text>2 Exploratory Data Analysis  2.1 Contents* [2 Exploratory Data Analysis](2_Exploratory_Data_Analsis)    * [2.1 Contents](2.1_Contents)    * [2.2 Introduction](2.2_Introduction)    * [2.3 Imports](2.3_Imports)    * [2.4 Load the Raw Data](2.4_Load_the_raw_data)    * [2.5 Explore the Raw Data](2.5_Explore_the_raw_data)        * [2.5.1 Categorical Features with Target](2.5.1_Categorical_Features_with_target)        * [2.5.2 Ordinal Features](2.5.2_Ordinal_Features)            * [2.5.2.1 Visualizing Ordinal Features](2.5.2.1_Visualizing_Ordinal_Features)            * [2.5.2.2 Summary of Ordinal Features](2.5.2.2_Summary_of_Ordinal_Features)        * [2.5.3 Continuous Features](2.5.3_Continuous_Features)    * [2.6 Summary of Exploratory Data Analysis](2.6_Summary_of_EDA)  2.2 Introduction In the previous notebook, we completed the data wrangling portion of this study. We divided the predictors into three sets of groups:    - Categorical Variables    - Ordinal Variables    - Continuous Variables    The goal of this notebook is to understand the relationship between the target `satisfaction` and the predictor variables  2.3 Imports<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale
from statsmodels.stats.outliers_influence import variance_inflation_factor 
import bioinfokit
from bioinfokit.visuz import cluster
import os<jupyter_output><empty_output><jupyter_text>2.4 Load the Raw data<jupyter_code>raw_data_clean = pd.read_csv('../data/interim/raw_data_clean.csv')
#check the first few obs
raw_data_clean.head()
#check the info
raw_data_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 103904 entries, 0 to 103903
Data columns (total 24 columns):
 #   Column                             Non-Null Count   Dtype  
---  ------                             --------------   -----  
 0   id                                 103904 non-null  int64  
 1   Gender                             103904 non-null  object 
 2   Customer Type                      103904 non-null  object 
 3   Age                                103904 non-null  int64  
 4   Type of Travel                     103904 non-null  object 
 5   Class                              103904 non-null  object 
 6   Flight Distance                    103904 non-null  int64  
 7   Inflight wifi service              103904 non-null  int64  
 8   Departure/Arrival time convenient  103904 non-null  int64  
 9   Ease of Online booking             103904 non-null  int64  
 10  Gate location                      103904 non-null  int64  
 11  Food and drink                     1039[...]<jupyter_text>2.5 Explore the Raw data<jupyter_code>#Preview the data
pd.set_option('display.max_columns', None)
with pd.option_context('display.max_rows',10):
    display(raw_data_clean)<jupyter_output><empty_output><jupyter_text>2.5.1 Categorical Features with target<jupyter_code>#Plot Categorical Variables by Satisfaction
%matplotlib inline
fig, ax = plt.subplots(2, 2, figsize=(12, 12))
plt.subplots_adjust(hspace=0.5);
sns.countplot(x = 'Gender', data = raw_data_clean, hue = 'satisfaction', ax= ax[0][0])
ax[0][0].set_title('Gender by Satisfaction')
sns.countplot(x = 'Customer Type', data = raw_data_clean, hue = 'satisfaction', ax= ax[0][1])
ax[0][1].set_title('Customer Type by Satisfaction')
sns.countplot(x = 'Type of Travel', data = raw_data_clean, hue = 'satisfaction', ax= ax[1][0])
ax[1][0].set_title('Type of Travel by Satisfaction')
sns.countplot(x = 'Class', data = raw_data_clean, hue = 'satisfaction', ax= ax[1][1])
ax[1][1].set_title('Class by Satisfaction')

target_categorical_cols = ['Gender', 'Type of Travel', 'Class', 'Customer Type', 'satisfaction']
categorical_df = raw_data_clean[target_categorical_cols]
print(categorical_df.groupby('Gender')['satisfaction'].mean())
print(categorical_df.groupby('Type of Travel')['satisfaction'].mean())
print(categorical_df.groupby('Class')['satisfaction'].mean())
print(categorical_df.groupby('Customer Type')['satisfaction'].mean())<jupyter_output>Gender
Female    0.427371
Male      0.439475
Name: satisfaction, dtype: float64
Type of Travel
Business travel    0.582597
Personal Travel    0.101678
Name: satisfaction, dtype: float64
Class
Business    0.694251
Eco         0.186138
Eco Plus    0.246064
Name: satisfaction, dtype: float64
Customer Type
Loyal Customer       0.477291
disloyal Customer    0.236658
Name: satisfaction, dtype: float64<jupyter_text>2.5.2 Ordinal Features<jupyter_code>ordinal_cols = raw_data_clean.loc[:, 'Inflight wifi service': 'Cleanliness'].columns
raw_data_clean[ordinal_cols].describe().T<jupyter_output><empty_output><jupyter_text>2.5.2.1 Visualizing Ordinal Features Part 1<jupyter_code>%matplotlib inline
fig, ax = plt.subplots(2, 7, figsize=(16, 8))
fig.tight_layout()
k = 0
for i in range(2):
    for j in range(7):
        ax[i][j].set(ylim=(0, 5))
        sns.barplot(x = 'satisfaction', y = ordinal_cols[k], data = raw_data_clean, ax = ax[i][j], palette= 'Purples')        
        k += 1

fig, ax = plt.subplots(figsize=(16,16))
target_ordinal_cols = list(ordinal_cols)
target_ordinal_cols.append('satisfaction')
#print(target_ordinal_cols)
#print(type(list(ordinal_cols)))
sns.heatmap(raw_data_clean[target_ordinal_cols].corr(), annot = True)<jupyter_output><empty_output><jupyter_text>Analysing the results There are some interesting findings here -- Most of the predictor variables seem to have a positive impact with increase in the survey response scores. `Gate location` does not seem to have any positive relation with satisfaction. `Departure/Arrival time convenient` is negatively correlated with satisfaction. That doesn't make much sense and it would seem to be random chance that there is a skew towards the negative satisfaction class. We should also be cognizant of the fact that some of these variables will be highly correlated with each other. So we should run some correlations to ensure that multi-collinearity is not an issue. __Analysis of Correlations / Heatmap:__As you can see from the correlations, `Gate location` and `Departure/Arrival time convenient` have no correlation with customer satisfaction.<jupyter_code>#Scale the ordinal columns
ordinal_cols_df_scale = pd.DataFrame(scale(raw_data_clean[ordinal_cols]), columns = ordinal_cols)
ordinal_cols_df_scale.drop(columns = ['Gate location', 'Departure/Arrival time convenient'], axis = 1, inplace = True)

# VIF dataframe 
vif_data = pd.DataFrame() 
vif_data["feature"] = ordinal_cols_df_scale.columns 
  
# calculating VIF for each feature 
vif_data["VIF"] = [variance_inflation_factor(ordinal_cols_df_scale.values, i) 
                          for i in range(len(ordinal_cols_df_scale.columns))] 
  
display(vif_data)<jupyter_output><empty_output><jupyter_text>__Analysis of VIFs__Typically VIFs greater > 2 indicate high multi-collinearity. `Inflight entertainment`, `Ease of Online booking` and `Cleanliness` have VIFs greater than 2, so we should check the VIFs without these features.  2.5.2.2 Visualizing Ordinal Features Part 2<jupyter_code>pca = PCA(n_components=2)
pca.fit(ordinal_cols_df_scale)
loadings = pca.components_
num_pc = pca.n_features_
pc_list = ["PC"+str(i) for i in list(range(1, num_pc+1))]
loadings_df = pd.DataFrame.from_dict(dict(zip(pc_list, loadings)))
loadings_df['variable'] = ordinal_cols_df_scale.columns.values
loadings_df = loadings_df.set_index('variable')
df = loadings_df.reset_index()
plt.figure(figsize = (16,8))
_ = sns.scatterplot(x='PC1', y='PC2', data=loadings_df, hue = 'variable', legend=False, s=100)

plt.text(df.PC1[df.variable=='Inflight wifi service']-0.018, \
         df.PC2[df.variable=='Inflight wifi service']+0.03, "Inflight wifi service", color='red')

plt.text(df.PC1[df.variable=='Ease of Online booking']-0.03, \
         df.PC2[df.variable=='Ease of Online booking']-0.03, "Ease of Online booking", color='darkorange')

plt.text(df.PC1[df.variable=='Food and drink']-0.01, \
         df.PC2[df.variable=='Food and drink']-0.04, "Food and drink", color='#c28825')

plt.text(df.PC1[df.variable=='Online boarding']-0.01, \
         df.PC2[df.variable=='Online boarding']-0.04, "Online boarding", color='#808760')

plt.text(df.PC1[df.variable=='Seat comfort']-0.01, \
         df.PC2[df.variable=='Seat comfort']-0.04, "Seat comfort", color='#62cc16')

plt.text(df.PC1[df.variable=='Inflight entertainment']-0.01, \
         df.PC2[df.variable=='Inflight entertainment']-0.04, "Inflight entertainment", color='#3cd6b7')

plt.text(df.PC1[df.variable=='On-board service']-0.015, \
         df.PC2[df.variable=='On-board service']+0.02, "On-board service", color='#3cd6b7')

plt.text(df.PC1[df.variable=='Leg room service']-0.015, \
         df.PC2[df.variable=='Leg room service']+0.02, "Leg room service", color='#3cd6b7')

plt.text(df.PC1[df.variable=='Cleanliness']-0.025, \
         df.PC2[df.variable=='Cleanliness']-0.007, "Cleanliness", color='#ae3cb5')

plt.text(df.PC1[df.variable=='Baggage handling']-0.041, \
         df.PC2[df.variable=='Baggage handling']-0.015, "Baggage handling", color='#ae3cb5')

plt.text(df.PC1[df.variable=='Checkin service']-0.016, \
         df.PC2[df.variable=='Checkin service']+0.02, "Checkin service", color='#8b75bf')


plt.text(df.PC1[df.variable=='Inflight service']+0.007, \
         df.PC2[df.variable=='Inflight service']+0.002, "Inflight service", color='#124cde')

plt.text(df.PC1[df.variable=='Cleanliness']-0.025, \
         df.PC2[df.variable=='Cleanliness']-0.007, "Cleanliness", color='#ae3cb5')<jupyter_output><empty_output><jupyter_text>As you can see from the plot, `Baggage handling` and `On-board service` are close to each other. This makes sense. `Cleanliness`, `Seat Comfort` and `Food and Drink` appear to be close to each other<jupyter_code>ordinal_cols_df_scale.drop(columns = ['Inflight entertainment', 'Ease of Online booking', 'Cleanliness'], axis = 1, inplace = True)

# VIF dataframe 
vif_data = pd.DataFrame() 
vif_data["feature"] = ordinal_cols_df_scale.columns 
  
# calculating VIF for each feature 
vif_data["VIF"] = [variance_inflation_factor(ordinal_cols_df_scale.values, i) 
                          for i in range(len(ordinal_cols_df_scale.columns))] 
  
display(vif_data)<jupyter_output><empty_output><jupyter_text>__As you can see, by eliminating the 3 features, the VIFs have now reduced to <= 2, indicating that there is less multi-collinearity between the variables__  2.5.2.3 Summary of Ordinal Features Based on the Exploratory Data Analysis of the Ordinal Features, we've identified variables that may be redundant. We can still keep these features in and apply a Random Forest model:    1. Low / No Correlation with the target (satisfaction):         - Gate location        - Departure/Arrival time convenient            2. Redudant Features due to high multi-collinearity            - Inflight entertainment        - Ease of Online booking         - Cleanliness  2.5.3 Continuous Features<jupyter_code>continuous_cols = ['Age', 'Flight Distance', 'Departure Delay in Minutes', 'Arrival Delay in Minutes']
raw_data_clean[continuous_cols].describe().T
def ecdf(data):
    """Compute ECDF for a one-dimensional array of measurements."""
    # Number of data points: n
    n = len(data)

    # x-data for the ECDF: x
    x = np.sort(data)

    # y-data for the ECDF: y
    y = np.arange(1, n + 1) / n

    return x, y
mean = np.mean(raw_data_clean['Departure Delay in Minutes'])
samples = np.random.exponential(mean, size = 1000000)
x_0, y_0 = ecdf(raw_data_clean.loc[raw_data_clean.satisfaction == 0, 'Departure Delay in Minutes'])
x_1, y_1 = ecdf(raw_data_clean.loc[raw_data_clean.satisfaction == 1, 'Departure Delay in Minutes'])
x_theor, y_theor = ecdf(samples)
_ = plt.plot(x_theor, y_theor, linewidth = 3, color = 'blue')
_ = plt.plot(x_0, y_0, marker = '.', linestyle = 'none', alpha = 0.002, color = 'red')
_ = plt.plot(x_1, y_1, marker = '.', linestyle = 'none', alpha = 0.002, color = 'lightgreen')
_ = plt.xlabel('Departure Delay')
_ = plt.ylabel('CDF')
_ = plt.axis([0, 250, 0, 1])
plt.show()
mean = np.mean(raw_data_clean['Arrival Delay in Minutes'])
samples = np.random.exponential(mean, size = 1000000)
x_0, y_0 = ecdf(raw_data_clean.loc[raw_data_clean.satisfaction == 0, 'Arrival Delay in Minutes'])
x_1, y_1 = ecdf(raw_data_clean.loc[raw_data_clean.satisfaction == 1, 'Arrival Delay in Minutes'])
x_theor, y_theor = ecdf(samples)
_ = plt.plot(x_theor, y_theor, linewidth = 3, color = 'blue')
_ = plt.plot(x_0, y_0, marker = '.', linestyle = 'none', alpha = 0.002, color = 'red')
_ = plt.plot(x_1, y_1, marker = '.', linestyle = 'none', alpha = 0.002, color = 'lightgreen')
_ = plt.xlabel('Arrival Delay')
_ = plt.ylabel('CDF')
_ = plt.axis([0, 250, 0, 1])
plt.show()
fig, ax = plt.subplots(2, 2, figsize = (12, 12))
sns.histplot(raw_data_clean['Age'], ax = ax[0][0])
sns.histplot(raw_data_clean['Flight Distance'].values, ax = ax[0][1])
sns.histplot(np.log(raw_data_clean['Arrival Delay in Minutes'].values + 1), ax = ax[1][0])
sns.histplot(np.log(raw_data_clean['Departure Delay in Minutes'].values + 1), ax = ax[1][1])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12843344400845155, 'fraction_numerical': 0.04316329610624811, 'mean_word_length': 4.234202211690364, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 7, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '307761', 'license': 'FTL', 'n_tokens_mistral': 4933, 'n_tokens_neox': 4497, 'n_words': 1114, 'path': 'notebooks/02_EDA.ipynb', 'repo_name': 'neeldeorukhkar/predict_customer_sat', 'size': '13252'}