<jupyter_start><jupyter_text>Exploratory Data Analysis  Outline1. Import packages2. Load data3. Examine data4. Preprocess data4. Visualize data5. Investigate hypothesis   1. Import packages<jupyter_code>#imports

import pandas as pd

#for visualization
from matplotlib import pyplot as plt
%matplotlib inline
import seaborn as sns
import matplotlib.pylab as pylab

#set display options
pd.set_option('display.max_columns', None)

#set plot theme and parameters
sns.set_theme(style = "darkgrid")
params = {'legend.fontsize': 'x-large',
          'figure.figsize': (8, 6),
         'axes.labelsize': 'x-large',
         'axes.titlesize':'x-large',
         'xtick.labelsize':'x-large',
         'ytick.labelsize':'x-large'}
pylab.rcParams.update(params)<jupyter_output><empty_output><jupyter_text>2. Load data<jupyter_code>#read in data 

client_date_cols = ['date_activ', 'date_end', 'date_modif_prod', 'date_renewal']
client_data = pd.read_csv("../data/raw/client_data.csv", parse_dates = client_date_cols)
price_data = pd.read_csv("../data/raw/price_data.csv", parse_dates = ['price_date'])<jupyter_output><empty_output><jupyter_text>3. Examine data<jupyter_code>client_data.head()
client_data.info()
client_data.describe()
price_data.head()
price_data.describe()
price_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 193002 entries, 0 to 193001
Data columns (total 8 columns):
 #   Column              Non-Null Count   Dtype         
---  ------              --------------   -----         
 0   id                  193002 non-null  object        
 1   price_date          193002 non-null  datetime64[ns]
 2   price_off_peak_var  193002 non-null  float64       
 3   price_peak_var      193002 non-null  float64       
 4   price_mid_peak_var  193002 non-null  float64       
 5   price_off_peak_fix  193002 non-null  float64       
 6   price_peak_fix      193002 non-null  float64       
 7   price_mid_peak_fix  193002 non-null  float64       
dtypes: datetime64[ns](1), float64(6), object(1)
memory usage: 11.8+ MB<jupyter_text>Pandas Profile<jupyter_code># from pandas_profiling import ProfileReport
# prof = ProfileReport(client_data, minimal = True)
# prof.to_file(output_file = 'client_data_prof.html')
# prof = ProfileReport(price_data)
# prof.to_file(output_file = 'price_data_prof.html')<jupyter_output><empty_output><jupyter_text>Missing Data<jupyter_code>#number of rows with missing channel_sales
client_data.loc[client_data['channel_sales'] == "MISSING"].shape
#number of rows with missing origin_ups
client_data.loc[client_data['origin_up'] == "MISSING"].shape
#check number of companies in client data
nid_price = price_data["id"].nunique()
nid_client = client_data["id"].nunique()

print(f"Note: There is{((nid_price - nid_client) / nid_price): .2%} of clients in the price_data dataset of whom we do not have client data for.")<jupyter_output>Note: There is 9.26% of clients in the price_data dataset of whom we do not have client data for.<jupyter_text>4. Preprocess data<jupyter_code>#set target column to categorical
client_data = client_data.astype({"churn" : 'category'})
client_data.churn.dtypes<jupyter_output><empty_output><jupyter_text>Merge Data<jupyter_code>joined_data = pd.merge(price_data, client_data, on = 'id', how = 'left')
joined_data.head()
joined_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 193002 entries, 0 to 193001
Data columns (total 33 columns):
 #   Column                          Non-Null Count   Dtype         
---  ------                          --------------   -----         
 0   id                              193002 non-null  object        
 1   price_date                      193002 non-null  datetime64[ns]
 2   price_off_peak_var              193002 non-null  float64       
 3   price_peak_var                  193002 non-null  float64       
 4   price_mid_peak_var              193002 non-null  float64       
 5   price_off_peak_fix              193002 non-null  float64       
 6   price_peak_fix                  193002 non-null  float64       
 7   price_mid_peak_fix              193002 non-null  float64       
 8   channel_sales                   175149 non-null  object        
 9   cons_12m                        175149 non-null  float64       
 10  cons_gas_12m                    175149 non-null  float64[...]<jupyter_text>Define Columns<jupyter_code># define columns

TARGET = "churn"
price_cols = list(set(price_data.columns.tolist()) - set(["price_date", "id"]))
price_cols<jupyter_output><empty_output><jupyter_text>5. Visualize data   Visualize Target Variable<jupyter_code>client_data[TARGET].value_counts()
client_data[TARGET].value_counts(normalize = True)
ax = sns.countplot(x = "churn", data = client_data, palette = "tab10")
ax.bar_label(ax.containers[0], label_type = 'center')
plt.title("Count plot of customers")
plt.xlabel("Churn")
plt.ylabel("Number of customers", );<jupyter_output><empty_output><jupyter_text>About 10% of the customers have churned.   Visualize Target Variable by Categories<jupyter_code>client_data.has_gas.value_counts()
client_data.has_gas.value_counts(normalize = True)
client_data.groupby("has_gas").churn.value_counts(normalize = True)<jupyter_output><empty_output><jupyter_text>We notice a slight different between the churn rates of customers who are also gas clients and customers who are not.<jupyter_code>ax = sns.countplot(x = "has_gas", data = client_data, hue = "churn")
ax.bar_label(ax.containers[0], label_type = 'center')
plt.title("Count plot of customers")
plt.xlabel("Gas client")
plt.ylabel("Number of customers");<jupyter_output><empty_output><jupyter_text>6. Investigate hypothesis Here, we use data visualization to verify the hypothesis of price sensitivity being to some extent correlated with churn.<jupyter_code>fig, axes = plt.subplots(nrows=3, ncols=2, figsize=(18, 15))

axes = axes.ravel()

for col, ax in zip(price_cols, axes):
    
    sns.lineplot(x = 'price_date', 
                 y = col,
                 data = joined_data,
                 hue = 'churn',
                 palette = 'tab10',
                 ax = ax)
    ax.set_title(col.replace("_", " "))
    ax.set_ylabel("")
    ax.set_xlabel("")
                 
plt.tight_layout();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11072223989818644, 'fraction_numerical': 0.03865733375755648, 'mean_word_length': 3.3032169746748803, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '134980', 'license': 'MIT', 'n_tokens_mistral': 2264, 'n_tokens_neox': 2043, 'n_words': 463, 'path': 'notebooks/eda.ipynb', 'repo_name': 'ciciecho-ds/bcg_vi_customer_churn', 'size': '6286'}