<jupyter_start><jupyter_text>**Problem Statement**Improve sales revenue of the telco by predicting(classifiying) customer Churn/No Churn using the provided features.Accuracy must be the same or better than baseline but we 'gotta-catch'em all' the Churn customers.<jupyter_code># read csv and display 5 rows
df = pd.read_csv('data/churn.csv')
df.head()

# drop 'customerID' as it's a serialize number
df.drop(columns = 'customerID',inplace=True)
# Churn is our target variables, the rest are predictor variables
df.columns
# Convert to numeric
df['TotalCharges'] = pd.to_numeric(df['TotalCharges'],errors="coerce")
df.isnull().sum().sort_values(ascending=False)
#display all rows that are null
df[df.isna().any(axis=1)]
# We will drop the 11 values since it's only 0.15% of the DF
# and we are not able to determine the Total Charges
# and the target variable is 'No Churn'
df.dropna(inplace=True)
# verify there is no null
df.isnull().sum().sort_values(ascending=False)
# backup processed df
df.to_csv('data/churn_done.csv',index=False)
df.head()
g= sns.catplot(x="Churn", kind="count", data=df)
g.fig.suptitle('Valuecount of Churn and No Churn');
# 27% of the customers churn
# there is a chance that we may need to consider this an imbalanced dataset
# The baseline accuracy is 73%
df.Churn.value_counts(normalize=True).round(2)
df.info()
# 7032 rows with 19 features ( less target variable)
df.shape
# display all the unique values
print('col name \t dtype \t Unique Values')
print('=======================================')
for col in df.columns:
    print(f'{col} \t | { df[col].dtype} -> { df[col].unique()} \n')<jupyter_output>col name 	 dtype 	 Unique Values
=======================================
gender 	 | object -> ['Female' 'Male'] 

SeniorCitizen 	 | int64 -> [0 1] 

Partner 	 | object -> ['Yes' 'No'] 

Dependents 	 | object -> ['No' 'Yes'] 

tenure 	 | int64 -> [ 1 34  2 45  8 22 10 28 62 13 16 58 49 25 69 52 71 21 12 30 47 72 17 27
  5 46 11 70 63 43 15 60 18 66  9  3 31 50 64 56  7 42 35 48 29 65 38 68
 32 55 37 36 41  6  4 33 67 23 57 61 14 20 53 40 59 24 44 19 54 51 26 39] 

PhoneService 	 | object -> ['No' 'Yes'] 

MultipleLines 	 | object -> ['No phone service' 'No' 'Yes'] 

InternetService 	 | object -> ['DSL' 'Fiber optic' 'No'] 

OnlineSecurity 	 | object -> ['No' 'Yes' 'No internet service'] 

OnlineBackup 	 | object -> ['Yes' 'No' 'No internet service'] 

DeviceProtection 	 | object -> ['No' 'Yes' 'No internet service'] 

TechSupport 	 | object -> ['No' 'Yes' 'No internet service'] 

StreamingTV 	 | object -> ['No' 'Yes' 'No internet service'] 

StreamingMovies 	 | object -> ['No' 'Yes' 'No[...]<jupyter_text>|   | Column           | Old D-type | New D-type | Change ? ||----|------------------|------------|------------|----------|| 0  | gender           | object     | int64      | No       || 1  | SeniorCitizen    | int64      | int64      | No       || 2  | Partner          | object     | int64      | Yes      || 3  | Dependents       | object     | int64      | Yes      || 4  | tenure           | int64      | int64      | No       || 5  | PhoneService     | object     | int64      | Yes      || 6  | MultipleLines    | object     | object     | No       || 7  | InternetService  | object     | object     | No       || 8  | OnlineSecurity   | object     | object     | No       || 9  | OnlineBackup     | object     | object     | No       || 10 | DeviceProtection | object     | object     | No       || 11 | TechSupport      | object     | object     | No       || 12 | StreamingTV      | object     | object     | No       || 13 | StreamingMovies  | object     | object     | No       || 14 | Contract         | object     | object     | No       || 15 | PaperlessBilling | object     | int64      | Yes      || 16 | PaymentMethod    | object     | object     | No       || 17 | MonthlyCharges   | float64    | float64    | No       || 18 | TotalCharges     | object     | float64    | Yes      || 19 | Churn            | object     | int64      | Yes      | There are 3 types of data:* Numerical - continuous values - 10.1cm , 5.2 seconds etc * Categorical ( Nominal - there is no order i.e blue, green , orange)* Categorical ( Ordinal - there is order i.e small, medium , large )<jupyter_code>binary_columns = ['Partner','Dependents','PhoneService','PaperlessBilling','Churn']
from sklearn.preprocessing import LabelEncoder

label = LabelEncoder()
#label.fit(df['Partner'])

def convert_to_binary(binary_columns):
    '''convert a list of columns to binary'''
    for col in binary_columns:
        #df[col]= label.transform(df[col])
        df[col]= label.fit_transform(df[col])
        
convert_to_binary(binary_columns)
# ['Partner','Dependents','PhoneService','PaperlessBilling','Churn'] already binarized
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 7032 entries, 0 to 7042
Data columns (total 20 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   gender            7032 non-null   object 
 1   SeniorCitizen     7032 non-null   int64  
 2   Partner           7032 non-null   int32  
 3   Dependents        7032 non-null   int32  
 4   tenure            7032 non-null   int64  
 5   PhoneService      7032 non-null   int32  
 6   MultipleLines     7032 non-null   object 
 7   InternetService   7032 non-null   object 
 8   OnlineSecurity    7032 non-null   object 
 9   OnlineBackup      7032 non-null   object 
 10  DeviceProtection  7032 non-null   object 
 11  TechSupport       7032 non-null   object 
 12  StreamingTV       7032 non-null   object 
 13  StreamingMovies   7032 non-null   object 
 14  Contract          7032 non-null   object 
 15  PaperlessBilling  7032 non-null   int32  
 16  PaymentMethod     7032 non-null   object 
[...]<jupyter_text>Numerical features<jupyter_code># Plot all numerical fields using "Churn" as hue
g = sns.pairplot(df, hue="Churn",vars=['MonthlyCharges','TotalCharges','tenure'],aspect=1.5);<jupyter_output><empty_output><jupyter_text>General feel* Most churn occurs when the customers are new ( low tenure ), higher Total Charges (~ `$`820) and higher Monthly Charges (~ `$`120) * Note : Orange is Churn  Categorical features This dataset has 16 categorical features:* Five binary features (Yes/No)* Nine features with three unique values * One feature with four unique values  Age(SeniorCitizen) and Gender<jupyter_code>g = sns.catplot(x="gender", hue='Churn',kind="count", data=df)
g.fig.suptitle('Churn based on Gender');
df.groupby(['SeniorCitizen'])['Churn'].value_counts(normalize=True).round(2)
df.groupby(['SeniorCitizen'])['Churn'].value_counts()
df['SeniorCitizen'].value_counts()
sns.catplot(x="SeniorCitizen", hue='Churn',kind="count", data=df)
g.fig.suptitle('Churn based on Age - SeniorCitizen');<jupyter_output><empty_output><jupyter_text>Senior Citizen churn % rate ( 42% vs 24% ) . Meaning almost 50% chance that a Senior Citizen Subsciber will churn eventhough the total count subscriber ratio is significantly lesser - 1 Senior vs 6 Non Senior * Not much effect between gender* Most churn occurs around `$`30, `$`50 and `$`70  Partner and Dependents<jupyter_code>pd.concat([
           df.groupby(['Partner'])['Churn'].value_counts().round().to_frame(),
           df.groupby(['Partner'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)
g = sns.catplot(x="Partner", hue='Churn',kind="count", data=df)
g.fig.suptitle('Churn based on hasPartner');<jupyter_output><empty_output><jupyter_text>If there is no Partner, the tendency to Churn is higher. Dual income has lesser chance to churn<jupyter_code>pd.concat([
           df.groupby(['Dependents'])['Churn'].value_counts().round().to_frame(),
           df.groupby(['Dependents'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)
g = sns.catplot(x="Dependents", hue='Churn',kind="count", data=df)
g.fig.suptitle('Churn based on hasDependents');<jupyter_output><empty_output><jupyter_text>If there is no Dependents, the tendency to Churn is higher. It could be adults that has Dependents are more responsible.  Multiple lines<jupyter_code>pd.concat([
           df.groupby(['MultipleLines'])['Churn'].value_counts().round().to_frame(),
           df.groupby(['MultipleLines'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)
g = sns.catplot(x="MultipleLines", hue='Churn',kind="count",data=df)
g.fig.suptitle('Churn based on MultipleLines');<jupyter_output><empty_output><jupyter_text>Multiple Lines doesn't seem to be a factor that contributes signifantly to the churn rate as all the 3 percentage are similar at around 25% and 'with MultiplineLines' registered the highest with 29%<jupyter_code>g = sns.catplot(data=df,x="MultipleLines", y="MonthlyCharges", hue="Churn", kind="violin",split=True)
g.fig.suptitle('Violin plot distributin of MonthlyCharges for Multiplelines');<jupyter_output><empty_output><jupyter_text>Between the MultipleLines , the most occurence for each of the category are * 'No phone service' ~`$`25* 'No MultipleLines' ~`$`75* 'With MultipleLines' ~`$`100  Phone Services<jupyter_code>pd.concat([
           df.groupby(['PhoneService'])['Churn'].value_counts().round().to_frame(),
           df.groupby(['PhoneService'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)
g = sns.catplot(x="PhoneService", hue='Churn',kind="count", data=df)
plt.subplots_adjust(top=0.9)
g.fig.suptitle('Churn based on PhoneService');<jupyter_output><empty_output><jupyter_text>Phone Lines doesn't seem to be a factor that contributes signifantly to the churn rate as it's similar to MultipleLines at around 25%<jupyter_code>g = sns.catplot(data=df,x="PhoneService", y="MonthlyCharges", hue="Churn", kind="violin",split=True)
plt.subplots_adjust(top=0.9)
g = g.fig.suptitle('Violin plot distribution of MonthlyCharges for PhoneService');<jupyter_output><empty_output><jupyter_text>Most occurence of Churn occurs ~$70  InternetService<jupyter_code>pd.concat([
           df.groupby(['InternetService'])['Churn'].value_counts().round().to_frame(),
           df.groupby(['InternetService'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)<jupyter_output><empty_output><jupyter_text>If one has Fiber optic, the churn rate is high at around 42%.<jupyter_code>g = sns.catplot(x="InternetService", hue='Churn',kind="count", data=df)
plt.subplots_adjust(top=0.9)
g.fig.suptitle('Churn based on InternetService');
g = sns.catplot(data=df,x="InternetService", y="MonthlyCharges", hue="Churn", kind="violin",split=True)
plt.subplots_adjust(top=0.9)
g = g.fig.suptitle('Violin plot distribution of MonthlyCharges for InternetService');<jupyter_output><empty_output><jupyter_text>For Fiber optic, the MonthlyCharges range is between `$`70 and `$`100  PaymentMethod<jupyter_code>pd.concat([
           df.groupby(['PaymentMethod'])['Churn'].value_counts().round().to_frame(),
           df.groupby(['PaymentMethod'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)
df.groupby(['PaymentMethod'])['MonthlyCharges','TotalCharges'].agg(['median','mean']).round(0)<jupyter_output><empty_output><jupyter_text>Those we paid using Electronic check has the highest tendency to churn at 45% eventhough the TotalCharges are not the highestWe need to convert Electronic Check to other forms of payment. A one time rebate for conversion to entice them<jupyter_code>g = sns.catplot(x="PaymentMethod", hue='Churn',kind="count", data=df)
plt.subplots_adjust(top=0.9)
g.fig.suptitle('Churn based on PaymentMethod');
g = sns.catplot(data=df,x="PaymentMethod", y="MonthlyCharges", hue="Churn", kind="violin",split=True)
plt.subplots_adjust(top=0.9)
g = g.fig.suptitle('Violin plot distribution of MonthlyCharges for InternetService');<jupyter_output><empty_output><jupyter_text>Contract<jupyter_code>g = sns.catplot(x="Contract", hue='Churn',kind="count", data=df)
plt.subplots_adjust(top=0.9)
g.fig.suptitle('Churn based on Contract');
pd.concat([
           df.groupby(['Contract'])['Churn'].value_counts().round().to_frame(),
           df.groupby(['Contract'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)<jupyter_output><empty_output><jupyter_text>The shortest/no contract will tend to encourage Churn  PaperlessBilling<jupyter_code>g = sns.catplot(x="PaperlessBilling", hue='Churn',kind="count", data=df)
plt.subplots_adjust(top=0.9)
g.fig.suptitle('Churn based on PaperlessBilling');
pd.concat([
           df.groupby(['PaperlessBilling'])['Churn'].value_counts().round().to_frame(name='Churn Count'),
           df.groupby(['PaperlessBilling'])['Churn'].value_counts(normalize=True).round(2).to_frame(name='%'), 
           ],
          axis=1)<jupyter_output><empty_output><jupyter_text>The Paperless Billing will tend to encourage Churn, twice of non-paperless With this information, we need to remind customers via email or SMS<jupyter_code>## Additional subscriptions
sns.scatterplot(x='MonthlyCharges',y='TotalCharges',hue='Churn',data=df);
df.head()
import pandas as pd
df = pd.read_csv('data/churn.csv')
df.head()
df.columns
df.PaperlessBilling.unique()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14632144456051335, 'fraction_numerical': 0.03141322190717803, 'mean_word_length': 3.8755911240451075, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '666673', 'license': 'CNRI-Python', 'n_tokens_mistral': 4777, 'n_tokens_neox': 4276, 'n_words': 1104, 'path': '1_EDA.ipynb', 'repo_name': 'andrewng88/telco_churn', 'size': '13402'}