<jupyter_start><jupyter_text>Imbalanced Data AnalysisThe objective of this notebook is to see how much value can be extracted while keeping the imbalance of the datset intact.  **Loading packages...**<jupyter_code>import numpy as np
import math
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Scikit-learn for ML prediction and modelling

from sklearn.preprocessing import StandardScaler
import sklearn
from sklearn.metrics import mean_squared_error, f1_score, roc_curve, auc, roc_auc_score, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn import tree
from mpl_toolkits import mplot3d
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, LabelEncoder
from sklearn.linear_model import LinearRegression, LogisticRegression, LogisticRegressionCV
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, GridSearchCV, train_test_split, KFold, RandomizedSearchCV

from xgboost.sklearn import XGBRegressor, XGBClassifier
import xgboost as xgb

from os import path

import warnings
warnings.filterwarnings("ignore")

%matplotlib inline
feat_df=pd.read_csv("/content/drive/MyDrive/Colab Notebooks/Robinhood/features_data.csv")
eq_df=pd.read_csv("/content/drive/MyDrive/Colab Notebooks/Robinhood/equity_value_data.csv")<jupyter_output><empty_output><jupyter_text>**Part 1**: Determine the user classes and find out the percentage of both.  Exploratory Data Analysis (**EDA**) to determine the users who have churned.<jupyter_code>print(eq_df.shape, '\n')
eq_df.head()
eq_df.info()
eq_df.isna().sum()
eq_df.timestamp=pd.to_datetime(eq_df.timestamp, errors='coerce')
eq_df.nunique()<jupyter_output><empty_output><jupyter_text>Visualizing Equity Vs Days for a few users<jupyter_code>for i in range(2):
    user_eq_df=eq_df[eq_df['user_id']==eq_df[eq_df.columns[2]].unique()[i]]
    
    plt.figure(figsize=(10,5))
    
    ts=user_eq_df["close_equity"]
    time=user_eq_df["timestamp"]
    
    plt.xlabel('Date')
    plt.ylabel('Equity (Dollars)')
    plt.plot(ts, '-', label='User {0}'.format(i));
    plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>We determine the two classes of the users - **Churned** or **Not Churned**. --- To determine that, we make use of the condition that if the equity of a user is lower than $10 for 28 consecutive calendar days or longer.From preliminary analysis of the equity value time-series of each user, we find that the minimum recorded equity value across users is \$10 and the maximum is ~\$4,50,000. This means that whenever the equity vaue goes below $10, those data are excluded from the table. This gives us an oppotunity to simplify the detection of churned users by simply determining the length of the missing time-series data for each user. If the length is over 28+2 (considering the weekend), that user is churned.--- The missing days (gap_days) from the dataset can be determined as shown below.<jupyter_code>eq_df['gap_days'] = eq_df.timestamp.diff()
eq_df.head(10)
# The number of churned users is obtained by comparing the gap-days with a time-delta of 30 days.

churned_user_count=eq_df[eq_df['gap_days']>=pd.Timedelta("30 days")]['user_id'].nunique()
churned_user_ids=eq_df[eq_df['gap_days']>=pd.Timedelta("30 days")]['user_id'].unique()
print(len(churned_user_ids))

# Total users

total_users=eq_df['user_id'].nunique()
total_user_ids=eq_df['user_id'].unique()
print(len(total_user_ids))<jupyter_output>279
5584<jupyter_text>**1. What fraction of the users have churned?**<jupyter_code>frac=np.round(100.0*(churned_user_count/total_users), decimals=2)
print('Fraction of churned users is {0}%'.format(frac))<jupyter_output>Fraction of churned users is 5.0%<jupyter_text>**Part 2: EDA and ML Model bulding to predict customer churn**<jupyter_code># The feature dataset
feat_df.head()
feat_df.columns
feat_df.info()
# Create a datset containing the class label(churn=1, no churn=0) by merging the churned user data with the 'feat_df'

churned_df=pd.DataFrame(churned_user_ids, columns=['user_id'])
churned_df['churn']=1
churned_df.head()
churned_df.info()
model_df=feat_df.merge(churned_df, how='left', on='user_id').fillna(0)
model_df['churn']=model_df['churn'].astype(int)
model_df=model_df.drop(['user_id'], axis=1)
model_df.head()
model_df.info()
cat_df = model_df.select_dtypes(include=['object']).copy()

for i in cat_df.columns:
  print(i, ' : ',cat_df[i].unique())
cat_df = model_df.select_dtypes(include=['object']).copy()

# Creating the categories

risk_categories=['low_risk_tolerance', 'med_risk_tolerance', 'high_risk_tolerance']
exp_categories=['no_investment_exp', 'limited_investment_exp', 'good_investment_exp', 'extensive_investment_exp']
liq_categories=['not_important_liq_need', 'somewhat_important_liq_need', 'very_important_liq_need' ]
platform_categories=['Android', 'iOS', 'both']
instrument_categories= ['stock', 'etp', 'adr', 'wrt', 'mlp', '0', 'lp', 'rlt', 'reit', 'cef', 'tracking']
horizon_categories=['short_time_horizon', 'med_time_horizon', 'long_time_horizon']

# Encoding the columns

enc_make = OrdinalEncoder(categories=[risk_categories, exp_categories, liq_categories, platform_categories, instrument_categories, horizon_categories])

cat_df_transformed = enc_make.fit_transform(cat_df[['risk_tolerance', 'investment_experience', 'liquidity_needs', 'platform', 
                                                    'instrument_type_first_traded','time_horizon']])

for i,j in enumerate(cat_df.columns):
  cat_df[j] = cat_df_transformed.transpose()[i]

# Adding converted labels to df
for i in model_df.columns:
  if i in cat_df.columns:
    model_df[i] = cat_df[i]

for i in cat_df.columns:
  print(i, ' : ',cat_df[i].unique())
model_df.head()
model_df.describe()
# Outlier detection using seaborn boxplot

model_df=model_df[(model_df['time_spent']<80) & (model_df['first_deposit_amount']<1000)] # Setting some realistic outlier bounds

sns.set_theme(style="whitegrid")
fig, axes = plt.subplots(2, 4, figsize=(20, 12))

count=0
for i in range(2):
    for j in range(4):
        sns.boxplot(ax=axes[i, j],data=model_df, x='churn', y=model_df.columns[count])
        axes[i,j].set_title(model_df.columns[count])
        count+=1<jupyter_output><empty_output><jupyter_text>**Scaling of Data**<jupyter_code># Scaling of the non-objects

from sklearn.preprocessing import StandardScaler, RobustScaler,MinMaxScaler

mm_scaler = MinMaxScaler()
std_scaler = StandardScaler()
rob_scaler = RobustScaler()

scaled_df=model_df.copy()
# We will use Robust scaler because it's ideal if there are outliers
for i in scaled_df.columns[:-1]: # -1 for not to scale the labels
  scaled_df[i] = mm_scaler.fit_transform(model_df[i].values.reshape(-1,1))

scaled_df.describe()
fig, axes = plt.subplots(2, 3, figsize=(18, 12))

sns.histplot(data=scaled_df, x="time_spent", color='purple', hue='churn', ax=
axes[0,0])
axes[0,0].set_xlabel('Time Spent')
axes[0,0].set_ylabel('Number of Users')
# axes[0,0].legend(loc=7, fontsize='xx-large' )
axes[0,0].set_title('Time Spent by Users', fontsize='large')
axes[0,0].set_yscale('log')

sns.histplot(data=scaled_df, x="risk_tolerance", color='purple', hue='churn', ax=
axes[0,1])
axes[0,1].set_xlabel('Risk_Tolerance')
axes[0,1].set_ylabel('Number of Users')
axes[0,1].set_title('Risk Tolerance by Users', fontsize='large')
axes[0,1].set_yscale('log')

sns.histplot(data=scaled_df, x="investment_experience", color='purple', hue='churn', ax=
axes[1,0])
axes[1,0].set_xlabel('Investment Experience')
axes[1,0].set_ylabel('Number of Users')
axes[1,0].set_title('Investment Experience by Users', fontsize='large')
axes[1,0].set_yscale('log')

sns.histplot(data=scaled_df, x="liquidity_needs", color='purple', hue='churn', ax=
axes[1,1])
axes[1,1].set_xlabel('Liquidity Needs')
axes[1,1].set_ylabel('Number of Users')
axes[1,1].set_title('Liquidity Needs by Users', fontsize='large')
axes[1,1].set_yscale('log')

sns.histplot(data=scaled_df, x="time_horizon", color='purple', hue='churn', ax=
axes[0,2])
axes[0,2].set_xlabel('Time Horizon')
axes[0,2].set_ylabel('Number of Users')
axes[0,2].set_title('Time Horizon by Users', fontsize='large')
axes[0,2].set_yscale('log')

sns.histplot(data=scaled_df, x="first_deposit_amount", color='purple', hue='churn', ax=
axes[1,2])
axes[1,2].set_xlabel('First Deposit Amount')
axes[1,2].set_ylabel('Number of Users')
axes[1,2].set_title('First Deposit Amount by Users', fontsize='large')
axes[1,2].set_yscale('log')

fig.subplots_adjust(wspace=0.3, hspace = 0.3)
plt.show()
# Class Imbalance

print('Churn events fraction in the data :',len(model_df[model_df['churn']==1])/len(model_df)*100)
print('Non-churn events fraction in the data :',len(model_df[model_df['churn']==0])/len(model_df)*100)

sns.set_theme(style="darkgrid")
g=sns.countplot(data=model_df, x='churn')

plt.title('Class Distributions \n (0: No Churn | 1: Churn)', fontsize=14)
plt.show()

# Class imbalance needs to be taken care before applying model
# Splitting the data

X = scaled_df.drop(columns = ['churn'])
y = scaled_df['churn']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

print('X shape -', X_train.shape)
print('y shape -', y_train.shape)<jupyter_output>X shape - (2928, 8)
y shape - (2928,)<jupyter_text>Mutual Information to see feature importance<jupyter_code>yy=sklearn.feature_selection.mutual_info_classif(X, y, discrete_features='auto', n_neighbors=20, copy=True, random_state=42)
xx=np.arange(len(yy))
plt.bar(xx, yy, color='black')
# plt.yscale('log')
plt.show()<jupyter_output><empty_output><jupyter_text>Feature selection using Chi-square for original, imbalanced data<jupyter_code># Chi squared feature selection for categorical data
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

# feature selection
def select_features(X_train, y_train, X_test):
	fs = SelectKBest(score_func=chi2, k='all')
	fs.fit(X_train, y_train)
	X_train_fs = fs.transform(X_train)
	X_test_fs = fs.transform(X_test)
	return X_train_fs, X_test_fs, fs

# feature selection
X_train_fs, X_test_fs, fs = select_features(X_train, y_train, X_test)

# what are scores for the features
for i in range(len(fs.scores_)):
	print('Feature %d: %f' % (i, fs.scores_[i]))
 
# plot the scores
plt.bar([i for i in range(len(fs.scores_))], fs.scores_)
plt.yscale('log')
plt.show()<jupyter_output>Feature 0: 4.295383
Feature 1: 0.054584
Feature 2: 0.171005
Feature 3: 0.143262
Feature 4: 12.097400
Feature 5: 0.103881
Feature 6: 1638.272402
Feature 7: 1.845807<jupyter_text>Random Forest with a single validation data<jupyter_code>from sklearn import tree
from sklearn.metrics import f1_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import precision_recall_fscore_support
from sklearn.metrics import r2_score
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=200)
clf = clf.fit(X_train_fs, y_train)

print("Accuracy (in %):", clf.score(X_test_fs, y_test)*100)

# The class determining probability was changed to 0.15 from 0.5 due to imbalance
y_pred = (clf.predict_proba(X_test_fs)[:,1] >= 0.15).astype(bool)
print('F1 score', f1_score(y_test, y_pred))

print('AUC Score = ', roc_auc_score(y_test, clf.predict_proba(X_test_fs)[:, 1]))<jupyter_output>Accuracy (in %): 94.5859872611465
F1 score 0.09794628751974724
AUC Score =  0.5561373539314717<jupyter_text>Poor performance from imbalanced data. No surprises here.Let's check the feature importance for the classifier<jupyter_code># Plot the feature importance

feat_scores = pd.DataFrame({'Fraction of Samples Affected' : clf.feature_importances_}, index=model_df.columns[:-1])
feat_scores = feat_scores.sort_values(by='Fraction of Samples Affected')
feat_scores.plot(kind='barh', figsize=(10,10), colormap='RdGy')
sns.despine()<jupyter_output><empty_output><jupyter_text>K-Fold Cross Validation using Decision Tree<jupyter_code>from sklearn.metrics import mean_squared_error
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import KFold
from sklearn.linear_model import LinearRegression, LogisticRegression

df=model_df

FEATURES=list(df.columns[0:8]) 
TARGET='churn'
f1score=[]
skf = StratifiedKFold(n_splits=3, shuffle=True, random_state=None)

for fold, (train_idx, valid_idx) in enumerate(skf.split(df[FEATURES], df[TARGET])):
    X_train, X_valid = df.iloc[train_idx], df.iloc[valid_idx]
    y_train = X_train[TARGET]
    y_valid = X_valid[TARGET]
    X_train = X_train.drop(TARGET, axis=1)
    X_valid = X_valid.drop(TARGET, axis=1)
    
    cbr = tree.DecisionTreeClassifier()

    cbr =  cbr.fit(X_train, y_train)
    y_pred = (cbr.predict_proba(X_valid)[:,1] >= 0.3).astype(bool)
    f1score.append(f1_score(y_valid, y_pred))
    print(f'Fold {fold}: F1: ', f1_score(y_valid, y_pred))<jupyter_output>Fold 0: F1:  0.03703703703703704
Fold 1: F1:  0.10062893081761007
Fold 2: F1:  0.11627906976744187<jupyter_text>Hyper-parameter tuning using grid-search CV<jupyter_code>from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
param_grid = {"max_depth": np.arange(2, 4, 6)}
parameters = {'n_estimators': [100, 200], 
              # 'max_features': ['log2', 'sqrt','auto'], 
              'criterion': ['entropy', 'gini'],
              # 'max_depth': [2, 3, 5, 10], 
              # 'min_samples_split': [2, 3, 5],
              # 'min_samples_leaf': [1,5,8]
             }

cbr = RandomForestClassifier()
clf = GridSearchCV(cbr, parameters, scoring='f1')
clf.fit(X_train, y_train)
pd.DataFrame(clf.cv_results_)<jupyter_output><empty_output><jupyter_text>Boosted Algorithms?<jupyter_code>from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import GradientBoostingClassifier

clf = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=2, random_state=42).fit(X_train, y_train)

print("Accuracy (in %):",clf.score(X_test, y_test)*100)

y_pred = (clf.predict_proba(X_test)[:,1] >= 0.15).astype(bool)
print('F1 score', f1_score(y_test, y_pred))

plot_confusion_matrix(clf, X_test, y_test)  
print(precision_recall_fscore_support(y_test, y_pred, average='binary'))


plt.show()
Q4: write code to trim outliers (defined as >/< mean+/-2std) in a df that has ID, 
50 columns of numeric variables (col1, col2, ... col50), 
and one target variable target 1/0

for i in df.columns:
    df=df[df>(df.mean() - 2*df.std()) & df<(df.mean() + 2*df.std())]

for i in df.index:
    df.loc[i]=df.loc[i][df.loc[i]>(df.mean()-2*df.std()) & df.loc[i]<(df.mean()+2*df.std())]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13595925297113753, 'fraction_numerical': 0.030831918505942274, 'mean_word_length': 6.6103359173126615, 'pattern_counts': {'":': 1, '<': 56, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '182093', 'license': 'MIT', 'n_tokens_mistral': 5781, 'n_tokens_neox': 5385, 'n_words': 1174, 'path': 'customer_churn_imb.ipynb', 'repo_name': 'Yadukrishnan1/Customer-Churn-Analysis', 'size': '14725'}