<jupyter_start><jupyter_text>What percentage of users have churned?<jupyter_code>churn = len(df[df['churn'] == 1])
percent_churn = round(churn / len(df) * 100, 2)
percent_churn<jupyter_output><empty_output><jupyter_text>Split Data<jupyter_code># Split Data
X = df.drop(columns='churn')
y = df['churn']
X.shape, y.shape
X.head()
# Train-Test-Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
df.head()<jupyter_output><empty_output><jupyter_text>OLS<jupyter_code>import statsmodels.api as sm
# X_cols = [col for col in df.drop(columns='churn').columns]
ohe = OneHotEncoder(use_cat_names=True)
ohe.fit(X_train)
XT_train = ohe.transform(X_train)
XT_test = ohe.transform(X_test)
X_columns = XT_train.columns.tolist()
def update_ols():
    X_ols = XT_train[X_columns]
    model = sm.OLS(y_train, X_ols).fit()
    print(model.summary())

update_ols()
# First iteration, drop statistically insignificant columns (p_val > 0.05)
remove_list = ['gender_Male', 'gender_Female', 'Partner_Yes', 'Partner_No', 'Dependents_Yes', 'Dependents_No',
              'PhoneService_Yes', 'PhoneService_No', 'MultipleLines_No', 'MultipleLines_Yes', 
               'MultipleLines_No phone service', 'InternetService_DSL', 'InternetService_No', 'InternetService_Fiber optic',
               'StreamingTV_Yes', 'StreamingTV_No', 'StreamingTV_No internet service', 'StreamingMovies_No internet service',
               'StreamingMovies_No', 'StreamingMovies_Yes', 'MonthlyCharges']

for feature in remove_list:
    X_columns.remove(feature)
    
update_ols()<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                  churn   R-squared:                       0.256
Model:                            OLS   Adj. R-squared:                  0.254
Method:                 Least Squares   F-statistic:                     137.7
Date:                Tue, 31 Aug 2021   Prob (F-statistic):               0.00
Time:                        16:35:48   Log-Likelihood:                -2536.7
No. Observations:                5625   AIC:                             5103.
Df Residuals:                    5610   BIC:                             5203.
Df Model:                          14                                         
Covariance Type:            nonrobust                                         
===========================================================================================================
                       [...]<jupyter_text>Dropping features that aren't statistically significant doesn't improve the fit of the OLS model. Instead of dimensionality reduction, I'm going to try fitting a Logistic Regression/Random Forest Classifier and use sklearn's Permutation Feature Importance to find the most influencing features on the accuracy of the model.   Logistic Regression<jupyter_code># Create sklearn pipeline to automate encoding, scaling, and training

model_lr = make_pipeline(
    OneHotEncoder(use_cat_names=True),
    StandardScaler(),
    LogisticRegression()
)

model_lr.fit(X_train, y_train)
# Churn probability
np.set_printoptions(suppress=True)
model_lr.predict_proba(X_test)
# Churn Predictions
model_lr.predict(X_test)
# Accuracy Score
model_lr.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Confusion Matrix, ROC Curve<jupyter_code>plot_confusion_matrix(model_lr, X_test, y_test, values_format='.0f',
                      display_labels=['No Churn', 'Churn'])
print(classification_report(y_test, model_lr.predict(X_test)))<jupyter_output>precision    recall  f1-score   support

         0.0       0.83      0.90      0.86      1018
         1.0       0.65      0.51      0.58       389

    accuracy                           0.79      1407
   macro avg       0.74      0.71      0.72      1407
weighted avg       0.78      0.79      0.78      1407<jupyter_text>ROC Curve<jupyter_code>lr = plot_roc_curve(model_lr, X_test, y_test)
# Classify each user as churned or not churned
classify_df = X_test.copy()
np.set_printoptions(suppress=True)
classify_df["Churn_Probability"] = model_lr.predict_proba(X_test)[:,1]
classify_df['Churn'] = model_lr.predict(X_test)
classify_df.head()
# Permutation Feature Importance

perm_imp = permutation_importance(
    model_lr, 
      X_test, 
      y_test, 
      n_repeats=5, 
      n_jobs=-1, 
      random_state=42
)
data = {'imp_mean': perm_imp['importances_mean'],
        'imp_std': perm_imp['importances_std']}

perm_df = pd.DataFrame(data, index=X_test.columns).sort_values('imp_mean')
perm_df['imp_mean'].tail(10).plot(kind='barh', title='Permutation Feature Importances')
perm_df<jupyter_output><empty_output><jupyter_text>Simplify LogReg Model<jupyter_code># Model 2: Simplified Model (Fewer features)
df2 = df[['tenure', 'InternetService', 'churn']]
df2.head()
# Split Data
X2 = df2.drop(columns='churn')
y2 = df2['churn']
# Train Test Split
X_train2, X_test2, y_train2, y_test2 = train_test_split(X2, y2, test_size=0.2)
# Create model pipeline
model_lr2 = make_pipeline(
    OneHotEncoder(use_cat_names=True),
    StandardScaler(),
    LogisticRegression()
)

model_lr2.fit(X_train2, y_train2)
# Model score
model_lr2.score(X_test2, y_test2)
# Permutation Feature Importances for new model
perm_imp2 = permutation_importance(
    model_lr2, 
      X_test2, 
      y_test2, 
      n_repeats=5, 
      n_jobs=-1, 
      random_state=42
)
data = {'imp_mean': perm_imp2['importances_mean'],
        'imp_std': perm_imp2['importances_std']}

perm_df2 = pd.DataFrame(data, index=X_test2.columns).sort_values('imp_mean')
perm_df2['imp_mean'].tail(10).plot(kind='barh', title='Permutation Feature Importances')<jupyter_output><empty_output><jupyter_text>Analysis:Model Performance ~79% accuracy. Precision and Recall are low for predicting churn, ~65% and ~60%, respectively. The most important features in predicting user churn are `tenure` and `InternetService` (Found using Permutation Feature Importances technique). Dropping columns that don't contribute to the model (features that contribute less than 1% increase to model performance) improved the accuracy of the model by simplifying it. If I wanted to improve the accuracy score further, I would tune the hyperparameters of Logistic Regression/Random Forest using Randomized Search to find important hyperparameters, then follow it up with brute force Grid Search on specific parameters to optimize further.For the purpose of this analysis, incremental accuracy improvements is not the goal. Understanding the patterns in the data and finding the most important features in correctly predicting churned users is the goal. That way these features can be focused on when diving deeper into the analysis, visualizing, and communicating these insights using Tableau.Next Step is to import this dataset into Tableau and focus on the top features (`tenure` and `InternetService`) that contribute to the accuracy of the model to try to find and visualize the relationships between these features and `Churn`. This will help us understand which users will likely churn in the future and how to create initiatives around decreasing churn and increasing the LTV of users.  Random Forest<jupyter_code>model_rf = make_pipeline(
    OneHotEncoder(),
    StandardScaler(),
    RandomForestClassifier(random_state=42)
)

model_rf.fit(X_train, y_train)
model_rf.predict_proba(X_test)
model_rf.predict(X_test)
model_rf.score(X_test, y_test)
# 2nd Random Forest Model
model_rf2 = make_pipeline(
    OrdinalEncoder(),
    StandardScaler(),
    RandomForestClassifier(random_state=42, n_jobs=-2)
)

model_rf2.fit(X_train2, y_train2)
model_rf2.score(X_test2, y_test2)<jupyter_output><empty_output><jupyter_text>Random EDA Practice<jupyter_code>df_gender = df.groupby(['gender']).mean()
df_gender<jupyter_output><empty_output><jupyter_text>isin()<jupyter_code>internet = ['Fiber optic', 'DSL']
df.loc[df['InternetService'].isin(internet)].head()<jupyter_output><empty_output><jupyter_text>Churn if tenure < 10 and  Contract = Month-to-month, One year and  Churn=1<jupyter_code>contract = ['Month-to-month', 'One year']
new_df = df[(df['tenure'] <= 10) & (df['Contract'].isin(contract)) & (df['churn'] == 1)]
new_df.head()<jupyter_output><empty_output><jupyter_text>Neural Network<jupyter_code>X_train.shape, y_train.shape
X_train.shape[1]
X_train.head()
ohe = OneHotEncoder(use_cat_names=True)

ohe.fit(X_train)

XT_train = ohe.transform(X_train)
XT_test = ohe.transform(X_test)
XT_train.head()
# Create NN Model

model = Sequential()

model.add(Dense(250, input_dim=XT_train.shape[1], activation='relu'))

model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(XT_train, y_train, validation_data=(XT_test, y_test), epochs=5)
model.summary()
_, train_acc = model.evaluate(XT_train, y_train)
_, test_acc = model.evaluate(XT_test, y_test)
model.predict(XT_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12053916694287924, 'fraction_numerical': 0.022207490884985085, 'mean_word_length': 4.0316842690383545, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '110511', 'license': 'MIT', 'n_tokens_mistral': 3062, 'n_tokens_neox': 2865, 'n_words': 685, 'path': 'customer_churn.ipynb', 'repo_name': 'andrewlee977/user_churn_analysis', 'size': '9051'}