<jupyter_start><jupyter_text>Survival Analysis[ Additional Reading If Needed](https://www.ics.uci.edu/~dgillen/STAT255/Handouts/lecture10.pdf)![My normal approach is useless here, too.](https://imgs.xkcd.com/comics/probability.png)https://xkcd.com/881/The aim of survival analysis is to analyze the effect of different risk factors and use them to predict the duration of time between one event ("birth") and another ("death").  Customer ChurnTreselle Systems, a data consulting service, [analyzed customer churn data using logistic regression](http://www.treselle.com/blog/customer-churn-logistic-regression-with-r/). For simply modeling whether or not a customer left this can work, but if we want to model the actual tenure of a customer, survival analysis is more appropriate.The "tenure" feature represents the duration that a given customer has been with them, and "churn" represents whether or not that customer left (i.e. the "event", from a survival analysis perspective). So, any situation where churn is "no" means that a customer is still active, and so from a survival analysis perspective the observation is censored (we have their tenure up to now, but we don't know their *true* duration until event).I have  [used their data](https://github.com/treselle-systems/customer_churn_analysis) to fit a survival model, and answered  the following questions:- What features best model customer churn?- What would you characterize as the "warning signs" that a customer may discontinue service?- What actions would you recommend to this business to try to improve their customer retention?I've created a few plots and visualizations to support my findings, and in general wrote my  summary/results targeting an "interested layperson" (e.g. your hypothetical business manager) as the audience. Enjoy! :)<jupyter_code># Loading the data to get you started
churn_data = pd.read_csv(
    'https://raw.githubusercontent.com/treselle-systems/'
    'customer_churn_analysis/master/WA_Fn-UseC_-Telco-Customer-Churn.csv')
churn_data.head()
#churn_data.info()  # A lot of these are "object" - some may need to be fixed...<jupyter_output><empty_output><jupyter_text>Clean the Data<jupyter_code>#@title
# Some columns we just don't need.
df = churn_data.drop(columns = 'customerID')

# The two numerical columns at the end should really be numerical
df['MonthlyCharges'] = pd.to_numeric(df['MonthlyCharges'])
df['TotalCharges'] = pd.to_numeric(df['TotalCharges'], 
                                       errors='coerce')

# There are 11 nulls in TotalCharges, which I'll replace with the mean
# value for that column
TotalCharges_mean = df['TotalCharges'].mean()
df['TotalCharges'].fillna(TotalCharges_mean, inplace=True)

# Lifelines requires numerical yes and no values.
#df = df.replace({'Yes':1, 'No':0})<jupyter_output><empty_output><jupyter_text>Encode these Vars!<jupyter_code>#@title
from sklearn.preprocessing import LabelEncoder
# Split categorical and numeric cols
numerical_cols = ['tenure','MonthlyCharges','TotalCharges']
df_nums = df[numerical_cols]
df_cats = df.drop(columns=numerical_cols)

# Label Encode the Categorica Data

df_labeled = df_cats.apply(LabelEncoder().fit_transform)

# Join the numeric and labeled data. 
df_mint = df_nums.join(df_labeled)
#print(df_mint.shape)
df_mint.head()<jupyter_output><empty_output><jupyter_text>Quick overview of the customer retention or "Lifetime" using `lifelines`The redlines indicate customers who have discontinued the service.The blue lines show the customers who were still customers and their lifespan as of the time of sampling.<jupyter_code>#@title
# Plot a random sample of customer retention. 
df_sort = df_mint.sort_values(['Churn', 'tenure'], ascending=[True, False])
df_sample = df_sort.sample(80)

time = df_sample.tenure.values
event = df_sample.Churn.values

ax = lifelines.plotting.plot_lifetimes(time, event_observed=event)
ax.set_xlim(0, 72)
ax.grid(axis='x')
ax.set_xlabel("Time in Months")
ax.set_title("Customer Retention");
plt.plot();<jupyter_output><empty_output><jupyter_text>Now for the KMF Survival Estimate CurveThis plot shows the normal loss of customers over the customer tenure starting at 100% and dropping over time. We will investigate the various factors that may be impacting this.<jupyter_code>#@title
time = df_mint.tenure.values;
event = df_mint.Churn.values;

kmf = lifelines.KaplanMeierFitter();
kmf.fit(time, event_observed=event);
kmf.plot();
plt.title(f'Baseline time before losing customer in Months');<jupyter_output><empty_output><jupyter_text>Lets run a CPH Regression and make some recommendations! Wow! We got some great results with some very significant features! That's terriffic. Looks like Monthly Charges, Tech Support, Contract, and Payment Method might be interesting.<jupyter_code>#@title
# I use the Cox Proportional Hazards model for a survival regression
# The matrix has high collinearity, so the internet suggests 
# adding a penalizer term.  The rationale here is not entirely clear
cph = lifelines.CoxPHFitter(penalizer=0.01)
cph.fit(df_mint, 'tenure', event_col='Churn')
cph.print_summary()<jupyter_output><lifelines.CoxPHFitter: fitted with 7043 observations, 5174 censored>
      duration col = 'tenure'
         event col = 'Churn'
number of subjects = 7043
  number of events = 1869
    log-likelihood = -12679.32
  time fit was run = 2019-01-25 06:39:36 UTC

---
                  coef  exp(coef)  se(coef)      z      p  log(p)  lower 0.95  upper 0.95     
MonthlyCharges    0.06       1.06      0.00  27.12 <0.005 -371.18        0.06        0.07  ***
TotalCharges     -0.00       1.00      0.00 -38.96 <0.005    -inf       -0.00       -0.00  ***
gender           -0.04       0.96      0.05  -0.91   0.36   -1.02       -0.13        0.05     
SeniorCitizen     0.04       1.04      0.06   0.69   0.49   -0.71       -0.07        0.15     
Partner          -0.18       0.84      0.05  -3.20 <0.005   -6.57       -0.28       -0.07    *
Dependents       -0.09       0.91      0.07  -1.33   0.18   -1.70       -0.23        0.04     
PhoneService     -0.21       0.81      0.12  -1.81   0.07   -2.66       -[...]<jupyter_text>Lets look at the `Log(Hazard Ratio)` and see a bit more insight into which variables are significant. Wow, look at that contract residual plot!<jupyter_code>#@title
# I plot out the log(Hazard Ratio) for each variable, to show which variables
# are close to being significant.
fig, ax = plt.subplots(figsize=(10,10));
cph.plot(ax=ax);<jupyter_output><empty_output><jupyter_text>Lets look at a few of these features. I'll start with Contract Type**Analysis:** Contract type has a very significant correlation with contract value. **Recommendation:** Do everything you can to move more customers onto contract.<jupyter_code>#@title
print(df.Contract.value_counts())
print(df_mint.Contract.value_counts())

cph.plot_covariate_groups('Contract', [0,1,2]);
plt.ylabel("Likelihood of remaining a customer")
plt.xlabel("Time in Months")
plt.title("Survival Analysis of Customers by Contract Type");<jupyter_output>Month-to-month    3875
Two year          1695
One year          1473
Name: Contract, dtype: int64
0    3875
2    1695
1    1473
Name: Contract, dtype: int64<jupyter_text>Lets take a look at the Cumulative Hazard Curve for Contract as well<jupyter_code>#@title
# Define the function
naf = lifelines.NelsonAalenFitter()

# A seperate plot for each label. 
m2m = df_mint['Contract'].values == 0
naf.fit(df_mint.tenure[m2m], event_observed=df_mint.Churn[m2m], label="Month-2-Month")
ax = naf.plot(loc=slice(0, 20))

oneyr = df_mint['Contract'].values == 1
naf.fit(df_mint.tenure[oneyr], event_observed=df_mint.Churn[oneyr], label="One Year")
naf.plot(ax=ax, loc=slice(0, 20))

twoyr = df_mint['Contract'].values == 2
naf.fit(df_mint.tenure[twoyr], event_observed=df_mint.Churn[twoyr], label="Two Year")
naf.plot(ax=ax, loc=slice(0, 20))

plt.title("Cumulative hazard function of different contract types");
plt.xlabel("Time in Months");
plt.ylabel("lowest to highest hazard level");<jupyter_output><empty_output><jupyter_text>I'm going to take a look at another feature: Payment Method**Analysis:** These are interesting because the survivability analysis seem nominal, yet on further inspection the increased rate of loss of e-check users compounds quickly resulting in an NAF cumulative hazard function that shows a ton of seperation between payment methods. **Recommendation: ** I suggest moving more customers to automatic payment methods to prevent the loss of additional customers.<jupyter_code>#@title
print("Count of Customers by Type")
print(df.PaymentMethod.value_counts())
print(df_mint.PaymentMethod.value_counts())

cph.plot_covariate_groups('PaymentMethod', [0,1,2,3]);
plt.ylabel("Likelihood of remaining a customer");
plt.xlabel("Time in Months");
plt.title("Survival Analysis of Customers by Payment Method");
#@title
# Define the function
naf = lifelines.NelsonAalenFitter()

# A seperate plot for each label. 
echeck = df_mint['PaymentMethod'].values == 2
naf.fit(df_mint.tenure[echeck], event_observed=df_mint.Churn[echeck], label="E Check")
ax = naf.plot(loc=slice(0, 20))

mail = df_mint['PaymentMethod'].values == 3
naf.fit(df_mint.tenure[mail], event_observed=df_mint.Churn[mail], label="Mailed Check")
naf.plot(ax=ax, loc=slice(0, 20))

Bank = df_mint['PaymentMethod'].values == 0
naf.fit(df_mint.tenure[Bank], event_observed=df_mint.Churn[Bank], label="Automatic Bank Transfer")
naf.plot(ax=ax, loc=slice(0, 20))

CC = df_mint['PaymentMethod'].values == 1
naf.fit(df_mint.tenure[CC], event_observed=df_mint.Churn[CC], label="Credit Card (Automatic)")
naf.plot(ax=ax, loc=slice(0, 20))

plt.title("Cumulative hazard function of different payment methods");
plt.xlabel("Time in Months");
plt.ylabel("lowest to highest hazard level");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1083003149446307, 'fraction_numerical': 0.031088082901554404, 'mean_word_length': 4.580498866213152, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 40, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '202559', 'license': 'MIT', 'n_tokens_mistral': 3394, 'n_tokens_neox': 3141, 'n_words': 982, 'path': 'module2-survival-analysis/LS_DS_232_Survival_Analysis_Assignment.ipynb', 'repo_name': 'BrittonWinterrose/DS-Unit-2-Sprint-3-Advanced-Regression', 'size': '9843'}