<jupyter_start><jupyter_text>**Coxph trainer - Survival analysis** The following function provides both [Cox proprotional hazards modelling](https://en.wikipedia.org/wiki/Proportional_hazards_model)and [Kaplan-Meier](https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator) plots.  **Basics of the Cox proportional hazards model**The purpose of the model is to evaluate simultaneously the effect of several factors on survival. In other words, it allows us to examine how specified factors influence the rate of a particular event happening (e.g., infection, death) at a particular point in time. This rate is commonly referred as the hazard rate ([link](http://www.sthda.com/english/wiki/cox-proportional-hazards-model)). **Kaplan-Meier survival estimate**The Kaplan-Meier (KM) method is a non-parametric method used to estimate the survival probability from observed survival times (Kaplan and Meier, 1958)([link](http://www.sthda.com/english/wiki/survival-analysis-basics)).  **The function**train models to predict the timing of events.Although identical in structure to other training functions, this onerequires generating a 'Y' that represents the age/duration/tenure ofthe obervation, designated 'tenure' here, and a binary labels columns thatrepresents the event of interest, churned/not-churned.In addition, there is a strata_cols parameter, representing a list ofstratification (aka grouping) variables. The following example covers:- [Data exploration](Data-exploration)- [Training Cox proprotional hazards and Kaplan-Meier model](Training-Cox-proprotional-hazards-and-Kaplan-Meier-model)- - [Importing the function](Importing-the-function)- - [Setup function parameters](Setup-function-parameters)- - [Running the function locally](Running-the-function-locally)- [A peek at a pickled kaplan-meier model](A-peek-at-a-pickled-kaplan-meier-model)- [A peek at a pickeld cox hazards default model](A-peek-at-a-pickeld-cox-hazards-default-model)- [Some potential default analyses of coxph](Some-potential-default-analyses-of-coxph)- - [Running the function remotely](Running-the-function-remotely)We will train on [Telco Customer Churn dataset](https://www.kaggle.com/blastchar/telco-customer-churn) from kaggle, click the link for context.The dataset is transformed using [one-hot-encoding](https://en.wikipedia.org/wiki/One-hot), check out [customer-churn-prediction demo](https://github.com/mlrun/demos/tree/master/customer-churn-prediction) in the clean_data section for further information.<jupyter_code>import warnings
warnings.filterwarnings("ignore")
# Make sure the following libraries are installed 
# !pip install lifelines<jupyter_output><empty_output><jupyter_text>**Data exploration**<jupyter_code># Getting raw data as - downloaded from kaggle
import pandas as pd

df = pd.read_csv("https://s3.wasabisys.com/iguazio/data/function-marketplace-data/coxph_trainer/WA_Fn-UseC_-Telco-Customer-Churn.csv")
print('raw dataset')
df.head()
df = pd.read_csv("https://s3.wasabisys.com/iguazio/data/function-marketplace-data/coxph_trainer/encoded-data.csv")
print('encoded dataset')
df.head()<jupyter_output>encoded dataset<jupyter_text>**Training Cox proprotional hazards and Kaplan-Meier model**  **Importing the function**<jupyter_code>import mlrun
mlrun.set_environment(project='function-marketplace')

fn = mlrun.import_function("hub://coxph_trainer")
fn.image='mlrun/mlrun'
fn.apply(mlrun.auto_mount())<jupyter_output>> 2021-10-13 13:22:07,678 [info] loaded project function-marketplace from MLRun DB<jupyter_text>**Setup function parameters**<jupyter_code>task = mlrun.new_task(name = "tasks-survive-trainer",
                      params = {"event_column" : "labels", 
                                "strata_cols"  : ['InternetService', 'StreamingMovies', 'StreamingTV', 'PhoneService'],
                                "p_value"      : 0.005,
                                "encode_cols"  : {"Contract"       : "Contract",
                                                  "PaymentMethod"  : "Payment"},
                                "models_dest"  : 'models/cox',
                                "file_ext" : "csv"})<jupyter_output><empty_output><jupyter_text>**Running the function locally**<jupyter_code># running the function with the task provided
coxph_run = fn.run(task,
                   local=True,
                   inputs={"dataset"  : "https://s3.wasabisys.com/iguazio/data/function-marketplace-data/coxph_trainer/encoded-data.csv"})<jupyter_output>> 2021-10-13 13:18:36,975 [info] starting run tasks-survive-trainer uid=c525c7402c7e4188b0e22996e8fc682c DB=http://mlrun-api:8080<jupyter_text>**A peek at a pickled kaplan-meier model**<jupyter_code># loading the model trained
from mlrun.artifacts import get_model
import pickle
model_file, model_obj, _ = get_model(coxph_run.artifact('km-model'))
model = pickle.load(open(model_file,'rb'))
model.predict([1,10,30,100,200])
m = model.plot(figsize=(11,6))<jupyter_output><empty_output><jupyter_text>**A peek at a pickeld cox hazards default model**<jupyter_code># loading the model trained
from mlrun.artifacts import get_model
import pickle
model_file, model_obj, _ = get_model(coxph_run.artifact('cx-model'))
model = pickle.load(open(model_file,'rb'))
model.print_summary()<jupyter_output><empty_output><jupyter_text>**Some potential default analyses of coxph**<jupyter_code>model.baseline_survival_.plot()<jupyter_output><empty_output><jupyter_text>* run the following for each of the lines that passes some test (p  `model.plot_covariate_groups('Contract_1', values=[0, 1]);` the plot needs to have the strata decoded  In the train_model above, set param `plot_cov_groups=True` and produce the following set of artifacts by selecting only those covariates whose p-values are below some threshold `p_value`  **Running the function remotely**<jupyter_code>fn.spec.build.commands=['pip install lifelines']
fn.deploy(with_mlrun=False)
coxph_run = fn.run(task,
                   local=False,
                   inputs={"dataset"  : "https://s3.wasabisys.com/iguazio/data/function-marketplace-data/coxph_trainer/encoded-data.csv"})<jupyter_output>> 2021-10-13 13:19:41,275 [info] starting run tasks-survive-trainer uid=8b4ee122120645be9eb29a646ef6e562 DB=http://mlrun-api:8080
> 2021-10-13 13:19:41,455 [info] Job is running in the background, pod: tasks-survive-trainer-swjkk
> 2021-10-13 13:19:50,461 [info] run executed, status=completed

A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
Column Contract_2 have very low variance when conditioned on death event present or not. This may harm convergence. This could be a form of 'complete separation'. For example, try the following code:

>>> events = df['labels'].astype(bool)
>>> print(df.loc[events, 'Contract_2'].var())
>>> print(df.loc[~events, 'Contract_2'].var())

A very low variance means that the column Contract_2 completely determines whether a subject dies or not. See https://stats.stackexchange.com/questions/11109/how-to-[...]
{'fraction_non_alphanumeric': 0.11609018344769641, 'fraction_numerical': 0.02506651729449657, 'mean_word_length': 4.820700896495517, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 47, 'https://': 11, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '235145', 'license': 'Apache-2.0', 'n_tokens_mistral': 2386, 'n_tokens_neox': 2165, 'n_words': 557, 'path': 'coxph_trainer/coxph_trainer.ipynb', 'repo_name': 'aviaIguazio/functions', 'size': '7141'}