<jupyter_start><jupyter_text>Any business wants to maximize the number of customers. To achieve this goal, it is important not only to try to attract new ones, but also to retain existing ones. Retaining a client will cost the company less than attracting a new one. In addition, a new client may be weakly interested in business services and it will be difficult to work with him, while old clients already have the necessary data on interaction with the service. Accordingly, predicting the churn, we can react in time and try to keep the client who wants to leave. Based on the data about the services that the client uses, we can make him a special offer, trying to change his decision to leave the operator. This will make the task of retention easier to implement than the task of attracting new users, about which we do not know anything yet. You are provided with a dataset from a telecommunications company. The data contains information about almost six thousand users, their demographic characteristics, the services they use, the duration of using the operator's services, the method of payment, and the amount of payment. The task is to analyze the data and predict the churn of users (to identify people who will and will not renew their contract). The work should include the following mandatory items: Description of the data (with the calculation of basic statistics); Research of dependencies and formulation of hypotheses; Building models for predicting the outflow (with justification for the choice of a particular model) based on tested hypotheses and identified relationships; Comparison of the quality of the obtained models. About this filecustomerID - customer idgender - client gender (male / female)SeniorCitizen - is the client retired (1, 0)Partner - is the client married (Yes, No)tenure - how many months a person has been a client of the companyPhoneService - is the telephone service connected (Yes, No)MultipleLines - are multiple phone lines connected (Yes, No, No phone service)InternetService - client's Internet service provider (DSL, Fiber optic, No)OnlineSecurity - is the online security service connected (Yes, No, No internet service)OnlineBackup - is the online backup service activated (Yes, No, No internet service)DeviceProtection - does the client have equipment insurance (Yes, No, No internet service)TechSupport - is the technical support service connected (Yes, No, No internet service)StreamingTV - is the streaming TV service connected (Yes, No, No internet service)StreamingMovies - is the streaming cinema service activated (Yes, No, No internet service)Contract - type of customer contract (Month-to-month, One year, Two year)PaperlessBilling - whether the client uses paperless billing (Yes, No)PaymentMethod - payment method (Electronic check, Mailed check, Bank transfer (automatic), Credit card (automatic))MonthlyCharges - current monthly paymentTotalCharges - the total amount that the client paid for the services for the entire timeChurn - whether there was a churn (Yes or No)<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder,StandardScaler
import pandas as pd 
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split,cross_val_score,GridSearchCV,KFold
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier,AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import SGDClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.svm import SVC
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve,confusion_matrix
from sklearn.metrics import roc_auc_score,recall_score
from sklearn.metrics  import accuracy_score,classification_report,roc_auc_score,plot_roc_curve,plot_precision_recall_curve
import scipy.stats as stats
from sklearn.decomposition import PCA
import warnings
warnings.filterwarnings('ignore')
data=pd.read_csv('/kaggle/input/telecom-users-dataset/telecom_users.csv')
data
data.columns<jupyter_output><empty_output><jupyter_text>There are 2 variable which is useless for our model..[Unnamed: 0', 'customerID'],so we drop it.<jupyter_code>data.drop(columns=['Unnamed: 0','customerID'],inplace=True)<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5986 entries, 0 to 5985
Data columns (total 20 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   gender            5986 non-null   object 
 1   SeniorCitizen     5986 non-null   int64  
 2   Partner           5986 non-null   object 
 3   Dependents        5986 non-null   object 
 4   tenure            5986 non-null   int64  
 5   PhoneService      5986 non-null   object 
 6   MultipleLines     5986 non-null   object 
 7   InternetService   5986 non-null   object 
 8   OnlineSecurity    5986 non-null   object 
 9   OnlineBackup      5986 non-null   object 
 10  DeviceProtection  5986 non-null   object 
 11  TechSupport       5986 non-null   object 
 12  StreamingTV       5986 non-null   object 
 13  StreamingMovies   5986 non-null   object 
 14  Contract          5986 non-null   object 
 15  PaperlessBilling  5986 non-null   object 
 16  PaymentMethod     5986 non-null   object 
[...]<jupyter_text>we see that variable TotalCharges is object type but it should be float type,so lets convert it.<jupyter_code>data['TotalCharges']=pd.to_numeric(data['TotalCharges'],errors='coerce')
data.isnull().sum()
data['TotalCharges'].fillna(data['TotalCharges'].median(),inplace=True)# missing value treated
data.describe()
data.describe(include=[np.object])
cat_var=data.select_dtypes(include=object).columns.to_list()
num_var=data.select_dtypes(include=np.number).columns.to_list()
data1=data.copy()<jupyter_output><empty_output><jupyter_text>UNIVARIATE ANALYSIS<jupyter_code>data1.hist(figsize=(10,10),color='red')
plt.show()
for i in num_var:
    sns.boxplot(data=data,x=i)
    plt.show()<jupyter_output><empty_output><jupyter_text>Here we can see that we did'nt get any outlier  BIVARIATE ANALYSIS<jupyter_code>sns.pairplot(data)
i=1
plt.figure(figsize=(15,50))
for col in cat_var:
    plt.subplot(9,2,i)
    sns.countplot(data=data1, x=col,hue='Churn')
    i+=1<jupyter_output><empty_output><jupyter_text>By observing above plot carefully ,we reach upto certain conclusion that there are some categories of variable where the customer has more probability to leave the services. So ,to retain them in  telecom services company will have to primarily focus on those factors. Here is the list of those key factor where there is more probability of customer getting churn, Organisation has to improve those field of services 1- Parner-No 2- Dependent-NO 3- Phone Services-Yes 4- Multiple lines-Yes 5- Internet Service- Fibre optics 6- Online security -NO 7- Online Backup-NO 8- Device Protection-No 9- Tech Support -No 10- Streaming TV- NO 11- Streaming Movies no 12- Contract-month to month 13- Paperless billing -YES 14- Payment Method - Electronic check So all above are the key area where services will have to be the best , otherwise customer will more tendency to churn. Telecom company has to focus on more to retain old customer rather than to find new customers only.<jupyter_code>for i in num_var:
    sns.boxplot(data=data,y=i,x='Churn')
    plt.show()<jupyter_output><empty_output><jupyter_text>The customers whose tenure is less, monthly charge is high , total charge is less are more probable to leave the services.<jupyter_code>lb=LabelEncoder()
for i in cat_var:
    data1[i]=lb.fit_transform(data[i])
data1
plt.figure(figsize=(15,15))
sns.heatmap(data1.corr(),linewidths=1,cmap='Accent_r',annot=True)
plt.figure(figsize=(10,10))
corr=data1.corr()['Churn'].sort_values(ascending=False)
corr.plot(kind='bar')
data2=data.copy()<jupyter_output><empty_output><jupyter_text>FEATURE ENGINEERING<jupyter_code>data2
data2=pd.get_dummies(data2,drop_first=True)
sc=StandardScaler()
scaled=sc.fit_transform(data2[['tenure','MonthlyCharges','TotalCharges']])
scaled=pd.DataFrame(scaled,columns=['tenure','MonthlyCharges','TotalCharges'])
scaled_data=pd.concat([scaled,data2.drop(columns=['tenure','MonthlyCharges','TotalCharges'])],axis=1)
scaled_data
X_scaled=scaled_data.drop(columns='Churn_Yes')
Y=scaled_data[['Churn_Yes']]
pca=PCA()
model=pca.fit_transform(X_scaled)
model
np.cumsum(pca.explained_variance_ratio_)
feature=range(pca.n_components_)
plt.figure(figsize=(20,5))
plt.bar(feature,pca.explained_variance_ratio_)
plt.step(feature,np.cumsum(pca.explained_variance_ratio_),color='red')
plt.axhline(y=.98,color='green')
plt.grid(axis='x')
plt.text(0.5,.85,'98% cutt-off thresold',color='green',fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>upto 19 PCA component 98% variance of the data has been covered,so we will take n_component=19<jupyter_code>pca1=PCA(n_components=19,whiten=True)
model1=pca1.fit_transform(X_scaled)
np.cumsum(pca1.explained_variance_ratio_)<jupyter_output><empty_output><jupyter_text>Model Selection  Before going into the Modelling ,it should be noted that our data is highly sensitive to False Negative. Let suppose a customer has high probability to leave the sevice(churn) ,but if our model predict that he is not going to leave (not churn) which is False Negative, then company will not be  primarily focusing to that customer's services and then customer will definatly leave the sevices(churn) which results in loss to the company. And if customer is not willing to leave the services and is satisfied, but our model tells that he is going to leave the services then company will provide him extra benefits ,offers and then customer will become extra happy. Thats why we will not focus on getting best accuracy but try to get best Recall Score.<jupyter_code>x_train,x_test,y_train,y_test=train_test_split(model1,Y,test_size=.3,random_state=0)
lg=LogisticRegression()
dt=DecisionTreeClassifier()
rf=RandomForestClassifier()
nb=GaussianNB()
knn=KNeighborsClassifier()
gbc=GradientBoostingClassifier()
adb=AdaBoostClassifier()
sgd=SGDClassifier()
svc=SVC()
mlpc=MLPClassifier()
algo=[lg,dt,rf,nb,knn,adb,sgd,svc,mlpc]
model=[]
acc=[]
recall=[]
for i in algo:
    i.fit(x_train,y_train)
    y_pred=i.predict(x_test)
    model.append(i)
    acc.append(i.score(x_test,y_test))
    recall.append(recall_score(y_pred,y_test))
table=pd.DataFrame([model,acc,recall]).T
table.columns=['model','accuracy','recall']
table<jupyter_output><empty_output><jupyter_text>We can see that logistic regression and SVC has the same accuracy and recall score. we choose SVC  for final model model building<jupyter_code>svc1=SVC(probability=True)
svc1.fit(x_train,y_train)
svc1.score(x_test,y_test)
y_pred=svc1.predict(x_test)
print(classification_report(y_pred,y_test))
plot_roc_curve(svc1,x_test,y_test)
roc_auc_score(y_test,y_pred)
plot_precision_recall_curve(svc1,x_test,y_test)<jupyter_output><empty_output><jupyter_text>We know that in any binary classification model, the cuttoff point for probability is by default =0.5. So we choose the cuttoff point to reduce the accuracy to 75%. so that recall can increase.<jupyter_code>y_pred_prob=svc1.predict_proba(x_test)[:,1]
FPR,TPR,THRESOLD=roc_curve(y_test,y_pred_prob)
for i in THRESOLD:
    y_test_pred=np.where(y_pred_prob>i,1,0)
    acc=accuracy_score(y_test,y_test_pred)
    print(acc,i)<jupyter_output>0.7332962138084632 1.930420249557542
0.7332962138084632 0.930420249557542
0.7505567928730512 0.8666414348914867
0.7511135857461024 0.8663085091678091
0.7527839643652561 0.8530003028533343
0.7527839643652561 0.850301442568213
0.7538975501113586 0.8417193639103868
0.7544543429844098 0.8411403369932694
0.7589086859688196 0.8326270763415344
0.7594654788418709 0.8319283950678412
0.7594654788418709 0.8279431206542932
0.7600222717149221 0.8262995158500265
0.7600222717149221 0.8223760998219447
0.7600222717149221 0.8210875339291348
0.7600222717149221 0.817040728942776
0.7605790645879733 0.8170348552433161
0.7605790645879733 0.8161466423767009
0.7611358574610245 0.8097325019168254
0.762249443207127 0.808480219549454
0.762249443207127 0.8061700961658075
0.7616926503340757 0.8060546134877351
0.762249443207127 0.8048679983003107
0.7661469933184856 0.798511395013268
0.7667037861915368 0.7981996729407522
0.7661469933184856 0.7980781815198693
0.7667037861915368 0.7962073884925411
0.767260579064588 0.7[...]<jupyter_text>From above we can see that at cuttoff point 0.17 we get accuracy upto 70%<jupyter_code>for i in THRESOLD:
    y_test_pred=np.where(y_pred_prob>i,1,0)
    recall=recall_score(y_test,y_test_pred)
    print(recall,i)
y_test_pred=np.where(y_pred_prob>=0.17,1,0)
print(classification_report(y_test,y_test_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.90      0.68      0.77      1317
           1       0.47      0.79      0.59       479

    accuracy                           0.71      1796
   macro avg       0.68      0.73      0.68      1796
weighted avg       0.78      0.71      0.72      1796
{'fraction_non_alphanumeric': 0.07979076106977087, 'fraction_numerical': 0.08649524791866205, 'mean_word_length': 5.439278937381404, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '335227', 'license': 'Apache-2.0', 'n_tokens_mistral': 5117, 'n_tokens_neox': 4341, 'n_words': 1305, 'path': 'Telecom User Churn .ipynb', 'repo_name': 'NaveenKumarMaurya/my-datascience-end-to-end-project-portfolio', 'size': '13573'}