<jupyter_start><jupyter_text>Intro  Context "Predict behavior to retain customers. You can analyze all relevant customer data and develop focused customer retention programs." [IBM Sample Data Sets]  Content Each row represents a customer, each column contains customer’s attributes described on the column Metadata.The data set includes information about:- Customers who left within the last month – the column is called `Churn`- Services that each customer has signed up for:    - phone (`PhoneService`)    - multiple lines (`MultipleLines`)    - internet (`InternetService`)    - online security (`OnlineSecurity`)    - online backup (`OnlineBackup`)    - device protection (`DeviceProtection`)    - tech support (`TechSupport`)    - streaming TV (`StreamingTV`)    - streaming movies (`StreamingMovies`)- Customer account information:    - how long they’ve been a customer (`tenure`)    - contract (`Contract`)    - payment method (`PaymentMethod`)    - paperless billing (`PaperlessBilling`)    - monthly charges (`MonthlyCharges`)    - total charges (`TotalCharges`)- Demographic info about customers    - Gender (`gender`)    - age range (`SeniorCitizen`)    - if they have partners (`Partner`)    - dependents (`Dependents`)  Import libraries First, let's import all libraries that we might need for our analysis.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import plotly.express as px
import matplotlib.pyplot as plt
from scipy import stats

# set seaborn theme
sns.set_style(style="whitegrid")<jupyter_output><empty_output><jupyter_text>Load data Let's load the *CSV* data:<jupyter_code>df = pd.read_csv("data/WA_Fn-UseC_-Telco-Customer-Churn.csv")
df.head()
df.columns
df.shape<jupyter_output><empty_output><jupyter_text>Getting to know the data The data consists of 21 attributes and about 7,000 samples.At this section, we'll try to *get to know* the data we're dealing with, and answer some questions like the following:- Do columns have the correct `dtype`?- What are the types of variables (Nominal, Ordinal, Discrete, Continuous, Binary)?- Are there any missing values?<jupyter_code>demographic_cols = [
    "gender",
    "SeniorCitizen",
    "Partner",
    "Dependents",
]
account_cols = [
    "tenure",
    "Contract",
    "PaymentMethod",
    "PaperlessBilling",
    "MonthlyCharges",
    "TotalCharges",
]
services_cols = [
    "PhoneService",
    "MultipleLines",
    "InternetService",
    "OnlineSecurity",
    "OnlineBackup",
    "DeviceProtection",
    "TechSupport",
    "StreamingTV",
    "StreamingMovies",
]<jupyter_output><empty_output><jupyter_text>Demographic attributes  `Gender`<jupyter_code>df.gender.head()
df.gender.value_counts()<jupyter_output><empty_output><jupyter_text>The `gender` variable is a binary variable, and it's represented as a `string`.  `SeniorCitizen`<jupyter_code>df.SeniorCitizen.head()
df.SeniorCitizen.value_counts()<jupyter_output><empty_output><jupyter_text>The `SeniorCitizen` variable is a binary variable, but it's represented as integer.It would be better (for visualization) to convert it to *categorical* variable (or `string`).  `Partner`<jupyter_code>df.Partner.head()
df.Partner.value_counts()<jupyter_output><empty_output><jupyter_text>The `Partner` variable is a binary variable, and its data type is `string`.  `Dependents`<jupyter_code>df.Dependents.head()
df.Dependents.value_counts()<jupyter_output><empty_output><jupyter_text>The `Dependents` variable is a binary variable, and its data type is `string`  Account attributes  `Tenure`<jupyter_code>df.tenure.head()
df.tenure.value_counts()<jupyter_output><empty_output><jupyter_text>The `tenure` variable is a dsicrete (numeric) variable, and its data type is integer.  `Contract`<jupyter_code>df.Contract.head()
df.Contract.value_counts()<jupyter_output><empty_output><jupyter_text>The `Contract` variable is an ordinal variable, and its data type is `string`  `PaymentMethod`<jupyter_code>df.PaymentMethod.head()
df.PaymentMethod.value_counts()<jupyter_output><empty_output><jupyter_text>The `PaymentMethod` is a nominal variable, and its data type is `string`  `MonthlyCharges`<jupyter_code>df.MonthlyCharges.head()<jupyter_output><empty_output><jupyter_text>The `MonthlyCharges` is a continuous variable, and its data type is `float`  `TotalCharges`<jupyter_code>df.TotalCharges.head()<jupyter_output><empty_output><jupyter_text>This variable is continuous variable, but its data type is `string` instead of `float`.Let's see if there are any rwos which have non-numeric values (letters, symbols, etc ...)Pandas string functions such as `isalnum`, `isdecimal`, `isdigit` and `isnumeric` won't work here, because we want to match a number with floating point number, instead, we'll use a regular expression to match it.The following pattern will match floating point numbers: `\d+(\.\d*)?`Let's display all rows which **don't** this pattern:<jupyter_code>df.loc[df.TotalCharges.str.match(r"[^\d+(\.\d*)?]"), "TotalCharges"]
df.loc[df.TotalCharges.str.match(r"[^\d+(\.\d*)?]"), "TotalCharges"].values<jupyter_output><empty_output><jupyter_text>It seems that thos are missing entries, represented a empty strings.We should replace them with `NaN` values, so we can deal with all missing values later.And then convert the variable data type to `float`, instead of `string`.  Service attributes  `PhoneService`<jupyter_code>df.PhoneService.head()
df.PhoneService.value_counts()<jupyter_output><empty_output><jupyter_text>The `PhoneService` is a binary variable, and its data type is `string`.  `MultipleLines`<jupyter_code>df.MultipleLines.head()
df.MultipleLines.value_counts()<jupyter_output><empty_output><jupyter_text>`InternetService`<jupyter_code>df.InternetService.head()
df.InternetService.value_counts()<jupyter_output><empty_output><jupyter_text>The `InternetService` is a nominal variable, and its data type is `string`.  `OnlineSecurity`<jupyter_code>df.OnlineSecurity.head()
df.OnlineSecurity.value_counts()<jupyter_output><empty_output><jupyter_text>The `OnlineSecurity` is an ordianl variable, and its data type as `string`.  `OnlineBackup`<jupyter_code>df.OnlineBackup.head()
df.OnlineBackup.value_counts()<jupyter_output><empty_output><jupyter_text>The `OnlineBackup` is an ordinal variable, and its data type is `string`.  `DeviceProtection`<jupyter_code>df.DeviceProtection.head()
df.DeviceProtection.value_counts()<jupyter_output><empty_output><jupyter_text>The `DeviceProtection` is an ordianl variable, and its data type is `string`.  `TechSupport`<jupyter_code>df.TechSupport.head()
df.TechSupport.value_counts()<jupyter_output><empty_output><jupyter_text>The `TechSupport` is an ordinal variable, and its data type is `string`.  `StreamingTV`<jupyter_code>df.StreamingTV.head()
df.StreamingTV.value_counts()
df.StreamingTV.value_counts()<jupyter_output><empty_output><jupyter_text>The `StreamingTV` is an ordinal variable, and its data type is `string`.  `StreamingMovies`<jupyter_code>df.StreamingMovies.head()
df.StreamingMovies.value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11323696145124716, 'fraction_numerical': 0.0008503401360544217, 'mean_word_length': 6.531483457844184, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 89, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '620582', 'license': 'MIT', 'n_tokens_mistral': 2329, 'n_tokens_neox': 2235, 'n_words': 596, 'path': '1-get-to-know-the-data.ipynb', 'repo_name': 'Reslan-Tinawi/Telco-Customer-Churn', 'size': '7056'}