<jupyter_start><jupyter_text>Challenge Overview:Blue Nova Telecoms was born out of a need for reliable and affordable PREMIUM internet connectivity.Below is a challenge which needs to be completed within 72 hours of receipt of this assessment. Task Detail:Predicting churn for telco customers. IntroductionIn this challenge we aim to determine whether or not a person will Churn using Customers who left within the last month data , Customer Demographics , Services that each customer has signed up for and Customer account information. As such this is a classification problem and we will be using classification techniques to gain useful insights and make predictions on our dataset.Our dataset contains structure and label data which contains both numerical and categorical data. Also because our data is labeled we will use Supervised Machine Learning Algorithms for our preditions. Aim* To make recommendations based on analysis* Create interesting data visualisations to demonstrate useful insights.* Make predictions on data to determine whether or not a customer will Churn  Importing Datasets¶The train perfromance dataset<jupyter_code>dataset = pd.read_csv("Downloads\data.csv")
dataset.head()<jupyter_output><empty_output><jupyter_text>Summary StatisticsIn this section we will explore the data types and the summary statistics of our variables.Let's start by looking at the data types and number of entries of each column.<jupyter_code>dataset.info()
dataset.describe()<jupyter_output><empty_output><jupyter_text>Due to the nature of some of the features we will need to standardise the values for better predictions  Exploratory Data Analysis¶ Data Visualisations of Customer account information<jupyter_code>dataset[dataset.Churn == 'Yes'].Contract.value_counts().plot(kind = 'bar')<jupyter_output><empty_output><jupyter_text>The plot shows the Customer Contract * We have a large number of customer on a month-to-month who will Churn * We have less than 200 customers on a one year contract who has a chace of churning * Very little number of customers on a two years contract that has a possibility of churning<jupyter_code>df1 = dataset[dataset.Churn == 'Yes'].SeniorCitizen.value_counts()
df2 = dataset[dataset.Churn == 'Yes'].gender.value_counts()
df3 = dataset[dataset.Churn == 'Yes'].Partner.value_counts()
df4 = dataset[dataset.Churn == 'Yes'].Dependents.value_counts()
#define number of rows and columns for subplots
nrow=2
ncol=2

# make a list of all dataframes 
df_list = [df1 ,df2, df3, df4]

df_titles = ['SeniorCitizen', 'gender', 'Partner', 'Dependents']

fig, axes = plt.subplots(2,2,figsize=(15,10))

# plot counter
count=0
for r in range(nrow):
    for c in range(ncol):
        df_list[count].plot(kind='bar', ax=axes[r,c] ,color = ['grey', 'blue'], alpha=0.5, title = df_titles[count], ylabel ='No. Of People')
        count+=1<jupyter_output><empty_output><jupyter_text>Data Visualisations of Customer Demographics* On the first graph on the qualtrics We have a large number of none-Senior Citizen that who has a high possibility to Churn * Second graph shows that gender has no effect on a person churning or not * Third graph show that number of people who does not have partners has greater risk of churning * Last graph tell us that people with no dependents have a high possibilty of of churning than those you have dependents<jupyter_code>df1 = dataset[dataset.Churn == 'Yes'].StreamingMovies.value_counts()
df2 = dataset[dataset.Churn == 'Yes'].PhoneService.value_counts()
df3 = dataset[dataset.Churn == 'Yes'].StreamingTV.value_counts()
df4 = dataset[dataset.Churn == 'Yes'].TechSupport.value_counts()
df5 = dataset[dataset.Churn == 'Yes'].OnlineBackup.value_counts()
df6 = dataset[dataset.Churn == 'Yes'].MultipleLines.value_counts()
df7 = dataset[dataset.Churn == 'Yes'].InternetService.value_counts()
df8 = dataset[dataset.Churn == 'Yes'].DeviceProtection.value_counts()
#define number of rows and columns for subplots
nrow=2
ncol=4

# make a list of all dataframes 
df_list = [df1 ,df2, df3, df4, df5, df6, df7, df8]

df_titles = ['StreamingMovies', 'PhoneService', 'StreamingTV', 'TechSupport', 'OnlineBackup', 'MultipleLines', 'InternetService', 'DeviceProtection']

fig, axes = plt.subplots(2,4,figsize=(20,15))

# plot counter
count=0
for r in range(nrow):
    for c in range(ncol):
        df_list[count].plot(kind='bar', ax=axes[r,c] ,color = ['red', 'green','grey'], alpha=0.5, title = df_titles[count], ylabel ='No. Of People')
        count+=1<jupyter_output><empty_output><jupyter_text>Data Visualisations of Services that each customer has signed up for* A large number of customers who do not stream movies have a high chance of churning followed by those who do stream movies and a small number of those who do not have internet service* We at a risk of customers who dont have phone services churning than those who have * A large number of customers that does not stream TV have a high chance of churning followed by those who have it and a very small number of those who does not have internet service * We have a large number of people who does not have tech supoort and have a high risk of churning followed by those who have tech support and a small number of those have dont have internet service * We have more than 90% of customers who dont have online backup and at risk of churning and less than 50% of those who do have online backup* There is a very small effect on a customer having mulyiplelines churning* A large number of customers on the fibre optic have a great chance to churn followed by DSL* Large number of people who does not have device protection has a large number of churning<jupyter_code>dataset[dataset.Churn == 'Yes'].PaymentMethod.value_counts().plot(kind = 'bar', title ='PaymentMethod')<jupyter_output><empty_output><jupyter_text>Data Visualisations of Customer payment Method* A large number of customers on electronic check has a high chance of churning followed by mailed check * Bank transfer and Credit card payment has no effect on customer churning<jupyter_code>dataset[dataset.Churn == 'Yes'].PaperlessBilling.value_counts().plot(kind = 'bar', title = 'PaperlessBilling')<jupyter_output><empty_output><jupyter_text>Data Visualisations of Customer Paperless Billing* The graph shows that a large number of customers on a paperless billing have churned  Data Cleaning Categorical Variable EncodingWe will encode categorical values with numberical ones on the some columns with catagorical values<jupyter_code># Function that takes and dataframe and categorical feature name to be encoded and returns encoded dataframe
def feat_encode(df, col_name):
    # Get unique categories from column
    keys = list(df[col_name].unique())
    # Get index for each unique category to be encoded value
    values = [val for val in range(len(keys))]
    # Create dictionary of the above keys and values
    encode_dict = dict(zip(keys,values))
    # Alter dataframe entries by swapping each category name with values
    df[col_name] = df[col_name].apply(lambda x: encode_dict[x])
    # Return altered dataframe
    return df
def data_en(df):
    le = LabelEncoder()
    for col in df:
        if df[col].dtypes== 'object':
            df[col]=le.fit_transform(df[col])
    return df
#cols = ['gender','Partner','Dependents','PhoneService','TotalCharges','PaymentMethod','PaperlessBilling','Contract','StreamingMovies','StreamingTV','TechSupport','OnlineBackup','OnlineSecurity','MultipleLines','InternetService','DeviceProtection','Churn']
#for name in cols:
dataset = data_en(dataset)
from statsmodels.graphics.correlation import plot_corr

fig = plt.figure(figsize=(10,10));
ax = fig.add_subplot(111);
plot_corr(dataset.corr(), xnames = dataset.corr().columns, ax=ax);<jupyter_output><empty_output><jupyter_text>* From the above correlation matrix we see that there is a perfect correlation between SiniorCitizen and PaymentMethod, which makes sense since we only have few SeniorCitizen using bank transfer payment method.* There is also a high correlation between the PaperlessBilling and the monthlyCharges. Since is an easy way to receive your monthly statement and it clears the clutter of paper bills.* Finally, the SeniorCitizen and PaymentMethod would make a fairly good predictor for whether or not a person will Churn.<jupyter_code>dataset.head()<jupyter_output><empty_output><jupyter_text>Variable Selection and Preprocessing<jupyter_code>def data_preprocess(df):
    le = LabelEncoder()
    for col in df:
        if df[col].dtypes== 'object':
            df[col]=le.fit_transform(df[col])
    X = df[['tenure','Contract', 'PaperlessBilling', 'MonthlyCharges', 'TechSupport']]
    y = df.Churn
    scaler = MinMaxScaler(feature_range=(0, 1))
    rescaledX = scaler.fit_transform(X)
    X_train, X_test, y_train, y_test = train_test_split(rescaledX,
                                y,
                                test_size=0.2,
                                random_state=42)
    return (X_train, y_train), (X_test, y_test)
(X_train, y_train), (X_test, y_test) = data_preprocess(dataset)
print(X_train[:1])
print(y_train[:1])
print(X_test[:1])
print(y_test[:1])
def train_model(X_train, y_train):
    lm = LogisticRegression()
    lm.fit(X_train,y_train)
    return lm<jupyter_output><empty_output><jupyter_text>Testing Our ModelsModels to EvaluateWe will use logistic regression validate scores<jupyter_code># Function to calculate mean absolute error
def cross_val(X_train, y_train, model):
    # Applying k-Fold Cross Validation
    from sklearn.model_selection import cross_val_score
    accuracies = cross_val_score(estimator = model, X = X_train, y = y_train, cv = 10)
    return accuracies.mean()
# Function to calculate predictions
def get_predicitons(model, X_test):
    # Make predictions using selected model
    model_pred = model.predict(X_test)
    # Return model predictions
    return model_pred
# Takes in a model, trains the model, and evaluates the model on the test set
def evaluate(model):
    # Train the model
    model.fit(X_train, y_train)
    # Make predictions and evalute
    model_cross = cross_val(X_train, y_train, model)
    # Return the performance metric
    return model_cross
# Classification Models used to classify the data
lm = LogisticRegression(solver='lbfgs')
print('Logistic Regression Cross Validation Score: ', evaluate(lm))<jupyter_output>Logistic Regression Cross Validation Score:  0.7878856861295885
{'fraction_non_alphanumeric': 0.08389935903568353, 'fraction_numerical': 0.008896967377786282, 'mean_word_length': 5.178486997635934, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '424111', 'license': 'MIT', 'n_tokens_mistral': 3060, 'n_tokens_neox': 2931, 'n_words': 1147, 'path': 'BlueNova_assessement_Itumeleng_Mabaso.ipynb', 'repo_name': 'itumelengmabaso/Bluenova_assesment', 'size': '10453'}