<jupyter_start><jupyter_text>Telecom: Customer-product classification  *Created by: <NAME>, data scientist  Antwerpen, Belgium  Date: October 28, 2019  Contacts: <EMAIL>*  *** Table of ContentsStage 1 : Business Understanding  Stage 2 : Analytic Approach  Stage 3 : Data Requirements  Stage 4 : Data Collection  Stage 5 : Data Understanding  Stage 6 : Data Preparation  Stage 7 : Modeling  Stage 8 : Evaluation  Stage 9 : Conclusion*** Stage 1 : Business Understanding**Problem:**  A telecommunications company has four services, as follows:  1. Basic Service  2. E-Service  3. Plus Service  4. Total Service  To increase conversion rate for a new customers,  the company needs to customize offers for individual prospective customers, depends on what product he is most likely to buy.  **Question:**  Can we predict the most appropriate service to a future customer?   Stage 2 : Analytic ApproachTo answer the question, we need to segment the customer base by service usage patterns, categorizing the customers into four groups. It is a classification problem. Given the dataset, with predefined labels, we need to build a model to be used to predict class of a new or unknown case. We will use a supervised machine learning technique "Classification" with a specific type of classification called K nearest neighbour.  To evaluate models performance, we use metrics accuracy score, which is the fraction of samples predicted correctly. Stage 3 : Data Requirements**Data content:** We need demographic data about current customers and company's services they use.**Data formats:** CSV format  **Data sources:** Corporative information from the company. Stage 4 :  Data CollectionImporting libraries:<jupyter_code>import itertools
import pandas as pd
import numpy as np

import matplotlib.ticker as ticker
import matplotlib.pyplot as plt
from matplotlib.ticker import NullFormatter
%matplotlib inline

from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn import metrics<jupyter_output><empty_output><jupyter_text>Dataset is collected from IBM Object Storage. To download the data:<jupyter_code>!wget -O teleCust1000t.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/teleCust1000t.csv
df = pd.read_csv('teleCust1000t.csv')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Stage 5 : Data Understanding **How many of each class is in our data set?**<jupyter_code>df['custcat'].value_counts()<jupyter_output><empty_output><jupyter_text>281 Plus Service, 266 Basic-service, 236 Total Service, and 217 E-Service customers  Stage 6 : Data Preparation Defining feature sets, X:<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>To use scikit-learn library, we have to convert the Pandas data frame to a Numpy array:<jupyter_code>X = df[['region', 'tenure','age', 'marital', 'address', 'income', 'ed', 'employ','retire', 'gender', 'reside']] .values  
X[0:5]<jupyter_output><empty_output><jupyter_text>Defining target, y:<jupyter_code>y = df['custcat'].values
y[0:5]<jupyter_output><empty_output><jupyter_text>**Normalizing Data**  As the KNN algorithm is based on distance of cases, we need to give data zero mean and unit variance, using Data Standardization:<jupyter_code>X = preprocessing.StandardScaler().fit(X).transform(X.astype(float))
X[0:5]<jupyter_output><empty_output><jupyter_text>**Train/test/split:**<jupyter_code>X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)<jupyter_output>Train set: (800, 11) (800,)
Test set: (200, 11) (200,)<jupyter_text>Stage 7 : Modeling<jupyter_code>k = 4
neigh = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)
neigh<jupyter_output><empty_output><jupyter_text>Predicting:<jupyter_code>yhat = neigh.predict(X_test)
yhat[0:5]<jupyter_output><empty_output><jupyter_text>Stage 8 : Evaluation<jupyter_code>print("Train set Accuracy: ", metrics.accuracy_score(y_train, neigh.predict(X_train)))
print("Test set Accuracy: ", metrics.accuracy_score(y_test, yhat))<jupyter_output>Train set Accuracy:  0.5475
Test set Accuracy:  0.32<jupyter_text>**Finding best K**<jupyter_code>Ks = 10
mean_acc = np.zeros((Ks-1))
std_acc = np.zeros((Ks-1))
ConfustionMx = [];
for n in range(1,Ks):
    
    #Train Model and Predict  
    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)
    yhat=neigh.predict(X_test)
    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)

    
    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])

mean_acc
#visualization:
plt.plot(range(1,Ks),mean_acc,'g')
plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)
plt.legend(('Accuracy ', '+/- 3xstd'))
plt.ylabel('Accuracy ')
plt.xlabel('Number of Nabors (K)')
plt.tight_layout()
plt.show()
print( "The best accuracy was with", mean_acc.max(), "with k=", mean_acc.argmax()+1)<jupyter_output>The best accuracy was with 0.34 with k= 9<jupyter_text>Re-run the model with k = 9 and evaluate the model:<jupyter_code>k = 9
neigh_9 = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)
neigh_9
yhat_9 = neigh_9.predict(X_test)
print("Train k-9 set Accuracy: ", metrics.accuracy_score(y_train, neigh_9.predict(X_train)))
print("Test k-9 set Accuracy: ", metrics.accuracy_score(y_test, yhat_9))<jupyter_output>Train k-9 set Accuracy:  0.5025
Test k-9 set Accuracy:  0.34
{'fraction_non_alphanumeric': 0.10943124550035997, 'fraction_numerical': 0.02411807055435565, 'mean_word_length': 6.311842105263158, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '480088', 'license': 'MIT', 'n_tokens_mistral': 1992, 'n_tokens_neox': 1836, 'n_words': 530, 'path': 'Labs/Telecom: Customer-products classification.ipynb', 'repo_name': 'AnnaSukhareva/DataScienceLabs', 'size': '5573'}