<jupyter_start><jupyter_text>Cancer Cell Samples: 4 Classification Models What is the likelihood that a patient's cancer is malignant? Today, we'll look into this question. We'll optimize, train, make predictions with, and evaluate four classification models - K Nearest Neighbor (KNN), Decision Tree, Support Vector Machine (SVM), and logistic regression - to predict the class of new patient cell sample. We'll work with cell sample data publicly available from the UCI ML repository (https://archive.ics.uci.edu/ml/index.php). The data set includes cell samples from 699 patients with cell characteristics such as clump thickness, uniformity of size, uniformity of shape, marginal adhesion, bare nuclei, single epithelial cell size, and mitoses. A hospital's oncology department, for example, could apply a predictive model alongside diagnostic procedures and patient evaluation to improve cancer diagnosis. Let's break it down:**Part 1**: Cleaning and wrangling, including replacing values**Part 2**: Exploratory analysis, including plotting stratified scatter and bubble plots and histograms, working with groupby, and making observations to determine key features**Part 3**: Feature selection of predictors (X) and labeled target (y)**Part 4**: Normalizing feature set using scikit learn's preprocessing.StandardScaler.fit.transform**Part 5**: KNN, including determining and plotting optimal k value, training model and making predictions on test set, generating a confusion matrix heatmap and report, evaluating jaccard and F1 scores**Part 6**: Decision Tree, including determining and plotting optimal max depth, training model and making predictions on test set, visualizing decision tree using pydotplus and graphviz, generating a confusion matrix heatmap and report, evaluating jaccard and F1 scores**Part 7**: SVM, including determining and plotting optimal kernel function, training model and making predictions on test set, generating a confusion matrix heatmap and report, evaluating jaccard and F1 scores**Part 8**: Logistic Regression, including determining and plotting optimal regularization and numerical solver, training model and making predictions on test set, calculating probability, generating a confusion matrix heatmap and report, evaluating jaccard, F1, and log loss scores**Part 9**: Evaluating model performance head-to-head by creating a dataframe of accuracy scores for KNN, Decision Tree, SVM, and Logistic Regression models to make comparisonsWe'll cover cleaning, wrangling, and visualizing techniques, apply important scikit learn libraries to develop, optimize, train, and make predictions, and walk through evaluating and comparing models. We'll discover that, using features in this data set, logistic regression generates the highest accuracy of our 4 models. It also has the added benefit of providing percent likelihood that a patient's cancer is malignant. Let's dig in.<jupyter_code># Import relevant libraries
import itertools
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import NullFormatter
import pandas as pd
import numpy as np
import matplotlib.ticker as ticker
import seaborn as sns

# Scikit learn libraries
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier 
from sklearn import tree
import scipy.optimize as opt
from sklearn.svm import SVC 
from sklearn import svm
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.metrics import jaccard_similarity_score
from sklearn.metrics import f1_score
from sklearn.metrics import log_loss

# Visualizing Decision Tree
!pip install graphviz
!pip install pydotplus
import graphviz 
import pydotplus
# The code was removed by Watson Studio for sharing.<jupyter_output><empty_output><jupyter_text>Part 1: Cleaning and Wrangling<jupyter_code># Read in data
pd.set_option("display.max_columns", 100)
df = pd.read_csv(body)
# Check first few rows
df.head()<jupyter_output><empty_output><jupyter_text>Our cell sample dataframe contains 11 attributes for each patient cell sample. Except for ID and Class, each attribute is a cell characteristic scored from 1 to 10 with 1 being closest to benign and 10 being closest to malignant. The variable Class is the confirmed diagnosis with 2 being benign and 4 being malignant. **ID**: Patient ID**Clump**: Clump thickness**UnifSize**: Uniformity of cell size**UnifShape**: Uniformity of cell shape**MargAdh**: Marginal adhesion**BareNuc**: Bare nuclei**SingEpiSize**: Single epithelial cell size**BlandChrom**: Bland chromatin**NormNucl**: Normal nucleoli**Mit**: Mitoses**Class**: Cancer class (2=benign, 4=malignant)<jupyter_code># Check dimensions of dataframe. We have 699 cell samples and 11 attributes
df.shape
# Check number of null values. There are no null values
df.isnull().sum().to_frame()
# Check datatypes.
df.dtypes
# The variable BareNuc is type object, let's convert that to integer. 
df = df[pd.to_numeric(df['BareNuc'], errors='coerce').notnull()]
df['BareNuc'] = df['BareNuc'].astype('int')
df.dtypes
# Convert gender strings 'Male' to 0 and 'Female' to 1
df['Class'].replace(to_replace=[2,4], value=["Benign","Malignant"],inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Part 2: Exploratory Analysis<jupyter_code># As we'll be plotting with seaborn, let's set the style to darkgrid
sns.set_style("darkgrid")
# Check cancer class split. We have 444 benign and 239 malignant.
df['Class'].value_counts().to_frame()<jupyter_output><empty_output><jupyter_text>Let's plot UnifSize vs Clump stratified by Class (benign or malignant). We see that benign samples mostly bunch together with Clump thickness and UnifSize of 5 or below while malignant samples tend toward Clump and UnifSize on the higher end, though are widely dispersed.<jupyter_code># Scatter plot of UnifSize vs Clump stratified by Class
g=sns.FacetGrid(df, col = "Class")
g.map(plt.scatter, "Clump", "UnifSize", alpha=0.7)<jupyter_output><empty_output><jupyter_text>Let's plot UnifShape vs UnifSize stratified by Class (benign or malignant). Here we can see a clearer difference. Benign samples mostly bunch together with UnifSize and UnifShape of 5 or below while malignant samples tend toward UnifSize and UnifShape on the higher end.<jupyter_code># Scatter plot of UnifShape vs UnifSize stratified by Class
g=sns.FacetGrid(df, col = "Class")
g.map(plt.scatter, "UnifSize", "UnifShape", alpha=0.7)<jupyter_output><empty_output><jupyter_text>We can look at it in a different way. Let's combinine in a single plot with a scatter plot of UnifShape vs UnifSize using a light color for benign and darker color for malignant. The trend we saw above looks clearer.<jupyter_code># Scatter plot of UnifShape vs UnifSize stratified by Class
plt.figure(dpi=140) # Figure size must be set before calling the plot
sns.scatterplot(df["UnifSize"], df["UnifShape"], hue = df["Class"], palette = "coolwarm")
plt.title("UnifShape vs UnifSize by Class", fontsize=10)<jupyter_output><empty_output><jupyter_text>We can add more dimensionality to the above by also adjusting the size of the bubbles according to clump thickness. Here we see that high clump values appear across benign and malignant samples.<jupyter_code># Scatter plot of UnifShape vs UnifSize stratified by Class and Clump
plt.figure(dpi=160) # Figure size must be set before calling the plot
sns.scatterplot(df["UnifSize"], df["UnifShape"], hue = df["Class"], size = df["Clump"], sizes = (10,200), palette="coolwarm", legend=False)
plt.title("UnifShape vs UnifSize by Class and Clump", fontsize=10)<jupyter_output><empty_output><jupyter_text>Let's check how UnifSize is distributed by Class. We see below that for benign samples, UnifSize is heavily right skewed, while for malignant samples, they are quite uniformly distributed (slight left skew).<jupyter_code># Plot histogram of UnifSize stratified by UnifShape and Class
g = sns.FacetGrid(df, col="Class", col_wrap=2)
g.map(plt.hist, 'UnifSize', ec="k")
plt.show()<jupyter_output><empty_output><jupyter_text>We can add more dimensionality to the above by stratifying by both Class and Clump thickness. We see that for low clump thickness, samples are entirely benign and UnifSize is nearly entirely the value 1. As clump thickness increases, samples tend to be malignant while UnifSize becomes more uniformly distributed.<jupyter_code># Plot histogram of UnifSize stratified by UnifShape and Class
g = sns.FacetGrid(df, col="Clump", hue="Class", palette="coolwarm")
g.map(plt.hist, 'UnifSize', ec="k")
plt.show()<jupyter_output><empty_output><jupyter_text>Grouping our data by UnifSize using groupby, we see that 99% of samples are benign with UnifSize of 1. As UnifSize increases, the percentage of benign steadily decreases while that of malignant steadily increases.<jupyter_code># Check percentage of Class by UnifSize 
df.groupby(['UnifSize'])['Class'].value_counts(normalize=True).to_frame()<jupyter_output><empty_output><jupyter_text>Part 3: Feature Selection Let's select our predictors, feature set X.<jupyter_code># Create a numpy array for the features we'll be selecting. Numpy array is required for the scikit learn alogrithms later. 
features = df[['Clump', 'UnifSize', 'UnifShape', 'MargAdh', 'SingEpiSize', 'BareNuc', 'BlandChrom', 'NormNucl', 'Mit']]
X=np.asarray(features)
X[0:5]<jupyter_output><empty_output><jupyter_text>Let's select our labeled target as Class, y.<jupyter_code>y = df['Class'].values
y[0:5]<jupyter_output><empty_output><jupyter_text>Part 4: Normalizing Feature Set It's important to normalize our feature set to a zero mean and variance to prevent individual feature from being over or underweighted and in order to generate interpretable, reliable predictions. We can do this using the scikit learn libraries StandardScaler, fit, and transform.<jupyter_code># Normalize using scikit learn preprocessing libraries StandardScaler, fit, transform. 
X = preprocessing.StandardScaler().fit(X).transform(X)
X[0:5]<jupyter_output>/opt/conda/envs/Python36/lib/python3.6/site-packages/sklearn/utils/validation.py:595: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)
/opt/conda/envs/Python36/lib/python3.6/site-packages/sklearn/utils/validation.py:595: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)<jupyter_text>Classification Models - Optimize Algorithms, Build Train and Test Sets, and Evaluate Models Let's build and compare 4 classification models: K Nearest Neighbor (KNN), Decision Tree, Support Vector Machine (SVM), and Logistic Regression.  Part 5: KNN<jupyter_code># Split into train and test sets 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=1)
print ('Training set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)<jupyter_output>Training set: (580, 9) (580,)
Test set: (103, 9) (103,)<jupyter_text>Let's determine our optimal value of K for the number of nearest neighbors. A K value that is too low will capture too much noise (overfit) while one that is too large will be over-generalized. Below, we determine our optimal K value is 8, which has a cross-validation accuracy of 0.97.<jupyter_code># Determine optimal k value. 
best_score = 0.0
acc_list=[]

for k in range(3,15):
    
    clf_knn = KNeighborsClassifier(n_neighbors = k, algorithm='auto')
    
    # using 10-fold cross validation for scoring the classifier's accuracy
    scores = cross_val_score(clf_knn, X, y, cv=10)
    score = scores.mean()
    acc_list.append(score)
    
    if score > best_score:
        best_score=score
        best_clf = clf_knn
        best_k = k
        
print("Best K is :",best_k,"| Cross validation Accuracy :",best_score)
clf_knn = best_clf<jupyter_output>Best K is : 8 | Cross validation Accuracy : 0.9708866155157715<jupyter_text>Let's plot accuracy across varying K values. We see our optimal K value, the one with the highest accuracy, is 8.<jupyter_code># Plot accuracy of various K values. 
plt.plot(range(3,15),acc_list, c="g")
plt.xlabel('K')
plt.ylabel('Cross-Validation Accuracy')
plt.title("Accuracy by K Value")
plt.show()
# Train model using our algorithm above with optimal value of K of 8
clf_knn.fit(X_train,y_train)
# Make predictions on test set using our model
y_hat_knn = clf_knn.predict(X_test)
y_hat_knn[0:5]<jupyter_output><empty_output><jupyter_text>Evaluating KNN Performance Let's calculate the confusion matrix to evaluate model performance.<jupyter_code># Calculate confusion matrix
cm = confusion_matrix(y_test, y_hat_knn,labels=["Benign","Malignant"]) 
print(cm)<jupyter_output>[[66  0]
 [ 1 36]]<jupyter_text>Let's plot a heatmap to make it easier to visualize and interpret:Top left is true negative (TN) Top right is false positive (FP)Bottom left is false negative (FN)Bottom right is true postivie (TP)<jupyter_code># Plot heatmap of confusion matrix
sns.heatmap(cm, annot=True)
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.title("KNN Confusion Matrix")
plt.show()<jupyter_output><empty_output><jupyter_text>Let's generate our confustion matrix report to evaluate model performance. Remember that:Precision is the ratio of true postives to all positive predicted by model, Precision = TP / (TP + FP).Recall is the ratio of true positives to all positives actually in our data set, Recall = TP / (TP + FN).F1-score is the harmonic mean of preicision and recall.<jupyter_code># Confusion matrix report
np.set_printoptions(precision=2)
print (classification_report(y_test, y_hat_knn))<jupyter_output>precision    recall  f1-score   support

      Benign       0.99      1.00      0.99        66
   Malignant       1.00      0.97      0.99        37

   micro avg       0.99      0.99      0.99       103
   macro avg       0.99      0.99      0.99       103
weighted avg       0.99      0.99      0.99       103<jupyter_text>We can also calculate jaccard similarity score and f1-score automaically using the jaccard_similarity_score and f1_score functions, respectively. Jaccard score is the intersection divided by the union of the two labeled sets (the test and fitted set). F1-score is the harmonic mean of precision and recall as we saw above.<jupyter_code># Jaccard similarity score for KNN model for test set
jaccard_knn = jaccard_similarity_score(y_test, y_hat_knn)
jaccard_knn
# F1 score for KNN model for test set
f1_knn = f1_score(y_test, y_hat_knn, average='weighted')
f1_knn<jupyter_output><empty_output><jupyter_text>Part 6: Decision Tree Before we train and test our decision tree, let's determine our max depth that will yield the highest accuracy. When training our decision tree, we'll focus here on reducing entropy as much as possible (maximizing information gain) in each node of the tree. We see below that our max depth that will achieve the highest accuracy is 4. We see in our plot below that accuracy drops off after this point.<jupyter_code># Test max depths from 1 to 19
depth_range = range(1, 20)

# Create empty array for jaccard and f1 scores 
jaccard_similarity_score_ = []
f1_score_ = []

# Use for loop to train decision tree using increasing max depth values.  
# Make predictions using test set, append jaccard and F1 score to arrays created above
for d in depth_range:
    dec_tree = DecisionTreeClassifier(criterion = 'entropy', max_depth = d)
    dec_tree.fit(X_train, y_train)
    y_hat_tree = dec_tree.predict(X_test)
    jaccard_similarity_score_.append(jaccard_similarity_score(y_test, y_hat_tree))
    f1_score_.append(f1_score(y_test, y_hat_tree, average = 'weighted'))
# Create dataframe with jaccard and F1 accuaracy values to determine best max depth. We see the best max depth is d = 4. 
result = pd.DataFrame([jaccard_similarity_score_, f1_score_], index = ['Jaccard', 'F1'], columns = ['d = 1','d = 2','d = 3','d = 4','d = 5','d = 6','d = 7','d = 8','d = 9','d = 10', 'd = 11', 'd = 12', 'd = 13', 'd = 14', 'd = 15', 'd = 16', 'd = 17', 'd = 18', 'd = 19'])
result.columns.name = 'Evaluation Metrics'
result
# Plot accuracy of various max depths, jaccard score in blue and F1 score in red. 
plt.plot(range(1,20),jaccard_similarity_score_)
plt.plot(range(1,20),f1_score_, c='r')
plt.xlabel('Max Depth')
plt.ylabel('Accuracy')
plt.title("Accuracy by Max Depth")
plt.show()
# Set tree algorithm to max depth 4
dec_tree = DecisionTreeClassifier(criterion="entropy", max_depth = 4)
dec_tree
# Train decision tree
dec_tree.fit(X_train,y_train)
# Make prediction on test set using our model
y_hat_tree = dec_tree.predict(X_test)
y_hat_tree[0:5]<jupyter_output><empty_output><jupyter_text>Let's visualize our decision tree, which has a depth of 4 nodes. Keep in mind that values for our features are normalized. Our initial node is a criterion for UnifSize, which splits into criteria for BareNuc and UnifSize. We can continue to follow the nodes and branches down, with each step maximizing reduction in entropy (maximizing information gain).<jupyter_code># Visualize decision tree using tree.export_graphviz
dot_data = tree.export_graphviz(dec_tree, out_file=None, 
                     feature_names=['Clump', 'UnifSize', 'UnifShape', 'MargAdh', 'SingEpiSize', 'BareNuc', 'BlandChrom', 'NormNucl', 'Mit'],  
                     class_names='Class',  
                     filled=True, rounded=True,  
                     special_characters=True) 

graph = pydotplus.graph_from_dot_data(dot_data)
graph.set_size('"12,12!"')
gvz_graph = graphviz.Source(graph.to_string())

gvz_graph<jupyter_output><empty_output><jupyter_text>Evaluating Decision Tree Performance Let's calculate the confusion matrix to evaluate model performance.<jupyter_code># Calculate confusion matrix
cm = confusion_matrix(y_test, y_hat_tree,labels=["Benign","Malignant"]) 
print(cm)<jupyter_output>[[65  1]
 [ 0 37]]<jupyter_text>Let's plot a heatmap to make it easier to visualize and interpret:Top left is true negative (TN) Top right is false positive (FP)Bottom left is false negative (FN)Bottom right is true postivie (TP)<jupyter_code># Plot heatmap of confusion matrix
sns.heatmap(cm, annot=True)
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.title("Decision Tree Confusion Matrix")
plt.show()<jupyter_output><empty_output><jupyter_text>Let's generate our confustion matrix report to evaluate model performance. Remember that:Precision is the ratio of true postives to all positive predicted by model, Precision = TP / (TP + FP).Recall is the ratio of true positives to all positives actually in our data set, Recall = TP / (TP + FN).F1-score is the harmonic mean of preicision and recall.<jupyter_code># Confusion matrix report
np.set_printoptions(precision=2)
print (classification_report(y_test, y_hat_tree))<jupyter_output>precision    recall  f1-score   support

      Benign       1.00      0.98      0.99        66
   Malignant       0.97      1.00      0.99        37

   micro avg       0.99      0.99      0.99       103
   macro avg       0.99      0.99      0.99       103
weighted avg       0.99      0.99      0.99       103<jupyter_text>We can also calculate jaccard similarity score and f1-score automaically using the jaccard_similarity_score and f1_score functions, respectively. Jaccard score is the intersection divided by the union of the two labeled sets (the test and fitted set). F1-score is the harmonic mean of precision and recall as we saw above.<jupyter_code># Jaccard similarity score for decision tree
jaccard_tree = jaccard_similarity_score(y_test, y_hat_tree)
jaccard_tree
# F1 score for decision tree
f1_tree = f1_score(y_test, y_hat_tree, average='weighted')
f1_tree<jupyter_output><empty_output><jupyter_text>Part 7: SVM For our SVM, let's first determine which kernel function - linear, polynomial, radial basis function (rbf), or sigmoid - generates the highest accuracy. We see that the rbf kernel function generates the highest F1 score of 1.<jupyter_code># Determine optimal kernel function

# Array of kernel functions
kernel_func = ['sigmoid', 'poly', 'rbf', 'linear']

# Empty array for accuracy score (F1 score)
accuracy_score = []

# For each kernel function, train SVM model, run prediction on test set, calculate F1 score and append it to accuracy_score array
for k in kernel_func:
    svc_model = SVC(kernel = k) 
    svc_model.fit(X_train, y_train)
    y_hat_svm = svc_model.predict(X_test)
    accuracy_score.append(f1_score(y_test, y_hat_svm, average = 'weighted'))

accuracy_score
# Bar plot of accuracy score for each kernel function

y_pos = np.arange(len(kernel_func))
plt.bar(y_pos, accuracy_score, align='center', alpha = 0.6)
plt.xticks(y_pos, kernel_func)
plt.xlabel('Kernel Functions')
plt.ylabel('Accuracy')
# Set SVM algorithm with rbf kernel function
svc_model = SVC(kernel = 'rbf')
# Train SVM model 
svc_model.fit(X_train, y_train)
svc_model
# Make prediction on test set using our model
y_hat_svm = svc_model.predict(X_test)
y_hat_svm [0:5]<jupyter_output><empty_output><jupyter_text>Evaluating SVM Performance Let's calculate the confusion matrix to evaluate model performance.<jupyter_code># Calculate confusion matrix
cm = confusion_matrix(y_test, y_hat_svm,labels=["Benign","Malignant"]) 
print(cm)<jupyter_output>[[66  0]
 [ 0 37]]<jupyter_text>Let's plot a heatmap to make it easier to visualize and interpret:Top left is true negative (TN) Top right is false positive (FP)Bottom left is false negative (FN)Bottom right is true postivie (TP)<jupyter_code># Plot heatmap of confusion matrix
sns.heatmap(cm, annot=True)
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.title("SVM Confusion Matrix")
plt.show()<jupyter_output><empty_output><jupyter_text>Let's generate our confustion matrix report to evaluate model performance. Remember that:Precision is the ratio of true postives to all positive predicted by model, Precision = TP / (TP + FP).Recall is the ratio of true positives to all positives actually in our data set, Recall = TP / (TP + FN).F1-score is the harmonic mean of preicision and recall.<jupyter_code># Confusion matrix report
np.set_printoptions(precision=2)
print (classification_report(y_test, y_hat_svm))<jupyter_output>precision    recall  f1-score   support

      Benign       1.00      1.00      1.00        66
   Malignant       1.00      1.00      1.00        37

   micro avg       1.00      1.00      1.00       103
   macro avg       1.00      1.00      1.00       103
weighted avg       1.00      1.00      1.00       103<jupyter_text>We can also calculate jaccard similarity score and f1-score automaically using the jaccard_similarity_score and f1_score functions, respectively. Jaccard score is the intersection divided by the union of the two labeled sets (the test and fitted set). F1-score is the harmonic mean of precision and recall as we saw above.<jupyter_code># Jaccard similarity score for SVM
jaccard_svm = jaccard_similarity_score(y_test, y_hat_svm)
jaccard_svm
# F1 score for SVM
f1_svm = f1_score(y_test, y_hat_svm, average='weighted')
f1_svm<jupyter_output><empty_output><jupyter_text>Part 8: Logistic Regression Logistic regression is best suited for binary categorical target variable like the one we have. Moreover, it offers the benefit of determining not just whether a cell sample is benign or malignant, but also the probabality. For our logistic regression, let's first determine which combination of regularization (to account for overfitting) and numerical solver to find parameters - newton-cg, lbfgs, liblinear, sag, saga - generates the highest accuracy score (log loss score). We find below that our highest accuracy is with regularization C = 0.001 and solver liblinear.<jupyter_code>solvers = ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga']
regularization_val = [0.1, 0.01, 0.001]
index = []
accuracy_score = []
iterations = 0

for index1, c in enumerate(regularization_val):
    for index2, solver in enumerate(solvers):
        index.append(index1 + index2 *5)
        iterations +=1
        lr_model = LogisticRegression(C = c, solver = solver)
        lr_model.fit(X_train, y_train)
        y_hat_lr = lr_model.predict(X_test)
        y_prob = lr_model.predict_proba(X_test)
        print('Test {}: Accuracy at C = {} when Solver = {} is : {}'.format(iterations, c, solver, log_loss(y_test, y_prob) ))
        accuracy_score.append(log_loss(y_test, y_prob))
                              
    print('\n')
# Visualize the above accuracy tests, with the peak at test 13, which corresponds to C = 0.001 and solver = liblinear
lr_prob = lr_model.predict_proba(X_test)
log_loss(y_test, lr_prob)
plt.plot(index, accuracy_score)
plt.xlabel('Parameter Value')
plt.ylabel('Testing Accuracy')
# Set logistic regression with optimal regularization of C = 0.001 and solver = 'liblinear'
lr_model = LogisticRegression(C = 0.001, solver = 'liblinear')
# Train logistic regression model
lr_model.fit(X_train, y_train)
lr_model
# Make prediction on test set using our model
y_hat_lr = lr_model.predict(X_test)
# Determine probabilities of loan classification using our test set. We'll need this for our logloss score. 
y_hat_lr_prob = lr_model.predict_proba(X_test)
y_hat_lr_prob[0:5]<jupyter_output><empty_output><jupyter_text>Evaluating Logistic Regression Performance Let's calculate the confusion matrix to evaluate model performance.<jupyter_code># Calculate confusion matrix
cm = confusion_matrix(y_test, y_hat_lr,labels=["Benign","Malignant"]) 
print(cm)<jupyter_output>[[66  0]
 [ 0 37]]<jupyter_text>Let's plot a heatmap to make it easier to visualize and interpret:Top left is true negative (TN) Top right is false positive (FP)Bottom left is false negative (FN)Bottom right is true postivie (TP)<jupyter_code># Plot heatmap of confusion matrix
sns.heatmap(cm, annot=True)
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.title("Logistic Regression Confustion Matrix")
plt.show()<jupyter_output><empty_output><jupyter_text>Let's generate our confustion matrix report to evaluate model performance. Remember that:Precision is the ratio of true postives to all positive predicted by model, Precision = TP / (TP + FP).Recall is the ratio of true positives to all positives actually in our data set, Recall = TP / (TP + FN).F1-score is the harmonic mean of preicision and recall.<jupyter_code># Confusion matrix report
np.set_printoptions(precision=2)
print (classification_report(y_test, y_hat_lr))<jupyter_output>precision    recall  f1-score   support

      Benign       1.00      1.00      1.00        66
   Malignant       1.00      1.00      1.00        37

   micro avg       1.00      1.00      1.00       103
   macro avg       1.00      1.00      1.00       103
weighted avg       1.00      1.00      1.00       103<jupyter_text>We can also calculate jaccard similarity score and f1-score automaically using the jaccard_similarity_score and f1_score functions, respectively. Jaccard score is the intersection divided by the union of the two labeled sets (the test and fitted set). F1-score is the harmonic mean of precision and recall as we saw above.<jupyter_code># Jaccard similarity score for logistic regression
jaccard_lr = jaccard_similarity_score(y_test, y_hat_lr)
jaccard_lr
# F1 score for logistic regression
f1_lr = f1_score(y_test, y_hat_lr, average='weighted')
f1_lr
# Logloss for logistic regression
logloss_lr = log_loss(y_test, y_hat_lr_prob) #Remember to use y_hat_lr_prob not just y_hat_lr
logloss_lr<jupyter_output><empty_output><jupyter_text>Evaluating Model Performance Head-to-Head Let's compare KNN, decision tree, SVM, and logistic regression head-to-head using our specified parameters for classifying loan status for this data set. Creating a data frame with our evaluation metrics, we see that SVM and logistic regression perform best, while decision tree is the weakest. Choosing a logistic regression model on our parameters to predict cancer diagnosis would perform best with the added benefit of predicting percentage likelihood that a cell sample is malignant.<jupyter_code># Create dataframe with evaluation metrics
evaluation = {"Jaccard":[jaccard_knn, jaccard_tree, jaccard_svm, jaccard_lr], 
             "F1-score":[f1_knn, f1_tree, f1_svm, f1_lr],
             "Log Loss":["NA", "NA", "NA", logloss_lr] 
             }
eval_df = pd.DataFrame(evaluation, columns=["Jaccard", "F1-score", "Log Loss"], index=["KNN", "Decision Tree", "SVM", "Logistic Regression"])
eval_df.columns.name = "Algorithm"
eval_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08259977036289622, 'fraction_numerical': 0.020075849831251522, 'mean_word_length': 5.024313561098302, 'pattern_counts': {'":': 3, '<': 142, '<?xml version=': 0, '>': 143, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '244528', 'license': 'MIT', 'n_tokens_mistral': 9322, 'n_tokens_neox': 8562, 'n_words': 3223, 'path': '4 Classification Models for Cancer Cell Samples .ipynb', 'repo_name': 'hrishipoola/Cancer_Cell_Samples_4_Classification_Models', 'size': '28741'}