<jupyter_start><jupyter_text>Breast Cancer Detection- categories: [machine_learning, scikit-learn, logistic_regression, kNN, SVM, decision_tree, random_forest, adaboost, naive_bayes, quadratic_discriminant_analysis, neural_network, gaussian_process, breast_cancer_detection, structured_data, uci_dataset] We will look at application of Machine Learning algorithms to one of the data sets from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php) to classify whether a set of readings from clinical reports are positive for breast cancer or not.This is one of the easier datasets to process since all the features have integer values.We will use the [scikit-learn](https://scikit-learn.org/stable/) algorithms to process this dataset.  Dataset We will use the [Breast Cancer Wisconsin (Original) Data Set](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Original%29).  Details about the Dataset - Data Set Characteristics: Multivariate- Attribute Characteristics: Integer- Associated Tasks: Classification- Number of instances: 699- Number of attributes: 10- Area: Life  Attribute Information 1. Sample code number: id number2. Clump Thickness: 1 - 103. Uniformity of Cell Size: 1 - 104. Uniformity of Cell Shape: 1 - 105. Marginal Adhesion: 1 - 106. Single Epithelial Cell Size: 1 - 107. Bare Nuclei: 1 - 108. Bland Chromatin: 1 - 109. Normal Nucleoli: 1 - 1010. Mitoses: 1 - 1011. Class: (2 for benign, 4 for malignant)  Class Distribution - Benign: 458 (65.5%)- Malignant: 241 (34.5%)  Missing Values There are 16 instances in Groups 1 to 6 that contain a single missing (i.e., unavailable) attribute value, denoted by "?".    Prepare Dataset for Machine Learning  Load Data<jupyter_code>import pandas as pd

df = pd.read_csv("data/breast-cancer-wisconsin.data", header=None) #file contains no header info
print(f"Read in {len(df)} rows")
df.head()<jupyter_output>Read in 699 rows<jupyter_text>Deal with Missing values There are 16 missing attribute values and we need to deal with them as they have the value '?' in them. We need integer values for processing the data. So let's deal with the missing values.<jupyter_code>df.replace("?", 10000, inplace=True) #10,000 is way beyond the range of columns provided so acts as an outlier<jupyter_output><empty_output><jupyter_text>Feature Selection The first column in the dataset is defined to be a "sample code number". This column should not have any bearing on the outcome of the test. So we will drop this column from our dataframe.<jupyter_code>df.drop([0], 1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Split dataset into Train and Test<jupyter_code>import numpy as np
from sklearn.model_selection import train_test_split

X = np.array(df.drop([10], 1)) #last column contains label, so ignore it when creating X
y = np.array(df[10]) #last column contains label which is our y

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=43)<jupyter_output><empty_output><jupyter_text>Train Models We will train different models from the sklearn library on this data and see how each one performs.<jupyter_code>names = ["Logistic Regression", "Nearest Neighbors", "Linear SVM", "RBF SVM", "Gaussian Process",
         "Decision Tree", "Random Forest", "Neural Net", "AdaBoost",
         "Naive Bayes", "QDA"]
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
classifiers = [
    LogisticRegression(max_iter=300),
    KNeighborsClassifier(),
    SVC(kernel="linear", C=0.025),
    SVC(gamma=2, C=1),
    GaussianProcessClassifier(1.0 * RBF(1.0)),
    DecisionTreeClassifier(max_depth=5, random_state=43),
    RandomForestClassifier(max_depth=5, random_state=43),
    MLPClassifier(alpha=1, max_iter=1000),
    AdaBoostClassifier(),
    GaussianNB(),
    QuadraticDiscriminantAnalysis()]
# iterate over classifiers
for name, clf in zip(names, classifiers):
    clf.fit(X_train, y_train)
    score = clf.score(X_test, y_test)
    print(f"Accuracy of {name} Classifier is:{score}")<jupyter_output>Accuracy of Logistic Regression Classifier is:0.9714285714285714
Accuracy of Nearest Neighbors Classifier is:0.9542857142857143
Accuracy of Linear SVM Classifier is:0.96
Accuracy of RBF SVM Classifier is:0.84
Accuracy of Gaussian Process Classifier is:0.96
Accuracy of Decision Tree Classifier is:0.9142857142857143
Accuracy of Random Forest Classifier is:0.9771428571428571
Accuracy of Neural Net Classifier is:0.8857142857142857
Accuracy of AdaBoost Classifier is:0.9542857142857143
Accuracy of Naive Bayes Classifier is:0.96
Accuracy of QDA Classifier is:0.9542857142857143<jupyter_text>Now lets see if we can improve the performance of these algorithms. Standardize features by removing the mean and scaling to unit variance. Will be used with some of the algorithms.<jupyter_code>from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.fit_transform(X_test)<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression

lr_model = LogisticRegression(random_state=43, class_weight={4: 2}, max_iter=500, n_jobs=-1)
lr_model.fit(X_train, y_train)
lr_accuracy = lr_model.score(X_test, y_test)
print(f"Accuracy of Logistic Regression Classifier is:{lr_accuracy}")<jupyter_output>Accuracy of Logistic Regression Classifier is:0.9885714285714285<jupyter_text>We managed an improvement of over 1.7% in the overall accuracy score from 97.14% to 98.85%.  K Nearest Neighbor Algorithm<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

knn_model = KNeighborsClassifier(n_neighbors=3, n_jobs=-1)
knn_model.fit(X_train_scaled, y_train)
knn_accuracy = knn_model.score(X_test_scaled, y_test)
print(f"Accuracy of kNN Classifier is:{knn_accuracy}")<jupyter_output>Accuracy of kNN Classifier is:0.9657142857142857<jupyter_text>We managed an improvement of over 1.1% in the overall accuracy score from 95.42% to 96.57%.  Linear Support Vector Machines (SVM)<jupyter_code>from sklearn import svm

svm_model = svm.SVC(random_state=43, kernel='linear', class_weight={4: 2})
svm_model.fit(X_train, y_train)

svm_accuracy = svm_model.score(X_test, y_test)
print(f"Accuracy of Linear SVM Classifier is:{svm_accuracy}")<jupyter_output>Accuracy of Linear SVM Classifier is:0.9885714285714285<jupyter_text>We managed an improvement of over 2.8% in the overall accuracy score from 96.00% to 98.85%.  RBF Support Vector Machines (SVM)<jupyter_code>from sklearn import svm
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.model_selection import GridSearchCV

C_range = np.logspace(-2, 10, 13)
gamma_range = np.logspace(-9, 3, 13)
param_grid = dict(gamma=gamma_range, C=C_range)
cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=43)
grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv)
grid.fit(X_train_scaled, y_train)

print("The best parameters are %s with a score of %0.2f"
      % (grid.best_params_, grid.best_score_))
rbf_svm_model = svm.SVC(gamma=0.01, C=100, class_weight={4: 2})
rbf_svm_model.fit(X_train_scaled, y_train)

rbf_svm_accuracy = rbf_svm_model.score(X_test_scaled, y_test)
print(f"Accuracy of RBF SVM Classifier is:{rbf_svm_accuracy}")<jupyter_output>Accuracy of RBF SVM Classifier is:0.9828571428571429<jupyter_text>We managed an improvement of over 14.2% in the overall accuracy score from 84.00% to 98.28%.  Gaussian Process Classifier<jupyter_code>from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF

kernel = 1.0 * RBF(1.0)
gpc_model = GaussianProcessClassifier(kernel, random_state=43, max_iter_predict=1000, n_jobs=-1)
gpc_model.fit(X_train_scaled, y_train)

gpc_accuracy = gpc_model.score(X_test_scaled, y_test)
print(f"Accuracy of Gaussian Process Classifier is:{gpc_accuracy}")<jupyter_output>Accuracy of Gaussian Process Classifier is:0.9657142857142857<jupyter_text>We managed an improvement of over 0.5% in the overall accuracy score from 96.00% to 96.57%.  Decision Trees<jupyter_code>from sklearn import tree

max_depth_range = np.linspace(1, 10, 10)
param_grid = dict(max_depth=max_depth_range)
cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=43)
grid = GridSearchCV(DecisionTreeClassifier(class_weight={4: 2}), param_grid=param_grid, cv=cv)
grid.fit(X_train, y_train)

print("The best parameters are %s with a score of %0.2f"
      % (grid.best_params_, grid.best_score_))


tree_model = tree.DecisionTreeClassifier(class_weight={4: 2}, max_depth=4, random_state=43)
tree_model.fit(X_train, y_train)

tree_accuracy = tree_model.score(X_test, y_test)
print(f"Accuracy of Decision Tree Classifier is:{tree_accuracy}")<jupyter_output>Accuracy of Decision Tree Classifier is:0.96<jupyter_text>We managed an improvement of over 4.58% in the overall accuracy score from 91.42% to 96.00%.  Random Forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier

max_depth_range = np.linspace(1, 10, 10)
max_features_range = np.arange(1, 10, 1)
param_grid = dict(max_depth=max_depth_range, max_features=max_features_range)
cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=43)
grid = GridSearchCV(RandomForestClassifier(class_weight={4: 2}, random_state=43), param_grid=param_grid, cv=cv)
grid.fit(X_train, y_train)

print("The best parameters are %s with a score of %0.2f"
      % (grid.best_params_, grid.best_score_))
from sklearn.ensemble import RandomForestClassifier

rf_model = RandomForestClassifier(class_weight={4: 2}, max_depth=4, n_estimators=300, max_features=2, random_state=43, n_jobs=-1)
rf_model.fit(X_train, y_train)

rf_accuracy = rf_model.score(X_test, y_test)
print(f"Accuracy of Random Forest Classifier is:{rf_accuracy}")<jupyter_output>Accuracy of Random Forest Classifier is:0.9771428571428571<jupyter_text>We are unable to improve score from 97.71%.  Neural Network<jupyter_code>from sklearn.neural_network import MLPClassifier

nn_model = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(100,), random_state=43, max_iter=1000, learning_rate='adaptive')
nn_model.fit(X_train_scaled, y_train)
nn_accuracy = nn_model.score(X_test_scaled, y_test)
print(f"Accuracy of MLP Classifier is:{nn_accuracy}")<jupyter_output>Accuracy of MLP Classifier is:0.9428571428571428<jupyter_text>We managed an improvement of over 5.85% in the overall accuracy score from 88.57% to 94.42%.  AdaBoost<jupyter_code>from sklearn.ensemble import AdaBoostClassifier

ada_model = AdaBoostClassifier(random_state=43, n_estimators=100)
ada_model.fit(X_train, y_train)
ada_accuracy = ada_model.score(X_test, y_test)
print(f"Accuracy of Ada Boost Classifier is:{ada_accuracy}")<jupyter_output>Accuracy of Ada Boost Classifier is:0.96<jupyter_text>We managed an improvement of over 0.58% in the overall accuracy score from 95.42% to 96.00%.  Gaussian Naive Bayes<jupyter_code>from sklearn.naive_bayes import GaussianNB

gnb_model = GaussianNB()
gnb_model.fit(X_train, y_train)
gnb_accuracy = gnb_model.score(X_test, y_test)
print(f"Accuracy of Gaussian Naive Bayes Classifier is:{gnb_accuracy}")<jupyter_output>Accuracy of Gaussian Naive Bayes Classifier is:0.96<jupyter_text>We are unable to improve score from 96.00%.  Quadratic Discriminant Analysis<jupyter_code>from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

qda_model = QuadraticDiscriminantAnalysis()
qda_model.fit(X_train, y_train)
qda_accuracy = qda_model.score(X_test, y_test)
print(f"Accuracy of Quadratic Discriminant Analysis Classifier is:{qda_accuracy}")<jupyter_output>Accuracy of Quadratic Discriminant Analysis Classifier is:0.9542857142857143
{'fraction_non_alphanumeric': 0.088112232340635, 'fraction_numerical': 0.049224710804824025, 'mean_word_length': 6.814102564102564, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '85018', 'license': 'Apache-2.0', 'n_tokens_mistral': 4600, 'n_tokens_neox': 4124, 'n_words': 1125, 'path': '_notebooks/2020-05-31-Breast Cancer Detection.ipynb', 'repo_name': 'amitpphatak/amitpphatak-website', 'size': '12189'}