<jupyter_start><jupyter_text>Phase 3 AssignmentKMeans algorithm Dataset: breast cancer data from the phase 1<jupyter_code>#import all required libraries

%matplotlib inline
import pandas as pd
import matplotlib.pylab as plt
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler<jupyter_output><empty_output><jupyter_text>Load and print head of dataset<jupyter_code>df = pd.read_csv('../data/BreastCancerWisconsin.csv') 
print(df[20:30])
df['A7'] = df['A7'].replace('?', np.NaN)
df['A7'] = pd.to_numeric(df['A7'])
df = df.fillna(df.mean(skipna=True))
print(df[20:30])<jupyter_output>Scn  A2  A3  A4  A5  A6  A7  A8  A9  A10  CLASS
20  1054590   7   3   2  10   5  10   5   4    4      4
21  1054593  10   5   5   3   6   7   7  10    1      4
22  1056784   3   1   1   1   2   1   2   1    1      2
23  1057013   8   4   5   1   2   ?   7   3    1      4
24  1059552   1   1   1   1   2   1   3   1    1      2
25  1065726   5   2   3   4   2   7   3   6    1      4
26  1066373   3   2   1   1   1   1   2   1    1      2
27  1066979   5   1   1   1   2   1   2   1    1      2
28  1067444   2   1   1   1   2   1   2   1    1      2
29  1070935   1   1   3   1   2   1   1   1    1      2
        Scn  A2  A3  A4  A5  A6         A7  A8  A9  A10  CLASS
20  1054590   7   3   2  10   5  10.000000   5   4    4      4
21  1054593  10   5   5   3   6   7.000000   7  10    1      4
22  1056784   3   1   1   1   2   1.000000   2   1    1      2
23  1057013   8   4   5   1   2   3.544656   7   3    1      4
24  1059552   1   1   1   1   2   1.000000   3   1    1      2
25  10[...]<jupyter_text>Get only A2-A10 colums from Dataset<jupyter_code>df_attributes = df.iloc[:, 1:10]
print(df_attributes.head(10))<jupyter_output>A2  A3  A4  A5  A6    A7  A8  A9  A10
0   5   1   1   1   2   1.0   3   1    1
1   5   4   4   5   7  10.0   3   2    1
2   3   1   1   1   2   2.0   3   1    1
3   6   8   8   1   3   4.0   3   7    1
4   4   1   1   3   2   1.0   3   1    1
5   8  10  10   8   7  10.0   9   7    1
6   1   1   1   1   2  10.0   3   1    1
7   2   1   2   1   2   1.0   3   1    1
8   2   1   1   1   2   1.0   1   1    5
9   4   2   1   1   2   1.0   2   1    1<jupyter_text>Cluster  Create KMeans model for n_cluster = 2 and iteration of 500 and initial centroids of 20Fit model to the dataCalculate centroids<jupyter_code>model = KMeans(n_clusters=2, n_init = 20, max_iter = 500)
# fit kmeans object to data df
model.fit(df_attributes)
labels = model.predict(df_attributes)
centroids = model.cluster_centers_<jupyter_output><empty_output><jupyter_text>Below is the values for labels and the Frequency of each Value<jupyter_code>print(labels)
value, count = np.unique(labels, return_counts=True)
np.asarray((value, count)).T<jupyter_output>[0 1 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1
 0 1 1 1 1 1 1 1 0 1 0 0 1 0 0 1 1 1 1 1 0 1 0 1 0 1 0 0 0 0 1 1 0 0 1 0 1
 1 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 1 0 1 0
 1 1 1 0 0 0 1 0 0 0 0 1 1 1 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0
 0 1 0 1 1 0 0 1 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1 1 0 1 0 1 0 0 0 1 1
 0 1 1 1 0 1 1 0 0 0 0 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 0 1 1 1 0 0 1 0 0 1
 0 1 1 0 1 1 0 1 1 1 0 1 0 0 1 1 1 1 0 0 0 0 0 0 1 1 0 0 0 1 1 1 1 1 0 0 0
 1 1 1 1 1 1 0 1 1 1 0 1 0 1 0 0 0 0 0 0 1 0 0 1 1 1 1 1 0 1 1 0 0 1 1 0 1
 1 0 0 1 1 0 1 0 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 1 1 0 0 1 0 0 1 0 1 1 1 0 0
 1 1 0 1 0 0 1 1 0 0 0 1 0 0 0 0 1 0 0 1 1 0 0 0 1 1 1 1 1 0 0 0 0 1 1 0 0
 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0 0
 0 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0
 0 1 1 0 0 0 1 1 0 0 1 0 [...]<jupyter_text>Below is the values for Centroids<jupyter_code>print(centroids)<jupyter_output>[[3.0472103  1.30257511 1.44635193 1.34334764 2.08798283 1.38000113
  2.10515021 1.26180258 1.10944206]
 [7.15879828 6.79828326 6.72961373 5.73390558 5.472103   7.87396553
  6.10300429 6.07725322 2.54935622]]<jupyter_text>Shape of Centoids should be 2 x 9<jupyter_code>print(centroids.shape)<jupyter_output>(2, 9)<jupyter_text>DataFrame -Create a new column in your dataframe and add Kmeans labels. -Change labels values so that 0 becomes 2 and 1 becomes 4.-Print the first 15 records from your dataframe<jupyter_code>#Add CLASS back to Dataframe and add Labels as a Column to the Dataframe
df_attributes = df.iloc[:,1:11]
df_attributes['labels'] = labels

# Replace values of the labels field (0 to 2; 1 to 4)

df_attributes['labels'] = df_attributes['labels'].map({0: 2, 1: 4})
print(df_attributes.head(15))<jupyter_output>A2  A3  A4  A5  A6    A7  A8  A9  A10  CLASS  labels
0    5   1   1   1   2   1.0   3   1    1      2       2
1    5   4   4   5   7  10.0   3   2    1      2       4
2    3   1   1   1   2   2.0   3   1    1      2       2
3    6   8   8   1   3   4.0   3   7    1      2       4
4    4   1   1   3   2   1.0   3   1    1      2       2
5    8  10  10   8   7  10.0   9   7    1      4       4
6    1   1   1   1   2  10.0   3   1    1      2       2
7    2   1   2   1   2   1.0   3   1    1      2       2
8    2   1   1   1   2   1.0   1   1    5      2       2
9    4   2   1   1   2   1.0   2   1    1      2       2
10   1   1   1   1   1   1.0   3   1    1      2       2
11   2   1   1   1   2   1.0   2   1    1      2       2
12   5   3   3   3   2   3.0   4   4    1      4       2
13   1   1   1   1   2   3.0   3   1    1      2       2
14   8   7   5  10   7   9.0   5   5    4      4       4<jupyter_text>Error Rate Function Create Error Rate FunctionPrint out all rates<jupyter_code>def ErrorRate(label_values, class_values):

    #Create a dataframe to hold our values
    combine_df = pd.DataFrame()
    combine_df['labels'] = label_values
    combine_df['class_values'] = class_values

    b = len(combine_df[(combine_df['labels'] == 4) & (combine_df['class_values'] == 2)])
    class_2 = len(combine_df[(combine_df['labels'] == 2)])
               
    benign = b / class_2
               
    m = len(combine_df[(combine_df['labels'] == 2) & (combine_df['class_values'] == 4)])
    class_4 = len(combine_df[(combine_df['labels'] == 4)])
               
    malign = m / class_4
               
               
    t = len(combine_df[(combine_df['labels']) != (combine_df['class_values'])])
    total_records = len(combine_df)
    
    total_error = t / total_records
               
    return benign, malign, total_error
                         
    
#Create label values and class values to pass into function    
label_values = df_attributes['labels']
class_values = df_attributes['CLASS']

benign, malign, total_error = ErrorRate(label_values, class_values)
print("Error Rate for Benign:", benign) 
print("Error Rate for Malign:", malign)
print("Total Error Rate:", total_error)<jupyter_output>Error Rate for Benign: 0.023605150214592276
Error Rate for Malign: 0.0815450643776824
Total Error Rate: 0.04291845493562232<jupyter_text>Density Curve for "CLASS" and "labels"<jupyter_code>from pandas.plotting import scatter_matrix
df_curve=df_attributes.iloc[:, 9:11]

ax = df_curve.plot.kde(figsize=(15,8))
plt.title('Density Curve for "CLASS" and "labels"')
plt.show()<jupyter_output><empty_output><jupyter_text>Report Statement: The model we constructed resulted in small benign, malign, and total error rates. Additionally, when comparing the density curve of the two variables, “CLASS” and “Labels”, we find that they looked identical. Lastly, the two variables have very similar standard deviations and distributions, also shown by the box plot (Given reference below from phase-2). With this analysis we can determine that the model is accurate and results in a valid analysis.We would need to evaluate the level of acceptable accuracy. We might find that although the model accurate it does not meet the level of accuracy we need. We could understand its accuracy level by comparing the performance of other similar models. We might find a more accurate or acceptable model by changing model parameters or creating a different model. Data science experience and testing the model with data overtime would also help us understand its level of accuracy. However, practically speaking, the model would be acceptable.<jupyter_code>#Create a box plot of the dataframe
import seaborn as sns
plt.figure(figsize=(15,8))
ax=sns.boxplot(x="variable", y="value", data=pd.melt(df_attributes))
plt.title('Box Plot of Data Frame for each Variable')
plt.xlabel('Data Variables')
plt.ylabel('Value')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06509821836455003, 'fraction_numerical': 0.16788487894015533, 'mean_word_length': 1.9504716981132075, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 110, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '406473', 'license': 'MIT', 'n_tokens_mistral': 4208, 'n_tokens_neox': 3372, 'n_words': 1516, 'path': 'code/FinalAssigment-Phase3.ipynb', 'repo_name': 'iu-data-science-python-i590-Group1/final-project-team-python-byte', 'size': '8756'}