<jupyter_start><jupyter_text>Lab 8 &ndash; DATA 3401 (Fall 2021) Lab Date: 11/12 Due Date 11/19 (before the beginning of lab) Lab DescriptionThe purpose of this lab is for you to become more familiar with pandas  Exercise 11. Download the `wdbc.names` and `wdbc.data` text files in the Lab8 folder on GitHub. These come from the UCI repository https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)1. Use the native `open` command of Python to display the txt file `wdbs.names`1. Read the dataset from the `wdbc.data` file into a variable called `data` using pandas<jupyter_code>import numpy as np
import pandas as pd
import matplotlib
from matplotlib import pyplot as plt
display = open('wdbc.names','r')
print(display.read())<jupyter_output>1. Title: Wisconsin Diagnostic Breast Cancer (WDBC)

2. Source Information

a) Creators: 

	Dr. <NAME>, General Surgery Dept., University of
	Wisconsin,  Clinical Sciences Center, Madison, WI 53792
	<EMAIL>

	<NAME>, Computer Sciences Dept., University of
	Wisconsin, 1210 West Dayton St., Madison, WI 53706
	<EMAIL>  608-262-6619

	<NAME>, Computer Sciences Dept., University of
	Wisconsin, 1210 West Dayton St., Madison, WI 53706
	<EMAIL> 

b) Donor: <NAME>

c) Date: November 1995

3. Past Usage:

first usage:

	<NAME>, <NAME> and <NAME> 
	Nuclear feature extraction for breast tumor diagnosis.
	IS&T/SPIE 1993 International Symposium on Electronic Imaging: Science
	and Technology, volume 1905, pages 861-870, San Jose, CA, 1993.

OR literature:

	<NAME>, <NAME> and <NAME>. 
	Breast cancer diagnosis and prognosis via linear programming. 
	Operations Research, 43(4), pag[...]<jupyter_text>Exercise 21. Display the data header using pandas1. Are there any columns that are not useful to analyzing this data?1. Decide what to do with the non-numerical data values, and modify your dataframe1. Extract the column of labels (Malignant or Benign, which should now be numerical values) from the dataframe and store it as a variable called `labels`<jupyter_code>df = pd.read_csv('wdbc.data',header=None)
df.head()<jupyter_output><empty_output><jupyter_text>We probably don't need to use the ID column since when we are analyzing the data we arent trying to point out specific data points we are trying to look at trends that multiple points have.<jupyter_code>df[1].replace(['M','B'],[1,0],inplace=True)
df
labels = df[1]
labels
new_df = df.copy()
new_df.pop(1)
new_df
new_df.pop(0)
new_df<jupyter_output><empty_output><jupyter_text>Exercise 31. Plot the data in 5 different ways of your choosing (i.e., plot column 2 vs. column 3 or column 8 vs. column 1, etc.)    1. Your plot should be a scatter plot and should color the points according to whether the cancer is Malignant or Benign1. Now, create new variables `x` and `y` which are numpy arrays from your `data` and `labels` dataframes1. Using these, run the segment of code in the cell below which implements PCA on x (color code is green for benign and red for malignant)1. Compare this plot to the ones from step 1. Do you notice any difference? Do you think it would be easier to separate malignant and benign tumors based on one of these plots over the others?<jupyter_code>M_only = df[df[1] != 0]
B_only = df[df[1] != 1] 
ax = M_only.plot.scatter(x=1, y=4, color="DarkRed", label="Malignant")

B_only.plot.scatter(x=1, y=4, color="DarkGreen", label="Benign", ax=ax);
ax = M_only.plot.scatter(x=1, y=3, color="DarkRed", label="Malignant")

B_only.plot.scatter(x=1, y=3, color="DarkGreen", label="Benign", ax=ax);

df.plot.scatter(x=3, y=5, c=1, cmap = 'RdBu_r', s=50);
df.plot.scatter(x=2, y=8, c=1, cmap = 'RdBu_r', s=50);
df.plot.scatter(x=10, y=2, c=1, cmap = 'Pastel2_r', s=50);
from sklearn.decomposition import PCA
import matplotlib
from matplotlib import pyplot as plt
x = np.array(new_df)
y = np.array(labels)
plt.figure()
PCA3=PCA(n_components=2)
XPCA = PCA3.fit_transform(x)
plt.plot(XPCA[y==0,0],XPCA[y==0,1],'g.')
plt.plot(XPCA[y==1,0],XPCA[y==1,1],'r.')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11210317460317461, 'fraction_numerical': 0.036210317460317464, 'mean_word_length': 5.119878603945372, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '357324', 'license': 'MIT', 'n_tokens_mistral': 1493, 'n_tokens_neox': 1344, 'n_words': 512, 'path': 'Labs/Lab8/Lab8.ipynb', 'repo_name': 'Ryan-Bui/DATA3401', 'size': '4155'}