<jupyter_start><jupyter_text>Import Datadf_data contains tumor data<jupyter_code>names =['ID','Diagnosis','radius1','texture1','perimeter1','area1','smoothness1','compactness1','concavity1','concave points1','symmetry1','fractal dimension1','radius2','texture2','perimeter2','area2','smoothness2','compactness2','concavity2','concave points2','symmetry2','fractal dimension2','radius3','texture3','perimeter3','area3','smoothness3','compactness3','concavity3','concave points3','symmetry3','fractal dimension3']
df_data = pd.read_table('wdbc.data.txt', sep=',',names = names)

#df_names = pd.read_clipboard('wdbc.names.txt')
df_data.head()<jupyter_output><empty_output><jupyter_text>Malignant and Benign Tumor Radius DistributionTumors were separated based on the diagnosis.Each radius measurement was plotted. Radius 1-3 were plotted separately for each diagnosis & tumor.Generally, the range of radii for malignant tumors were higher for each measurement than the benign tumors.<jupyter_code>mask_M = df_data['Diagnosis'].map(lambda x: 'M' in str(x) if not '' else False)
malignant = df_data[mask_M]

mask_B = df_data['Diagnosis'].map(lambda x: 'B' in str(x) if not '' else False)
benign = df_data[mask_B]

fm, axm = plt.subplots()
malignant.boxplot(['radius1','radius2','radius3'])
axm.set_ylabel('radius')
axm.set_title('Malignant Tumor Size Distribution')

fb, axb = plt.subplots()
benign.boxplot(['radius1','radius2','radius3'])
axb.set_ylabel('radius')
axb.set_title('Benign Tumor Size Distribution')<jupyter_output><empty_output><jupyter_text>Assessing Balance of DatasetThere are 212 malignant and 357 benign tumors in the sample.<jupyter_code>diag = df_data['Diagnosis']
fig, ax = plt.subplots()
diag.value_counts().plot(ax=ax, title='Diagnoses Counts', kind='bar',color=['Green','Red'])
ax.set_ylabel('Frequency')
ax.set_xlabel('Diagnosis')<jupyter_output><empty_output><jupyter_text>Distribution of Various Features<jupyter_code># mean features calculated below (combined feature1-3)

df_data['mean_radius']=(df_data['radius1']+df_data['radius2']+df_data['radius3'])/3
df_data['mean_texture']=(df_data['texture1']+df_data['texture2']+df_data['texture3'])/3
df_data['mean_perimeter']=(df_data['perimeter1']+df_data['perimeter2']+df_data['perimeter3'])/3
df_data['mean_area']=(df_data['area1']+df_data['area2']+df_data['area3'])/3
df_data['mean_smoothness']=(df_data['smoothness1']+df_data['smoothness2']+df_data['smoothness3'])/3
df_data['mean_compactness']=(df_data['compactness1']+df_data['compactness2']+df_data['compactness3'])/3
df_data['mean_concavity']=(df_data['concavity1']+df_data['concavity2']+df_data['concavity3'])/3
df_data['mean_concave_points']=(df_data['concave points1']+df_data['concave points2']+df_data['concave points3'])/3
df_data['mean_symmetry']=(df_data['symmetry1']+df_data['symmetry2']+df_data['symmetry3'])/3
df_data['mean_fractal_dimension']=(df_data['fractal dimension1']+df_data['fractal dimension2']+df_data['fractal dimension3'])/3
df_data.head()<jupyter_output><empty_output><jupyter_text>Mean Distributions (Malignant vs Benign)Malignant tumors plotted in RedBenign tumors plotted in GreenVariables with significant overlap for Malignant and Benign tumors might suggest that it would be difficult to use that variable to predict malignancy.Low level of correlation with diagnosis.<jupyter_code>mal = df_data['Diagnosis'].map(lambda x: 'M' in str(x) if not '' else False)
df_mal = df_data[mal]
ben = df_data['Diagnosis'].map(lambda x: 'B' in str(x) if not '' else False)
df_ben = df_data[ben]

# RADIUS
plt.figure("Radius")
sns.distplot(df_mal['mean_radius'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_radius'], label='Benign', color='green')
plt.title('Mean Radius Distribution')
plt.legend()
plt.show()

# TEXTURE
plt.figure("Texture")
sns.distplot(df_mal['mean_texture'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_texture'], label='Benign', color='green')
plt.title('Mean Texture Distribution')
plt.legend()
plt.show()

# Perimeter
plt.figure("Perimeter")
sns.distplot(df_mal['mean_perimeter'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_perimeter'], label='Benign', color='green')
plt.title('Mean Perimeter Distribution')
plt.legend()
plt.show()

# AREA
plt.figure("Area")
sns.distplot(df_mal['mean_area'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_area'], label='Benign', color='green')
plt.title('Mean Area Distribution')
plt.legend()
plt.show()

# SMOOTHNESS
plt.figure("Smoothness")
sns.distplot(df_mal['mean_smoothness'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_smoothness'], label='Benign', color='green')
plt.title('Mean Smoothness Distribution')
plt.legend()
plt.show()

# COMPACTNESS
plt.figure("Compactness")
sns.distplot(df_mal['mean_compactness'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_compactness'], label='Benign', color='green')
plt.title('Mean Compactness Distribution')
plt.legend()
plt.show()

# CONCAVITY
plt.figure("Concavity")
sns.distplot(df_mal['mean_concavity'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_concavity'], label='Benign', color='green')
plt.title('Mean Concavity Distribution')
plt.legend()
plt.show()

# CONCAVE POINTS
plt.figure("Concave Points")
sns.distplot(df_mal['mean_concave_points'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_concave_points'], label='Benign', color='green')
plt.title('Mean Concave Points Distribution')
plt.legend()
plt.show()

# SYMMETRY
plt.figure("Symmetry")
sns.distplot(df_mal['mean_symmetry'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_symmetry'], label='Benign', color='green')
plt.title('Mean Symmetry Distribution')
plt.legend()
plt.show()

# FRACTAL DIMENSION
plt.figure("Fractal Dimension")
sns.distplot(df_mal['mean_fractal_dimension'], label='Malignant', color='Red')
sns.distplot(df_ben['mean_fractal_dimension'], label='Benign', color='green')
plt.title('Mean Fractal Dimension Distribution')
plt.legend()
plt.show()<jupyter_output>/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "
/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>Visual AnalysisFeatures that appeared to show large difference in distribution, based on diagnois: Radius, Perimeter, Area, Compactness, Concavity, Concave PointsFeatures that showed some/little deviation: Texture, Smoothness, Symmetry, Fractal Dimension.  Correlation Heatmap<jupyter_code>df_corr = df_data.iloc[:,32:]
corr = df_corr.corr()
sns.heatmap(corr)
# Refined feature list
df_corr2 = df_data[['Diagnosis','mean_area','mean_compactness','mean_concave points']]
corr2 = df_corr2.corr()
sns.heatmap(corr2)<jupyter_output><empty_output><jupyter_text>Feature CorrelationIn general it would be better to have features that are less correlated.Including radius, perimeter, and area would be redundant because the three are highly correlated and have nearly the same amount of correlation for the other features. The same is true for symmetry and fractal dimension, where the heat map shows very similar correlations. Mean concave points also generally reflects mean concavity and shows similar correlation to other features.I would remove radius, perimeter, and symmetry. Area encompasses the size reflected by radius and perimeter.Other features that show low correlation were also removed to simplify the model(texture, smoothness, symmetry, fractal dimension)  Machine Learning Models Variable set up I first select features that I want to use to train the model.From the heatmaps above, I selected Area, Compactness, and Concave Points.The response variable is the Diagnosis.<jupyter_code>_data = df_data[['Diagnosis','mean_area','mean_compactness','mean_concave points']]

X = _data[['mean_area','mean_compactness','mean_concave points']]
y = _data['Diagnosis']

_data.sample(n=3, random_state=3)<jupyter_output><empty_output><jupyter_text>1 - Logistic RegressionThe first model will fit the data against a logistic regression. (Hopefully there is enough difference between the features to generate a decent model.<jupyter_code>clf = linear_model.LogisticRegression()
clf.fit(X,y)
print(clf.intercept_, clf.coef_)
clf.score(X,y)<jupyter_output><empty_output><jupyter_text>This is a good score!The chosen features fit relatively well with a logistic regression.This is likely a sufficient fit. Fewer variables allows for less opportunity for "cheating", but this uses enough features to get a decent score.<jupyter_code>l = model_selection.cross_val_score(clf,X,y,cv=20)
print(l.mean())
print(l.std())<jupyter_output>0.9110039226418538
0.06649655053999222<jupyter_text>The score produced above is within the range (<1 std away from mean) of the cross validated values. This suggests it was a sufficiently accurate model for predicting tumor malignancy.  2 - RandomForestClassifier<jupyter_code># Processed Data Frame
_data.head()<jupyter_output><empty_output><jupyter_text>The RandomForestClassifier only works with numeric data, so I first need to convert the Diagnosis results into numbers.The Malignant tumors will be set to 1 and Benign to 0.<jupyter_code># Encoder Selection
d = defaultdict(preprocessing.LabelEncoder)
# Apply encoder to categorial variables (Diagnosis: M-1)
data_encoded = _data.copy()
for col in ['Diagnosis']:
    data_encoded[[col]] = data_encoded[[col]].apply(lambda x: d[x.name].fit_transform(x))

data_encoded.head()<jupyter_output><empty_output><jupyter_text>I will use the RandomForestClassifier module<jupyter_code># Model Selection
clf = RandomForestClassifier()<jupyter_output><empty_output><jupyter_text>Defined a base set of parameters to test.This could need refinement to achieve a better model<jupyter_code># Initial parameters
parameters = {
    'n_estimators': [6, 8, 10, 12],
    'max_features': ['sqrt', 'log2'],
    'criterion': ['gini', 'entropy'],
    'max_depth': [2, 3, 5, 10, None],
    'min_samples_split': [2, 3, 5],
    'min_samples_leaf': [1,5,10]
}
# Search for best parameters using crossvalidation. `cv=3` means 3-fold crossvalidation
cv = GridSearchCV(clf, parameters, cv=3)  # takes ~30 sec. Could speed up with multiprocessing, `n_jobs=4` for 4 processes
cv.fit(data_encoded, y)
print('Best params: ', cv.best_params_)
print('Best score: ', cv.best_score_)<jupyter_output>Best params:  {'criterion': 'gini', 'max_depth': 2, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 6}
Best score:  1.0<jupyter_text>The best score is already 1.0 and the best parameters were the smallest in the range, but I will try much larger parameters to make sure this is actually working.<jupyter_code># Second parameters attempt (large)

param2 = {
    'n_estimators': [202,300],
    'max_features': ['sqrt', 'log2'],
    'criterion': ['gini', 'entropy'],
    'max_depth': [800, 1401],
    'min_samples_split': [500, 600],
    'min_samples_leaf': [1000, 1202]
}
cv2 = GridSearchCV(clf, param2, cv=3)  # takes ~30 sec. Could speed up with multiprocessing, `n_jobs=4` for 4 processes
cv2.fit(data_encoded, y)
print('Best params: ', cv2.best_params_)
print('Best score: ', cv2.best_score_)<jupyter_output>Best params:  {'criterion': 'gini', 'max_depth': 800, 'max_features': 'sqrt', 'min_samples_leaf': 1000, 'min_samples_split': 500, 'n_estimators': 202}
Best score:  0.6274165202108963<jupyter_text>As expected, the best score decreased all the way to 0.627...Now I will try a new range of parameters that are lower than the first attempt.<jupyter_code># Third parameters attempt (small)

param3 = {
    'n_estimators': [1, 3, 5, 7],
    'max_features': ['sqrt', 'log2'],
    'criterion': ['gini', 'entropy'],
    'max_depth': [1, 2, 3, None],
    'min_samples_split': [2, 3, 4],
    'min_samples_leaf': [1,2,3]
}
cv3 = GridSearchCV(clf, param3, cv=3)  # takes ~30 sec. Could speed up with multiprocessing, `n_jobs=4` for 4 processes
cv3.fit(data_encoded, y)
print('Best params: ', cv3.best_params_)
print('Best score: ', cv3.best_score_)<jupyter_output>Best params:  {'criterion': 'gini', 'max_depth': 1, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 5}
Best score:  1.0<jupyter_text>Like the first attempt, this set of parameters also produced a best score of 1.0. The first set was likely good enough.  Test/Train Splits<jupyter_code># Create test train split
X_train = data_encoded[::2]
X_test = data_encoded[1::2]
y_train = y[::2]
y_test = y[1::2]<jupyter_output><empty_output><jupyter_text>Created test/train splits using every other row of the original X and y dataframes.<jupyter_code># Initialize classifier using best parameters
# double stars mean "use this dictionary as the kwargs for the function"
clf = RandomForestClassifier(**cv.best_params_)

# fit/train it on the training data
clf.fit(X_train, y_train)
# predict y values from X_test
y_test_predicted = clf.predict(X_test)
# Print the accuracy score of your predicted y values
accuracy_score(y_test, y_test_predicted)
# Create a bar plot of relative feature importances of each feature
# You can access the feature importances with `clf.feature_importances_` 
# and the column names with `data_encoded.columns`f, ax = plt.subplots()

plt.figure('Feature Importances')
pd.DataFrame(clf.feature_importances_, index=data_encoded.columns).plot(kind='bar',color='Green',legend=False)
ax.set_ylabel('relative importance')
ax.set_xlabel('feature name')
plt.title('Feature importances')
plt.show()
clf.feature_importances_<jupyter_output><empty_output><jupyter_text>The feature importance values show how each value is used for the model. The compactness had an importance of 0 so it was unnecessary with the other features.<jupyter_code>lrfc = model_selection.cross_val_score(clf,X,y,cv=20)
print(lrfc.mean())
print(lrfc.std())<jupyter_output>0.9265257252326217
0.047748789543087554<jupyter_text>Either way, the best score was still very good.  Parameter implicationsBest parameters of first and second set tends towards the lower numbers, suggesting that smaller parameters could be preferred.Even when decreasing the parameters even smaller than the first parameter set (param3), the best score was still 1.Best used parameters were:{'criterion': 'gini', 'max_depth': 1, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'min_samples_split': 3, 'n_estimators': 3}However, both parameter sets 1 and 3 produced a best score of 1.0 so they are both sufficient.It does seem a bit unlikely that it could be a perfect 1.0 though.  Comparison of two models (LogRegression vs RandomForestClassifier)The two models both returned high scores (0.91 and 1.0). Clearly the RandomForestClassifier method was more accurate, but the Logistic Regression model could also be acceptable given its good score.The assumption that the data set followed a logistical trend may not have been accurate, resulting in a slightly lower score. On the other hand, the RandomForestClassifier model must have satisfied each category of the tree, using the provided parameters.<jupyter_code># The END - Happy Holidays!

#      ^
#     ^O^
#    ^O^O^
#   ^O^O^O^
#  ^O^O^O^O^
# ^O^O^O^O^O^
#      U<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13065131621499257, 'fraction_numerical': 0.022831640902917016, 'mean_word_length': 7.439956331877729, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '248182', 'license': 'MIT', 'n_tokens_mistral': 5410, 'n_tokens_neox': 5058, 'n_words': 1356, 'path': 'predicting_malignancy.ipynb', 'repo_name': 'mkim5517/machine-learning-pset-mkim5517', 'size': '15461'}