<jupyter_start><jupyter_text>Detección de cancerPara este demo utilizaremos el breast-cancer-wisconsin-dataset disponible en https://www.kaggle.com/uciml/breast-cancer-wisconsin-data.      Importamos las librerias necesarias<jupyter_code># Manejo de datos
import pandas as pd

# Computacion cientifica 
import numpy as np 

# Funciones de machine learning y modelos de aprendizaje 
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import KFold   #For K-fold cross validation
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn import metrics

# Librerias de visualizacion
import matplotlib.pyplot as plt 
from matplotlib import style 
style.use('ggplot')
import seaborn as sns
sns.set(color_codes=True)<jupyter_output>c:\users\hp\anaconda3\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>1. Limpieza de datosEl primer paso es limpiar los datos y prepararlos para su siguiente analisis.<jupyter_code># Load the data
df = pd.read_csv('data.csv')

#Let's check the size of the dataframe
print('Shape of dataframe: ', df.shape)

#Let's visualize the data in matrix form 
df.head(50)
# Mostramos informacion de las variables.
print('Features: ', df.columns)
print('\n\n\n Type of features: ', df.info())<jupyter_output>Features:  Index(['id', 'diagnosis', 'radius_mean', 'texture_mean', 'perimeter_mean',
       'area_mean', 'smoothness_mean', 'compactness_mean', 'concavity_mean',
       'concave points_mean', 'symmetry_mean', 'fractal_dimension_mean',
       'radius_se', 'texture_se', 'perimeter_se', 'area_se', 'smoothness_se',
       'compactness_se', 'concavity_se', 'concave points_se', 'symmetry_se',
       'fractal_dimension_se', 'radius_worst', 'texture_worst',
       'perimeter_worst', 'area_worst', 'smoothness_worst',
       'compactness_worst', 'concavity_worst', 'concave points_worst',
       'symmetry_worst', 'fractal_dimension_worst', 'Unnamed: 32'],
      dtype='object')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 569 entries, 0 to 568
Data columns (total 33 columns):
id                         569 non-null int64
diagnosis                  569 non-null object
radius_mean                569 non-null float64
texture_mean               569 non-null float64
perimeter_mean             569[...]<jupyter_text>Analizamos el datasetAntes de limpiar los datos, debemos analizar las variables y determinar cuales son utiles para resolver nuestro problema. Por ejemplo, la columna "id" no nos sirve de nada.<jupyter_code># Quitar la columna id.
df = df.drop('id', axis = 1)
df.head()<jupyter_output><empty_output><jupyter_text>Encontramos los missing values en el dataset<jupyter_code># El siguiente comando nos dice si hay missing values (nans).
df.isnull().values.any()
df.head()
# Los eliminamos con el siguiente comando.
df = df.drop('Unnamed: 32', axis = 1)
df.head()
print(df.shape)
print(df.columns)<jupyter_output>(569, 31)
Index(['diagnosis', 'radius_mean', 'texture_mean', 'perimeter_mean',
       'area_mean', 'smoothness_mean', 'compactness_mean', 'concavity_mean',
       'concave points_mean', 'symmetry_mean', 'fractal_dimension_mean',
       'radius_se', 'texture_se', 'perimeter_se', 'area_se', 'smoothness_se',
       'compactness_se', 'concavity_se', 'concave points_se', 'symmetry_se',
       'fractal_dimension_se', 'radius_worst', 'texture_worst',
       'perimeter_worst', 'area_worst', 'smoothness_worst',
       'compactness_worst', 'concavity_worst', 'concave points_worst',
       'symmetry_worst', 'fractal_dimension_worst'],
      dtype='object')<jupyter_text>2. PreprocesamientoUna vez que nuestros datos han sido limpiados, podemos pasar a procesarlos para su exploracion.<jupyter_code># Separamos los datos de las etiquetas.
# Y 
y = df['diagnosis']
# X
x = df.drop('diagnosis', axis = 1)
#x = np.array(x)
print(x.shape, y.shape)
x.head()<jupyter_output>(569, 30) (569,)<jupyter_text>Estandarizacion La mayor parte de los modelos de Machine Learning asumen que los datos estan normalizados. De otro modo, afectamos la eficiencia estadistica y el modelo tiende a aprender los datos con las desviaciones estandar mas altas.<jupyter_code># Creamos un objeto escalar.
scaler = preprocessing.StandardScaler(copy = True, with_mean = True, with_std = True).fit(x)

# Transformamos los datos y los casteamos a un dataframe de pandas.
x = pd.DataFrame(scaler.transform(x))

### Datos normalizados de la matriz x.

# Renombramos las columnas. 
dict_ = {}
for a,b in zip(x.columns, df.drop('diagnosis', axis = 1).columns):
    dict_[a] = b
#print(dict_)
x = x.rename(columns = dict_)
x.head()<jupyter_output><empty_output><jupyter_text>One-hot encodingFijate que las etiquetas del tensor estan compuestas de variables tipo string que indican si un punto de datos es benigno o maligno con respecto a la distribucion de cancer. Debemos convertir las etiquetas a un valor numerico de tal manera que el modelo de clasificacion pueda leerlo.<jupyter_code>y_save = df["diagnosis"]
y = y_save.map({'M': 1, 'B':0})
print([[i,j] for i,j in zip(y, y_save)])
# {a:b for a,b in zip(range(4), range(5))}
#[[a,b] for a,b in zip(y, y_)]<jupyter_output>[[1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [0, 'B'], [0, 'B'], [0, 'B'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [0, 'B'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [1, 'M'], [0, 'B'], [1, 'M'], [0, 'B'], [0, 'B'], [0, 'B'], [0, 'B'], [0, 'B'], [1, 'M'], [1, 'M'], [0, 'B'], [1, 'M'], [1, 'M'], [0, 'B'], [0, 'B'], [0, 'B'], [0, 'B'], [1, 'M'], [0, 'B'], [1, 'M'], [1, 'M'], [0, 'B'], [0, 'B'], [0, 'B'], [0, 'B'], [1, 'M'], [0, 'B'], [1, 'M'], [1, 'M'], [0, 'B'], [1, 'M'], [0, 'B'], [1, 'M'], [1, 'M'], [0, 'B'], [0, 'B'], [0, 'B'], [1, 'M'], [1, 'M'], [0, 'B'], [1, 'M'], [1, 'M'], [1, 'M'], [0, 'B'], [0, 'B'], [0, 'B'], [1, 'M'], [0, 'B'], [0, 'B'], [1, 'M'], [1, 'M'], [0, 'B'], [0, 'B'], [0, 'B'], [1, 'M'],[...]<jupyter_text>3. Exploratory Data Analysis (EDA)El siguiente paso es analizar los datos para obtener informacion acerca de cual modelo podriamos aplicar.<jupyter_code># statistical description 
x.describe()
df.describe()<jupyter_output><empty_output><jupyter_text>3.1 Visualizacion univariablePara este paso vemos la distribucion de cada feature del problema.<jupyter_code>features_mean = list(df.columns[1:11])
# Dividimos el dataframe con respecto a las clases maligna y benigna.
dfM=x[y == 1]
dfB=x[y == 0]
print(dfM.shape, dfB.shape)
dfM.head()
# Select all columns
features_mean=list(x.columns)
print(features_mean, len(features_mean))

#Stack the data
plt.rcParams.update({'font.size': 8})
fig, axes = plt.subplots(nrows=10, ncols=3, figsize=(16,20))
axes = axes.ravel()
for idx, ax in enumerate(axes):
    ax.figure
    ax.hist([dfM[features_mean[idx]],dfB[features_mean[idx]]], alpha=0.5, stacked=True, normed = True, label=['M','B'], color=['r','g'])
    ax.legend(loc='upper right')
    ax.set_title(features_mean[idx])
plt.tight_layout()
plt.show()<jupyter_output>['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean', 'smoothness_mean', 'compactness_mean', 'concavity_mean', 'concave points_mean', 'symmetry_mean', 'fractal_dimension_mean', 'radius_se', 'texture_se', 'perimeter_se', 'area_se', 'smoothness_se', 'compactness_se', 'concavity_se', 'concave points_se', 'symmetry_se', 'fractal_dimension_se', 'radius_worst', 'texture_worst', 'perimeter_worst', 'area_worst', 'smoothness_worst', 'compactness_worst', 'concavity_worst', 'concave points_worst', 'symmetry_worst', 'fractal_dimension_worst'] 30<jupyter_text>Observaciones - Los valores medios del radio de la celda, el perímetro, el área, la compacidad, la concavidad y los puntos cóncavos se pueden usar en la clasificación del cáncer. Los valores más grandes de estos parámetros tienden a mostrar una correlación con tumores malignos.- Los valores medios de textura, suavidad, simetría o dimensión fractual no muestran una preferencia particular de un diagnóstico sobre el otro.   3.2 Visualizacion bivariable.En este paso, trazaremos la relación de una variable a otra variable usando una matriz de pares. Este diagrama nos dará la correlación de cada característica con otra.<jupyter_code># sns.pairplot(df.drop('diagnosis', axis = 1))
# plt.show()<jupyter_output>IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_data_rate_limit`.<jupyter_text>4. Seleccion del modelo.Primero dividamos los datos en train y test.<jupyter_code>x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 42)
print(x_train.shape, x_test.shape)<jupyter_output>(455, 30) (114, 30)<jupyter_text>Vamos a crear una función que se utilizará para múltiples algoritmos de clasificación.<jupyter_code># Generic function for making a classification model and accessing the performance. 
# From AnalyticsVidhya tutorial
def classification_model(model):
    #Fit the model:
    model.fit(x_train, y_train)
  
    #Make predictions on training set:
    predictions = model.predict(x_train)
  
    #Print accuracy
    accuracy = metrics.accuracy_score(predictions, y_train)
    print("Accuracy : %s" % "{0:.3%}".format(accuracy))

    #Perform k-fold cross-validation with 5 folds
    kf = KFold(x_train.shape[0], n_folds=5)
    error = []
    for train, test in kf:
        # Filter training data
        train_predictors = (x_train.iloc[train,:])

        # The target we're using to train the algorithm.
        train_target = y_train.iloc[train]

        # Training the algorithm using the predictors and target.
        model.fit(train_predictors, train_target)

        #Record error from each cross-validation run
        error.append(model.score(x_train.iloc[test,:], y_train.iloc[test]))

    print("Cross-Validation Score : %s" % "{0:.3%}".format(np.mean(error)))<jupyter_output><empty_output><jupyter_text>Logistic regression * Slightly good result<jupyter_code>logistic_regression = LogisticRegression()
classification_model(logistic_regression)<jupyter_output>Accuracy : 98.681%
Cross-Validation Score : 96.923%<jupyter_text>Decision trees Overfitting<jupyter_code>decision_trees = DecisionTreeClassifier()
classification_model(decision_trees)<jupyter_output>Accuracy : 100.000%
Cross-Validation Score : 93.407%<jupyter_text>Random forests Good classifier<jupyter_code>random_forests = RandomForestClassifier(n_estimators=500, min_samples_split=25, max_depth=7, max_features=2)
classification_model(random_forests)<jupyter_output>Accuracy : 98.681%
Cross-Validation Score : 94.286%<jupyter_text>Test set* Let's test the data in the test set<jupyter_code>print('Logistic regression: ', str(logistic_regression.score(x_test, y_test)*100)[:5] )
print('Decision trees: ', str(decision_trees.score(x_test, y_test)*100)[:5] )
print('Random forests: ', str(random_forests.score(x_test, y_test)*100)[:5] )<jupyter_output>Logistic regression:  97.36
Decision trees:  92.98
Random forests:  95.61
{'fraction_non_alphanumeric': 0.1504627664091025, 'fraction_numerical': 0.023605332427613146, 'mean_word_length': 5.380281690140845, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '462679', 'license': 'Apache-2.0', 'n_tokens_mistral': 4448, 'n_tokens_neox': 4243, 'n_words': 1191, 'path': 'Clase 2 - Clasificacion/Clasificacion de celulas cancerigenas/Deteccion de celulas cancerigenas.ipynb', 'repo_name': 'widslapaz/curso_data_science_version_2_emi', 'size': '11777'}