<jupyter_start><jupyter_text>Cargamos las imágenes el Drive e importamos Tensorflow<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
!ls "/content/drive/MyDrive/Celulas"

#raiz
PATH = "/content/drive/My Drive"

#ipunt

INPATH = PATH + '/Celulas'



#checkpoints

CPATH = PATH + '/checkpointsCelulas'
# Reestablecer tf a la versión 2.4.1

!pip uninstall tensorflow -y
!pip install  tensorflow==2.4.1<jupyter_output><empty_output><jupyter_text>Importamos las librerías generales para el desarrollo<jupyter_code>%tensorflow_version 2.x
import tensorflow as tf
from tensorflow import keras

import numpy as np
import matplotlib.pyplot as plt

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import AveragePooling2D

print(tf.__version__)<jupyter_output><empty_output><jupyter_text>Cargamos el set de Datos y Preprocesado*   Defininmos el ImageDataGenerator para aumentar el dataset, con zoom, flip,rotación..*  Hacemos plot para compronbar como quedan las imágenes generadas<jupyter_code>import pathlib
data_train = pathlib.Path(INPATH + '/entrenamiento/') 
count = len(list(data_train.glob('*/*.tiff')))
print('Entrenamiento : ' + str(count))
data_test = pathlib.Path(INPATH + '/test/') 
count = len(list(data_test.glob('*/*.tiff')))
print('Test : ' + str(count))
data_train
from tensorflow.python.keras.preprocessing.image import ImageDataGenerator


batch_size = 32
img_height = 224
img_width = 224

train_datagen = ImageDataGenerator(
    rescale=1./255,
    horizontal_flip=True,
    vertical_flip=True,
    validation_split=0.1) # set validation split

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
    data_train,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    shuffle=True,
    class_mode='categorical',
    subset='training',
    seed=42) # set as training data

validation_generator = train_datagen.flow_from_directory(
    data_train, # same directory as training data
    target_size=(img_height, img_width),
    batch_size=batch_size,
    shuffle=True,
    class_mode='categorical',
    subset='validation',
    seed=42) # set as validation data

test_generator = test_datagen.flow_from_directory(INPATH + '/test',
                                                target_size=(224, 224),
                                                batch_size=1,
                                                shuffle=False,
                                             class_mode='categorical')


def plotImages(images_arr):
    fig, axes = plt.subplots(1, 5, figsize=(20,20))
    axes = axes.flatten()
    for img, ax in zip( images_arr, axes):
        ax.imshow(img)
    plt.tight_layout()
    plt.show()
    
    
augmented_images = [train_generator[1][0][0] for i in range(5)]
plotImages(augmented_images)<jupyter_output><empty_output><jupyter_text>Cargamos el modelo Xception* Usamos los pesos de imagenet* No incluimos la capa final, ya que debemos ajustarla a nuestro problema* Damos la forma de entrada de nuestros datos<jupyter_code>from tensorflow.keras.layers import Dropout, BatchNormalization
#from keras.applications.xception import Xception
from tensorflow.python.keras.applications.xception import Xception

baseModel = Xception(weights="imagenet", include_top=False,
	input_shape =  (224, 224, 3) )<jupyter_output><empty_output><jupyter_text>Fine Tuning* Ajustamos la cabeza de la red para que nos de una salida categórica de 4 elementos acorde con nuestro problema<jupyter_code>headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(256, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(4, activation="softmax")(headModel)
from tensorflow.python.keras.models import Model
model = Model(inputs=baseModel.input, outputs=headModel)
model.summary()
print("Number of layers in the base model: ", len(model.layers))<jupyter_output><empty_output><jupyter_text>Fine TuningCongelamos las primeras 40 capas del modelo, para utilizar el mapa de caracteristicas general que ha cargado de los pesos.<jupyter_code>##Solo
for layer in model.layers[:30]:
    layer.trainable = False

for layer in model.layers[:]:
  print(layer.trainable)<jupyter_output><empty_output><jupyter_text>Calculamos los pasos por epoca y validación<jupyter_code>steps_per_epoch = train_generator.n // batch_size
validation_steps = validation_generator.n // batch_size

print(steps_per_epoch)
print(validation_steps)<jupyter_output><empty_output><jupyter_text>Cargamos el modelo del checkpoint<jupyter_code>from tensorflow.python.keras.models import Sequential, load_model
from tensorflow.keras.callbacks import EarlyStopping
earlystopping = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=20)
optimizer = tf.keras.optimizers.Adam (lr=0.001)

checkpoint_filepath = CPATH + '/fulldata/'

model.compile(optimizer=optimizer,
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.load_weights(checkpoint_filepath)

model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_filepath,
    save_weights_only=True,
    monitor='val_accuracy',
    save_freq = 30,
    save_best_only=False)

reduce_lr = tf.keras.callbacks.LearningRateScheduler(lambda x: 1e-3 * 0.9 ** x)<jupyter_output><empty_output><jupyter_text>Opcional : Seguir con el entrenamiento<jupyter_code>history = model.fit(train_generator,
          steps_per_epoch=steps_per_epoch,
          epochs=1500,
          validation_data=validation_generator,
          validation_steps=validation_steps,
          callbacks=[reduce_lr,model_checkpoint_callback,earlystopping],verbose=1)<jupyter_output><empty_output><jupyter_text>Evaluamos el entrenamiento<jupyter_code>model.evaluate(x=validation_generator,
steps=validation_steps)<jupyter_output><empty_output><jupyter_text>Predicción de resultados<jupyter_code>test_predict = model.predict(test_generator, steps = test_generator.n // 1, verbose =1)
test_predict.shape
test_predict
predict = []

for i in test_predict:
  predict.append(int(np.argmax(i)))

predict = np.asarray(predict)
np.asarray(predict)<jupyter_output><empty_output><jupyter_text>Métricas de exactitud y Matriz de Confusión<jupyter_code># Obtenemos la tasa de acierto del modelo
from sklearn.metrics import accuracy_score

accuracy = accuracy_score(test_generator.classes, np.asarray(predict))
accuracy
test_generator.class_indices
# Representamos la matriz de confusión
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(test_generator.classes, predict)
plt.figure(figsize = (7,7))
sns.heatmap(cm, annot=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09545061906133026, 'fraction_numerical': 0.013245033112582781, 'mean_word_length': 5.20822162645219, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '334748', 'license': 'MIT', 'n_tokens_mistral': 2340, 'n_tokens_neox': 2251, 'n_words': 474, 'path': 'Xception_FullDataSet_CargaCheckPoint.ipynb', 'repo_name': 'satfail/AI-Reading-Materials', 'size': '6946'}