<jupyter_start><jupyter_text>Invasive Species Caso 1:   Importacion de Librerias<jupyter_code>import cv2
import numpy as np
import pandas as pd
from sklearn.model_selection import KFold
from sklearn import metrics
import keras
from keras.models import Model
from keras.optimizers import Adam
from keras.applications.inception_v3 import InceptionV3
from keras.layers import Dense, Input, Flatten, Dropout, GlobalAveragePooling2D
from keras.layers.normalization import BatchNormalization
from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau<jupyter_output>Using TensorFlow backend.<jupyter_text>Carga de DatosSe usará un generador para la carga de datos  Se genera un dataframe con la ruta a las imagenes de entrenamiento y su categoria: 0 No invasora, 1 Invasora<jupyter_code># Se cargan los path a los archivos de entrenamiento
path_images = "../Data/train/"
path_labels = "../Data/train_labels.csv"

#se define una funcion para cargar los datos
def load_train(path_images, path_labels):
    #path_images: carpeta donde estan las imagenes de entreniento
    #path_labels: ubicacion del archivo csv con dos columnas: name - invasive (binario)
    train_set = pd.read_csv(path_labels)  #se carga el archivo csv con pandas
    train_label = np.array(train_set['invasive'].iloc[: ])  #se transforma a numpy array
    train_files = []
    for i in range(len(train_set)):
        train_files.append(path_images + str(int(train_set.iloc[i][0])) +'.jpg')
    train_set['name'] = train_files
    return train_files, train_set, train_label

train_files, train_set, train_label = load_train(path_images,path_labels)

train_set.head()
train_files[0:5]
train_label[0:5]<jupyter_output><empty_output><jupyter_text>Se cargan los datos de entrenamiento<jupyter_code>path = "../input/test/"

def load_test(path):
    test_set = pd.read_csv('../Data/sample_submission.csv')
    test_files = []
    for i in range(len(test_set)):
        test_files.append(path + str(int(test_set.iloc[i][0])) +'.jpg')
    return test_files, test_set

test_files, test_set = load_test(path)

test_set.head()<jupyter_output><empty_output><jupyter_text>Definición del ModeloSe usa transfer learning con un modelo pre entrenado Inception V3 que fue creado por Google en 2015 con un entrenamiento sobre 1.000 clases y entrenado sobre un millón de imágenes.Keras tiene un modelo inceptionV3 incorporado cuya entrada por defecto es 299x299<jupyter_code>img_height = 800
img_width = 800
img_channels = 3
img_dim = (img_height, img_width, img_channels)

#función para defenir el modelo. No es necesario pero es una buena práctica
def inceptionv3(img_dim=img_dim):
    input_tensor = Input(shape=img_dim)  #definición del input con las dimensiones de las imagenes
    base_model = InceptionV3(include_top=False,
                   weights='imagenet',
                   input_shape=img_dim)  #se carga el modelo pre entrenado inceptionV3 con los weigths de imagenet, se expluye la capa final
    bn = BatchNormalization()(input_tensor)  #aquí se hace el re escalado en lugar de las imagenes
    x = base_model(bn)  #se incorporan las imagenes normalizadas a modelo inception
    x = GlobalAveragePooling2D()(x)  #transforma los tensores en un vector
    x = Dropout(0.5)(x)  #capa droput para evitar overfitting
    output = Dense(1, activation='sigmoid')(x)  #se define la salida que es binaria según el problema a resolver
    model = Model(input_tensor, output)  #paso final en la definción del modelo en Keras
    return model

model = inceptionv3()  #Definición del modelo
model.summary()<jupyter_output>Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 800, 800, 3)       0         
_________________________________________________________________
batch_normalization_95 (Batc (None, 800, 800, 3)       12        
_________________________________________________________________
inception_v3 (Model)         (None, 23, 23, 2048)      21802784  
_________________________________________________________________
global_average_pooling2d_1 ( (None, 2048)              0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 2048)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 2049      
===========================================================[...]<jupyter_text>Entrenamiento del Modelo<jupyter_code>def train_model(model, batch_size, epochs, img_size, x, y, test, n_fold, kf):
    roc_auc = metrics.roc_auc_score
    preds_train = np.zeros(len(x), dtype = np.float)
    preds_test = np.zeros(len(test), dtype = np.float)
    train_scores = []; valid_scores = []

    i = 1

    for train_index, test_index in kf.split(x):
        x_train = x.iloc[train_index]; x_valid = x.iloc[test_index]
        y_train = y[train_index]; y_valid = y[test_index]
        #se realiza Data Augmentation en el generador de datos - es más eficiente
        def augment(src, choice):
            if choice == 0:
                # Rotate 90
                src = np.rot90(src, 1)
            if choice == 1:
                # flip vertically
                src = np.flipud(src)
            if choice == 2:
                # Rotate 180
                src = np.rot90(src, 2)
            if choice == 3:
                # flip horizontally
                src = np.fliplr(src)
            if choice == 4:
                # Rotate 90 counter-clockwise
                src = np.rot90(src, 3)
            if choice == 5:
                # Rotate 180 and flip horizontally
                src = np.rot90(src, 2)
                src = np.fliplr(src)
            return src
        #se define el generador de datos para que el modelo tome imagenes de manera continua según la cantidad de datos
        #el keras ya lo trae incorporado pero aquí se define manualmente para tener certeza del data augmentation
        def train_generator():
            while True:
                for start in range(0, len(x_train), batch_size):
                    x_batch = []
                    y_batch = []
                    end = min(start + batch_size, len(x_train))
                    train_batch = x_train[start:end]
                    for filepath, tag in train_batch.values:
                        img = cv2.imread(filepath)
                        img = cv2.resize(img, img_size)
                        img = augment(img, np.random.randint(6))
                        x_batch.append(img)
                        y_batch.append(tag)
                    x_batch = np.array(x_batch, np.float32) / 255.
                    y_batch = np.array(y_batch, np.uint8)
                    yield x_batch, y_batch

        def valid_generator():
            while True:
                for start in range(0, len(x_valid), batch_size):
                    x_batch = []
                    y_batch = []
                    end = min(start + batch_size, len(x_valid))
                    valid_batch = x_valid[start:end]
                    for filepath, tag in valid_batch.values:
                        img = cv2.imread(filepath)
                        img = cv2.resize(img, img_size)
                        img = augment(img, np.random.randint(6))
                        x_batch.append(img)
                        y_batch.append(tag)
                    x_batch = np.array(x_batch, np.float32) / 255.
                    y_batch = np.array(y_batch, np.uint8)
                    yield x_batch, y_batch

        def test_generator():
            while True:
                for start in range(0, len(test), batch_size):
                    x_batch = []
                    end = min(start + batch_size, len(test))
                    test_batch = test[start:end]
                    for filepath in test_batch:
                        img = cv2.imread(filepath)
                        img = cv2.resize(img, img_size)
                        x_batch.append(img)
                    x_batch = np.array(x_batch, np.float32) / 255.
                    yield x_batch

        callbacks = [EarlyStopping(monitor='val_loss', patience=3, verbose=1, min_delta=1e-4),
             ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=1, cooldown=1, 
                               verbose=1, min_lr=1e-7),
             ModelCheckpoint(filepath='../Saved_Model/model' + str(i) + '.hdf5', verbose=1,
                             save_best_only=True, save_weights_only=True, mode='auto')]

        train_steps = len(x_train) / batch_size
        valid_steps = len(x_valid) / batch_size
        test_steps = len(test) / batch_size
        
        model = model

        model.compile(optimizer=Adam(lr=1e-4), loss='binary_crossentropy', 
                      metrics = ['accuracy'])

        model.fit_generator(train_generator(), train_steps, epochs=epochs, verbose=1, 
                            callbacks=callbacks, validation_data=valid_generator(), 
                            validation_steps=valid_steps)

        model.load_weights(filepath='../Saved_Model/model' + str(i) + '.hdf5')

        print('Running validation predictions on fold {}'.format(i))
        preds_valid = model.predict_generator(generator=valid_generator(),
                                      steps=valid_steps, verbose=1)[:, 0]

        print('Running train predictions on fold {}'.format(i))
        preds_train = model.predict_generator(generator=train_generator(),
                                      steps=train_steps, verbose=1)[:, 0]

        valid_score = roc_auc(y_valid, preds_valid)
        train_score = roc_auc(y_train, preds_train)
        print('Val Score:{} for fold {}'.format(valid_score, i))
        print('Train Score: {} for fold {}'.format(train_score, i))

        valid_scores.append(valid_score)
        train_scores.append(train_score)
        print('Avg Train Score:{0:0.5f}, Val Score:{1:0.5f} after {2:0.5f} folds'.format
              (np.mean(train_scores), np.mean(valid_scores), i))

        print('Running test predictions with fold {}'.format(i))

        preds_test_fold = model.predict_generator(generator=test_generator(),
                                              steps=test_steps, verbose=1)[:, -1]

        preds_test += preds_test_fold

        print('\n\n')

        i += 1

        if i <= n_fold:
            print('Now beginning training for fold {}\n\n'.format(i))
        else:
            print('Finished training!')

    preds_test /= n_fold


    return preds_test
batch_size = 2
epochs = 5
n_fold = 2
img_size = (img_height, img_width)
kf = KFold(n_splits=n_fold, shuffle=True)
test_pred = train_model(model, batch_size, epochs, img_size, train_set, 
                        train_label, test_files, n_fold, kf)<jupyter_output>WARNING:tensorflow:From C:\Users\Usuario\Anaconda3\envs\GPU_TF2\lib\site-packages\tensorflow_core\python\ops\nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Epoch 1/5
574/573 [==============================] - 777s 1s/step - loss: 0.3983 - accuracy: 0.8317 - val_loss: 5.3964e-11 - val_accuracy: 0.9155

Epoch 00001: val_loss improved from inf to 0.00000, saving model to ../Saved_Model/model1.hdf5
Epoch 2/5
574/573 [==============================] - 726s 1s/step - loss: 0.1879 - accuracy: 0.9268 - val_loss: 1.3956e-13 - val_accuracy: 0.9686

Epoch 00002: ReduceLROnPlateau reducing learning rate to 9.999999747378752e-06.

Epoch 00002: val_loss improved from 0.00000 to 0.00000, saving model to ../Saved_Model/model1.hdf5
Epoch 3/5
574/573 [==============================] - 728s 1s/step - loss: 0.0943 - accuracy: 0.9730 - val_loss: [...]
{'fraction_non_alphanumeric': 0.10711304057438638, 'fraction_numerical': 0.02888629153447988, 'mean_word_length': 2.473180632067266, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '241329', 'license': 'MIT', 'n_tokens_mistral': 3881, 'n_tokens_neox': 3519, 'n_words': 917, 'path': 'Train_Code/Invasive_Species.ipynb', 'repo_name': 'efvillar/EFVO_Invasive_Species_Monitoring', 'size': '11978'}