<jupyter_start><jupyter_text>Criar um classificador binário de imagens utilizando Keras e Tensorflow.Para isso deixo ao teu critério escolheres 2 classes para construir o classificador.Como dataset para treino usa 70% do dataset, 20% para test e 10% para validação. Parafraseando este [artigo](https://https://repositorio-aberto.up.pt/bitstream/10216/129193/2/418655.pdf): "A validação em machine learning é o processo de avaliar um modelo treinado num conjunto dedados de teste de modo a estimar a sua capacidade de generalização em novos contextos. Assim,permite identificar situações como o sobreajuste (do inglês overfitting), na qual um modelo estádemasiado adaptado aos dados de treino. Geralmente os dados são divididos em três partes: treino,validação e teste. Pela mesma ordem, estes são usados para o treino do modelo, ajuste dos hiperparâmetros, e na avaliação imparcial do modelo final. Este método de separação é denominado devalidação cruzada holdout."<jupyter_code>pip install Pillow
# check Pillow version number
import PIL
print('Pillow Version:', PIL.__version__)
import tensorflow as tf
import numpy as np
import PIL.Image as PImage
import matplotlib.pyplot as plt

IMAGE_RES = 224  # resize das imagens para correrem na imagenet 224 linhas,224 colunas,3 canais
batch_size=4<jupyter_output><empty_output><jupyter_text>Load Mobilenet model<jupyter_code>model = tf.keras.applications.mobilenet_v2.MobileNetV2(weights='imagenet') # Rede pré-treinada, estrutura da MobileNet2 (tem mil classes)
model.summary()
labels_path = tf.keras.utils.get_file('ImageNetLabels.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt')
imagenet_labels = np.array(open(labels_path).read().splitlines())

image_path = tf.keras.utils.get_file('grace_hopper.jpg', 'https://storage.googleapis.com/download.tensorflow.org/example_images/grace_hopper.jpg')
image1 = PImage.open(image_path).resize((IMAGE_RES, IMAGE_RES))
image_path = tf.keras.utils.get_file('STOP.jpg', 'https://upload.wikimedia.org/wikipedia/commons/f/f9/STOP_sign.jpg')
image2 = PImage.open(image_path).resize((IMAGE_RES, IMAGE_RES))

plt.subplot(1,2,1)
plt.imshow(image1)
plt.axis('off')
_ = plt.title("Grace")

plt.subplot(1,2,2)
plt.imshow(image2)
plt.axis('off')
_ = plt.title("Stop sign")

plt.show()<jupyter_output>Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt
16384/10484 [==============================================] - 0s 0us/step
24576/10484 [======================================================================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/download.tensorflow.org/example_images/grace_hopper.jpg
65536/61306 [================================] - 0s 0us/step
73728/61306 [====================================] - 0s 0us/step
Downloading data from https://upload.wikimedia.org/wikipedia/commons/f/f9/STOP_sign.jpg
811008/806107 [==============================] - 0s 0us/step
819200/806107 [==============================] - 0s 0us/step<jupyter_text>Make a test batch<jupyter_code>x1 = np.array(image1) #matriz 4D ao invês de 3D,  dou a primeira img, pacote de imagens para a rede. 1º el. 1a img, 2ºel. 2a img 
x2 = np.array(image2) #batch o conj. das duas iamgens
print("x1.shape:", x1.shape)
print("x1.shape:", x1.shape)
x=np.zeros( (2,IMAGE_RES, IMAGE_RES, 3)) # 2 elementos
#x = x1[np.newaxis, ...]
x[0] = x1 # 1st batch item 
x[1] = x2 # 2nd batch item

print("batch shape", x.shape)
#x = x / 255.0
x = x / 255.0 # rede treinada com píxeis, entre -1 e 1 ... adjusts: pass values between [0, 255] to [-1,1], 3 canais destes píxeis
x = x * 2.0
x = x - 1.0<jupyter_output>x1.shape: (224, 224, 3)
x1.shape: (224, 224, 3)
batch shape (2, 224, 224, 3)<jupyter_text>Use the model to predict classes<jupyter_code>result = model.predict(x) # parameter: batch
print(result.shape) # cada um dos resultaados são mil probabilidades!
predicted_class1 = np.argmax(result[0], axis=-1) # class position most probable for img1
print("predicted_class1:", predicted_class1)
predicted_class2 = np.argmax(result[1], axis=-1)
print("predicted_class2:", predicted_class2)

predicted_class_name1 = imagenet_labels[predicted_class1+1]
print("predicted_class_name1:", predicted_class_name1)
predicted_class_name2= imagenet_labels[predicted_class2+1]
print("predicted_class_name2:", predicted_class_name2)<jupyter_output>(2, 1000)
predicted_class1: 652
predicted_class2: 919
predicted_class_name1: military uniform
predicted_class_name2: street sign<jupyter_text>Custom Image Classifier<jupyter_code>import fastai
fastai.__version__
from fastai import *
from fastai.vision import *
 from google.colab import drive
 drive.mount('/content/gdrive/')
# from google.colab import auth
# auth.authenticate_user()
# with open('/content/gdrive/MyDrive/') as f:
#   ('Hello Google Drive!')
# !cat /content/gdrive/MyDrive/

path=Path('/content/gdrive/Shareddrives/Datasets')
print(path)

import os
#os.chdir("/content/gdrive/MyDrive/ColabNotebooks")
os.chdir("/content/gdrive/Shareddrives/Datasets")
!ls

classes= ['Beagle','Pinscher']
# breed = bool(input('For Beagle choose 0, for Pinscher choose 1'))
#os
#path=!pwd

# def create_folder():
#   path = Path('/Beagle/')
for c in classes:
  print(c)
  verify_images(path/c,delete=True, max_size=400)
#!cd Beagle;ls
from fastai.vision.transform import get_transforms
from fastai.vision.data import ImageDataBunch
print(path)
np.random.seed(42)
#batch_size = 24
data = ImageDataBunch.from_folder(path, train='./', valid_pct=0.2,ds_tfms=get_transforms(),bs=4, size=224, num_workers=2).normalize(imagenet_stats)
data.classes<jupyter_output><empty_output><jupyter_text><jupyter_code>data.show_batch(rows=2, figsize=(7,8))<jupyter_output><empty_output><jupyter_text>Creating a new model<jupyter_code>learn = cnn_learner(data, models.resnet34, metrics=error_rate)
learn.fit_one_cycle(4)
#learn.save('stage-1')
learn.unfreeze()
learn.lr_find()
learn.fit_one_cycle(2, max_lr=slice(3e-5,3e-4))<jupyter_output><empty_output><jupyter_text>Interpretation<jupyter_code>from fastai.train import ClassificationInterpretation
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()<jupyter_output><empty_output><jupyter_text>Validation<jupyter_code>from sklearn.model_selection import cross_val_score
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import KFold
# pip install kaggle
# mkdir ~/.kaggle
# cp kaggle.json ~/.kaggle/
# ls -l
# #chmod 600 ~/.kaggle/kaggle.json
# #kaggle competi
# kfold=KFold(n_splits=5)
# from sklearn.datasets import load_iris
# from sklearn.linear_model import LogisticRegression

# iris = load_iris()
# logreg = LogisticRegression()
# print("Cross-validation scores:\n{}".format(cross_val_score(logreg, iris.data, iris.target, cv=kfold)))
# type(iris.data)
# scores = cross_val_scores(model. )

ds_train = tf.keras.preprocessing.image_dataset_from_directory(
    path,
    labels='inferred', # inferred from structure of te subfolders in alpabetical order
    label_mode='int', # most common but can be also categorical, binary
    # class_name = ['0', '1', '2', '3', ...]
    color_mode='rgb', #'grayscale', 
    batch_size= batch_size, 
    image_size = (IMAGE_RES, IMAGE_RES), # reshape if not in this size
    shuffle=True, # to obtain in randomize order
    seed=123, # is important set the seed, to obtain same validation and training set every time that we running!
    validation_split=0.2,
    subset="training",
)
ds_validation = tf.keras.preprocessing.image_dataset_from_directory(
    path,
    labels='inferred', # inferred from structure of te subfolders in alpabetical order
    label_mode='int', # most common but can be also categorical, binary
    # class_name = ['0', '1', '2', '3', ...]
    color_mode='grayscale', #'rgb', 
    batch_size= batch_size, 
    image_size = (IMAGE_RES, IMAGE_RES), # reshape if not in this size
    shuffle=True, # to obtain in randomize order
    seed=123, # is important set the seed, to obtain same validation and training set every time that we running!
    validation_split=0.1,
    subset="validation",
)
# def augment(x,y):
#   img = tf.image.random_brightness(x, max_delta=0.05)
#   return img, y
# ds_train = ds_train.map(augment)
# from tensorflow.keras.optimizers import Adam
# import keras
# import keras.utils
# from keras import utils as np_utils
# from keras.models import Sequential
# from keras.layers import Dense, Dropout, Activation
# import tensorflow as tf 


# # Custom Loops
# for epochs in range(10):
#   for x, y in ds_train:
#     # train here
#     pass

# model.compile(
#     optimizer=tf.keras.optimizers.Adam(),
#     loss=[
#           keras.losses.SparseCategoricalCrossentropy(from_logits=True),
#     ],
#     metrics=["accuracy"],
# )

# model.fit(ds_train, epochs=10, verbose=2)
class_names = ds_train.class_names
print(class_names)<jupyter_output><empty_output><jupyter_text>Visualize the data<jupyter_code>import matplotlib.pyplot as plt

plt.figure(figsize=(5, 5))
for images, labels in ds_train.take(1):
  for i in range(3):
    ax = plt.subplot(2, 2, i+1)
    plt.imshow(images[i].numpy().astype("uint8"))
    plt.title(class_names[labels[i]])
    plt.axis("off")
for image_batch, labels_batch in ds_train:
  print(image_batch.shape)
  print(labels_batch.shape)
  break<jupyter_output><empty_output><jupyter_text>Standardize the data<jupyter_code>normalization_layer = tf.keras.layers.Rescaling(1./255)
normalized_ds = ds_train.map(lambda x, y: (normalization_layer(x), y))
image_batch, labels_bath = next(iter(normalized_ds))
first_image = image_batch[0]
# Notice the pixel values are now in `[0,1]`.
print(np.min(first_image), np.max(first_image))<jupyter_output><empty_output><jupyter_text>Configure the dataset for performance<jupyter_code>AUTOTUNE = tf.data.AUTOTUNE

ds_train = ds_train.cache().prefetch(buffer_size=AUTOTUNE)
ds_validation  = ds_validation.cache().prefetch(buffer_size=AUTOTUNE)

num_classes=2 #confere?
model = tf.keras.Sequential([
  tf.keras.layers.Rescaling(1./255),
  tf.keras.layers.Conv2D(32, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Conv2D(32, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Conv2D(32, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(num_classes)
  ])
model.compile(
    optimizer="adam",
    loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy'])
model.fit(
    ds_train,
    validation_data=ds_validation,
    epochs=3
)
!pwd
!ls<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14403215253762033, 'fraction_numerical': 0.03075053743340499, 'mean_word_length': 5.8152866242038215, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '184860', 'license': 'MIT', 'n_tokens_mistral': 4036, 'n_tokens_neox': 3808, 'n_words': 831, 'path': 'Desafio.ipynb', 'repo_name': 'ManJ-PC/HelloCodeSchoolProject', 'size': '10699'}