<jupyter_start><jupyter_text>Training  - Loading images  - Build a simple CNN model and perform training - Deployment of model (waiting for AWS work account to activate first)<jupyter_code># import matplotlib.pyplot as plt # not used at the moment
import PIL
import numpy as np
import os
import tensorflow as tf
from tensorflow.keras.preprocessing import image_dataset_from_directory
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
# path variables
PROJ_PATH = os.path.abspath(os.curdir)
PATH_TRAIN_DIR = os.path.join(PROJ_PATH, 'dataset\\train\\')
print(PROJ_PATH)<jupyter_output>d:\Work\Personal projects\teppen-deck-card-classification<jupyter_text>Set up parameters and load images to split into training and validation set<jupyter_code># load data from training folder, split 80% as training, 20% as validation
BATCH_SIZE = 64
IMG_SIZE = (160, 160)
EPOCHS = 10

train = image_dataset_from_directory(
    PATH_TRAIN_DIR, 
    labels='inferred', 
    shuffle=True, 
    batch_size=BATCH_SIZE, 
    image_size=IMG_SIZE,
    validation_split=0.2,
    subset='training',
    seed=23
    )
    
validation = image_dataset_from_directory(
    PATH_TRAIN_DIR, 
    labels='inferred', 
    shuffle=True, 
    batch_size=BATCH_SIZE, 
    image_size=IMG_SIZE,
    validation_split=0.2,
    subset='validation',
    seed=23
    )

num_of_classes = len(train.class_names)

# apply bufferred prefetching to improve training performance
AUTOTUNE = tf.data.AUTOTUNE
train = train.prefetch(buffer_size=AUTOTUNE)
validation = validation.prefetch(buffer_size=AUTOTUNE)<jupyter_output>Found 1365 files belonging to 13 classes.
Using 1092 files for training.
Found 1365 files belonging to 13 classes.
Using 273 files for validation.<jupyter_text>Build model and start training<jupyter_code># create convolutional neural network model
model = Sequential([
  layers.experimental.preprocessing.Rescaling(1./255, input_shape=(160, 160, 3)),
  layers.Conv2D(8, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dropout(0.2),
  layers.Dense(32, activation='relu'),
  layers.Dropout(0.2),
  layers.Dense(num_of_classes)
])

# compile model and train
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

history = model.fit(
  train,
  validation_data=validation,
  epochs=EPOCHS
)
score = model.evaluate(validation)
print('[validation] loss:', score[0], ' accuracy:', score[1])<jupyter_output>5/5 [==============================] - 1s 94ms/step - loss: 1.1235e-04 - accuracy: 1.0000
[validation] loss: 0.00011235338752157986  accuracy: 1.0<jupyter_text>Test model's performance by using the original 13 images used for data augmentation<jupyter_code>pip install pillow
from tensorflow.keras.preprocessing.image import load_img
PATH_TRAIN_DIR = PROJ_PATH + '\\dataset\\train\\'
PATH_IMG_DIR = PROJ_PATH + '\\dataset\\images\\'

labels = []
print('Loading training dataset...')
for index, directory in enumerate(os.listdir(PATH_TRAIN_DIR)):
    labels.append(directory)
print('Completed!')

def prediction(card_path):
    img = load_img(
        card_path, target_size=(160, 160)
    )
    #plt.figure()
    #plt.imshow(img)
    img_array = tf.keras.preprocessing.image.img_to_array(img)
    img_array = tf.expand_dims(img_array, 0) # Create a batch

    prediction = model.predict(img_array)
    print('class : ' + labels[np.argmax(prediction)])

print('Performing classification...')
for index, filename in enumerate(os.listdir(PATH_IMG_DIR)):
    print("*******************")
    print('Input image ' + filename)
    prediction(card_path = PATH_IMG_DIR + filename)
print('Completed!')<jupyter_output>Loading training dataset...
Completed!
Performing classification...
*******************
Input image CHUN-LI 001.png
class : CHUN-LI 001
*******************
Input image CHUN-LI 002.png
class : CHUN-LI 002
*******************
Input image CHUN-LI 003.png
class : CHUN-LI 003
*******************
Input image CHUN-LI 004.png
class : CHUN-LI 004
*******************
Input image CHUN-LI 005.png
class : CHUN-LI 005
*******************
Input image CHUN-LI 006.png
class : CHUN-LI 006
*******************
Input image CHUN-LI 007.png
class : CHUN-LI 007
*******************
Input image CHUN-LI 008.png
class : CHUN-LI 008
*******************
Input image CHUN-LI 009.png
class : CHUN-LI 009
*******************
Input image CHUN-LI 010.png
class : CHUN-LI 010
*******************
Input image CHUN-LI 011.png
class : CHUN-LI 011
*******************
Input image CHUN-LI 012.png
class : CHUN-LI 012
*******************
Input image CHUN-LI 013.png
class : CHUN-LI 013
Completed!
{'fraction_non_alphanumeric': 0.16240357287860333, 'fraction_numerical': 0.04141291108404385, 'mean_word_length': 4.972121212121212, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '181524', 'license': 'FSFAP', 'n_tokens_mistral': 1774, 'n_tokens_neox': 1615, 'n_words': 389, 'path': 'cards_classification.ipynb', 'repo_name': 'changyenliu92/teppen-deck-predict', 'size': '4926'}