<jupyter_start><jupyter_text>모바일넷을 이용한 이미지분류(Image Classification using Mobilenet)성  민  석(<NAME>)--- 강의목차1&nbsp;&nbsp;필요한 라이브러리 및 옵션1.1&nbsp;&nbsp;기본 라이브러리(Library)1.2&nbsp;&nbsp;옵션(Option)2&nbsp;&nbsp;예제 - CIFAR1002.1&nbsp;&nbsp;CIFAR-100 데이터 불러오기2.2&nbsp;&nbsp;CIFAR10 데이터 형태 확인하기2.2.1&nbsp;&nbsp;Train 데이터셋2.2.2&nbsp;&nbsp;Validation 데이터셋2.2.3&nbsp;&nbsp;Test 데이터셋2.3&nbsp;&nbsp;데이터 시각화하기2.4&nbsp;&nbsp;모델링2.4.1&nbsp;&nbsp;ResNet50모델 생성2.4.2&nbsp;&nbsp;ResNet50 미세 조정2.4.3&nbsp;&nbsp;전체적인 모델 구성2.5&nbsp;&nbsp;모델 컴파일2.6&nbsp;&nbsp;모델 확인하기2.7&nbsp;&nbsp;모델 학습하기2.8&nbsp;&nbsp;모델 평가하기2.8.1&nbsp;&nbsp;Train / Validation Loss2.8.2&nbsp;&nbsp;Train / Validation Accuaracy2.8.3&nbsp;&nbsp;Train / Validation Top5 Accuracy2.9&nbsp;&nbsp;신경망 모델 검증하기2.10&nbsp;&nbsp;다음 예제에서는3&nbsp;&nbsp;참고  필요한 라이브러리 및 옵션 기본 라이브러리(Library)<jupyter_code>import os
import sys
import glob
import random
import warnings
import itertools
from tqdm import tqdm
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from collections import Counter
import cv2
from PIL import Image
from PIL import ImageFont
from PIL import ImageDraw
from IPython.display import SVG
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
import xml.etree.ElementTree as ET
from xml.etree.ElementTree import Element, ElementTree
import keras
import tensorflow as tf
from tensorflow.keras.datasets import mnist,cifar10,cifar100
from tensorflow.keras.preprocessing.image import load_img,img_to_array,ImageDataGenerator
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.models import Model,Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D, Conv2D,GlobalAveragePooling2D
from tensorflow.keras.optimizers import RMSprop, Adam
from tensorflow.keras.utils import to_categorical,plot_model
from tensorflow.keras.losses import categorical_crossentropy,binary_crossentropy
from tensorflow.keras.callbacks import Callback
from tensorflow.keras.metrics import top_k_categorical_accuracy
from tensorflow.python.client import device_lib<jupyter_output>Using TensorFlow backend.
/home/minsuk/anaconda3/envs/keras/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/minsuk/anaconda3/envs/keras/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/minsuk/anaconda3/envs/keras/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/minsuk/anaconda3/[...]<jupyter_text>옵션(Option)<jupyter_code>os.environ["CUDA_VISIBLE_DEVICES"]="0"

warnings.filterwarnings(action='ignore')
warnings.filterwarnings(action='default')

%matplotlib inline
print(device_lib.list_local_devices())
keras.backend.tensorflow_backend._get_available_gpus()<jupyter_output>[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 6251351141964846220
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 17933432099555749685
physical_device_desc: "device: XLA_CPU device"
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 7881329869
locality {
  bus_id: 1
  links {
  }
}
incarnation: 1496858449212004281
physical_device_desc: "device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1"
, name: "/device:XLA_GPU:0"
device_type: "XLA_GPU"
memory_limit: 17179869184
locality {
}
incarnation: 145465887796347891
physical_device_desc: "device: XLA_GPU device"
]
WARNING:tensorflow:From /home/minsuk/anaconda3/envs/keras/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /home/minsuk/anaconda3/envs/keras/lib/python3[...]<jupyter_text>--- 예제 - CIFAR100![](https://miro.medium.com/max/356/1*fTQtXyApxWPoW2vzSEk_Pw.png) 이 데이터 세트는 CIFAR-10과 똑같다. 단, 각각 600개의 이미지가 포함된 100개의 클래스가 있다. 학습용 영상은 500개, 시험용 영상은 클래스당 100개씩이다. CIFAR-100의 100개 클래스는 20개의 슈퍼 클래스로 분류된다. 각 이미지에는 "fine"클래스과 "coarse" 라벨이 함께 나온다.CIFAR-100의 클래스 목록은 다음과 같다.출처 : [The CIFAR-100 dataset document](https://www.cs.toronto.edu/~kriz/cifar.html)<jupyter_code># CIFAR-100 클래스
CIFAR100_CLASSES = sorted(['beaver', 'dolphin', 'otter', 'seal', 'whale',  # aquatic mammals
                           'aquarium' 'fish', 'flatfish', 'ray', 'shark', 'trout',  # fish
                           'orchids', 'poppies', 'roses', 'sunflowers', 'tulips', # flowers
                           'bottles', 'bowls', 'cans', 'cups', 'plates', # food containers
                           'apples', 'mushrooms', 'oranges', 'pears', 'sweet peppers', # fruit and vegetables
                           'clock', 'computer' 'keyboard', 'lamp', 'telephone', 'television', # household electrical devices
                           'bed', 'chair', 'couch', 'table', 'wardrobe', # household furniture
                           'bee', 'beetle', 'butterfly', 'caterpillar', 'cockroach', # insects
                           'bear', 'leopard', 'lion', 'tiger', 'wolf', # large carnivores
                           'bridge', 'castle', 'house', 'road', 'skyscraper', # large man-made outdoor things
                           'cloud', 'forest', 'mountain', 'plain', 'sea', # large natural outdoor scenes
                           'camel', 'cattle', 'chimpanzee', 'elephant', 'kangaroo', # large omnivores and herbivores
                           'fox', 'porcupine', 'possum', 'raccoon', 'skunk', # medium-sized mammals
                           'crab', 'lobster', 'snail', 'spider', 'worm', # non-insect invertebrates
                           'baby', 'boy', 'girl', 'man', 'woman', # people
                           'crocodile', 'dinosaur', 'lizard', 'snake', 'turtle', # reptiles
                           'hamster', 'mouse', 'rabbit', 'shrew', 'squirrel', # small mammals
                           'maple', 'oak', 'palm', 'pine', 'willow', # trees
                           'bicycle', 'bus', 'motorcycle', 'pickup truck', 'train', # vehicles 1
                           'lawn-mower', 'rocket', 'streetcar', 'tank', 'tractor' # vehicles 2
                          ])<jupyter_output><empty_output><jupyter_text>CIFAR-100 데이터 불러오기<jupyter_code># Keras에서 제공하는 CIFAR-100 데이터를 불러오는 함수 : cifar100.load_data()
(X_train, y_train), (X_test,y_test) = cifar100.load_data()
# Train / Validation 데이터 분리하기
# Train 데이터를 Train / Validation 으로 나누어줌으로써 Overfitting 여부를 파악
# X_train, X_valid = X_train[:40000], X_train[40000:]
# y_train, y_valid = y_train[:40000], y_train[40000:]
X_train,X_valid,y_train,y_valid = train_test_split(X_train,y_train,test_size=0.2,shuffle=True,random_state=0)
# Train / Test 데이터의 크기 확인
print('CIFAR-100 Train 데이터의 크기 : {}'.format(len(X_train)))
print('CIFAR-100 Validation 데이터의 크기 : {}'.format(len(X_valid)))
print('CIFAR-100 Test 데이터의 크기 : {}'.format(len(X_test)))<jupyter_output>CIFAR-100 Train 데이터의 크기 : 40000
CIFAR-100 Validation 데이터의 크기 : 10000
CIFAR-100 Test 데이터의 크기 : 10000<jupyter_text>CIFAR10 데이터 형태 확인하기  Train 데이터셋<jupyter_code>print("X_train Shape : ",X_train.shape) # 32*32짜리 크기의 RGB 이미지 40000개
print("y_train Shape : ",y_train.shape) # 각 이미지별 레이블 40000개<jupyter_output>X_train Shape :  (40000, 32, 32, 3)
y_train Shape :  (40000, 1)<jupyter_text>Validation 데이터셋<jupyter_code>print("X_val Shape : ",X_valid.shape) # 32*32짜리 크기의 RGB 이미지 10000개
print("y_val Shape : ",y_valid.shape) # 각 이미지별 레이블 10000개<jupyter_output>X_val Shape :  (10000, 32, 32, 3)
y_val Shape :  (10000, 1)<jupyter_text>Test 데이터셋<jupyter_code>print("X_train Shape : ",X_test.shape) # 32*32짜리 크기의 RGB 이미지 10000개
print("y_train Shape : ",y_test.shape) # 각 이미지별 레이블 10000개<jupyter_output>X_train Shape :  (10000, 32, 32, 3)
y_train Shape :  (10000, 1)<jupyter_text>데이터 시각화하기CIFAR100의 Train 데이터셋의 첫번재 데이터를 확인해보자<jupyter_code>idx = 0
X_train[idx] # 3채널 RGB 이미지 
plt.imshow(X_train[idx], interpolation='nearest')
plt.title('Label : {}'.format(CIFAR100_CLASSES[y_train[idx][0]]))
plt.show()<jupyter_output><empty_output><jupyter_text>조금 더 많은 CIFAR 데이터셋을 확인해보자<jupyter_code># 재연성을 위하여 랜덤시드 고정
np.random.seed(1234) 

# random 함수를 통해서 임의의 16개 데이터 가져오기
samples = np.random.randint(0,len(X_train)+1,size=16)

# MNIST를 그릴 Figure 준비
plt.figure(figsize=(12,8))

# 16개의 이미지 시각화
for count, n in enumerate(samples,start=1):
    plt.subplot(4, 4, count)
    plt.imshow(X_train[n], interpolation='nearest')
    label_name = "Label:" + str(CIFAR100_CLASSES[y_train[n][0]])
    plt.title(label_name)

plt.tight_layout()
plt.show()
# 데이터 크기 조정(Data Reshape)
X_train = X_train.reshape(X_train.shape[0],32,32,3)
X_valid = X_valid.reshape(X_valid.shape[0],32,32,3)
X_test = X_test.reshape(X_test.shape[0],32,32,3)

# 데이터 포맷 바꾸기
# 정수(int)인 데이터에서 실수(float)으로 변환
X_train = X_train.astype('float32')
X_valid = X_valid.astype('float32')
X_test = X_test.astype('float32')

# 데이터 정규화(Data Regularization)
# 이 과정을 통해서 추후 학습할 신경망이 조금 더 학습이 원할히 될 수 있게함
X_train = X_train / 255
X_valid = X_valid / 255
X_test = X_test / 255

# 원-핫 인코딩(One Hot Encoding)
# Keras의 to_categorical함수를 통해서 모든 Train 데이터의 레이블을 벡터화(Vectorize)
# ex) [3] -> [0 0 0 1 0 0 0 0 0 0]
y_train = to_categorical(y_train, 100) 
y_valid = to_categorical(y_valid,100)
y_test = to_categorical(y_test, 100)<jupyter_output><empty_output><jupyter_text>모델링 앞선 예제에서는 직접 CNN 모델을 구성하여 학습을 진행하였지만, 이번 예제에서는 VGGNet, ResNet, MobileNet과 같이 보편적으로 성능이 좋다고 알려져있는 CNN 모델을 사용해보자. Keras에서는 application 모듈에서 Pretrained된 모델을 사용할 수 있다. 이미지 분류에 사용할 수 있는 모델의 목록은 아래와 같다. - Xception- VGG16- VGG19- ResNet, ResNetV2- InceptionV3- InceptionResNetV2- MobileNet- MobileNetV2- DenseNet- NASNet출처 : [Keras Documentation](https://keras.io/applications/) 이번 예제에서는 **`ResNet50`을 사용**해보도록 하자.  ResNet50모델 생성<jupyter_code># CIFAR-10의 이미지 형태
INPUT_SHAPE = (32, 32, 3)

# VGG16 객체 생성
base_model = ResNet50(input_shape=INPUT_SHAPE,  # CIFAR-10의 이미지 형태
                      include_top=False,  # FC Layer를 제거
                      weights='imagenet')

# ResNet50의 모든 Layer를 Freezing
base_model.trainable = True<jupyter_output>/home/minsuk/anaconda3/envs/keras/lib/python3.6/site-packages/keras_applications/resnet50.py:265: UserWarning: The output shape of `ResNet50(include_top=False)` has been changed since Keras 2.2.0.
  warnings.warn('The output shape of `ResNet50(include_top=False)` '<jupyter_text>ResNet50 미세 조정<jupyter_code># ResNet의 상단 Layer를 학습가능여부
set_trainable = False

# res5c_branch2a, res5c_branch2b, res5c_branch2c 라는 이름의 Layer를 만나면 학습가능하게끔
for layer in tqdm(base_model.layers):
    if layer.name in ['res5a_branch2a','res5a_branch2b','res5a_branch2c',
                      'res5b_branch2a','res5b_branch2b','res5b_branch2c',
                      'res5c_branch2a','res5c_branch2b','res5c_branch2c']:
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False
layers = [(layer, layer.name, layer.trainable) for layer in base_model.layers]
pd.DataFrame(layers, columns=['Layer Type', 'Layer Name', 'Layer Trainable'])<jupyter_output><empty_output><jupyter_text>전체적인 모델 구성<jupyter_code>model = Sequential()

model.add(base_model) # ResNet50 추가
model.add(GlobalAveragePooling2D()) # Transfer Learning을 진행할때 항상 GAP Layer 추가
model.add(Dense(len(CIFAR100_CLASSES), activation='softmax')) # 분류할 FC Layer 추가<jupyter_output>WARNING:tensorflow:From /home/minsuk/anaconda3/envs/keras/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor<jupyter_text>모델 컴파일<jupyter_code>model.compile(loss=categorical_crossentropy, 
              optimizer=Adam(learning_rate=0.0001), 
              metrics=['acc','top_k_categorical_accuracy'])<jupyter_output><empty_output><jupyter_text>모델 확인하기<jupyter_code>model.summary()

plot_model(model, to_file='./img/model/cifar100_resnet_model.png', show_shapes=True)<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
resnet50 (Model)             (None, 1, 1, 2048)        23587712  
_________________________________________________________________
global_average_pooling2d (Gl (None, 2048)              0         
_________________________________________________________________
dense (Dense)                (None, 100)               204900    
=================================================================
Total params: 23,792,612
Trainable params: 15,180,900
Non-trainable params: 8,611,712
_________________________________________________________________<jupyter_text>모델 학습하기Keras의 `fit`의 메소드를 통해서 간단하게 학습가능하다. 이번 예제에서는 여기서는 10번의 epoch만으로 학습을 진행하도록 한다. 그리고 앞서 준비한 검증 데이터(Validation Set)을 통해서 신경망의 오버피팅 여부를 판단하도록 하자.<jupyter_code>EPOCHS = 10
BATCH_SIZE = 64
history = model.fit(X_train,  # 학습할 데이터
                    y_train,  # 학습할 레이블
                    epochs=EPOCHS,  # 전체 학습할 횟수
                    batch_size=BATCH_SIZE,  # 배치 사이즈
                    use_multiprocessing=True,
                    validation_data=(X_valid, y_valid) # 검증 데이터로 확인
                   )
model.save('./bin/cifar100_resnet50.h5')<jupyter_output><empty_output><jupyter_text>모델 평가하기<jupyter_code># Train 데이터로 평가하기
train_loss, train_acc, train_top5_acc= model.evaluate(X_train,y_train,verbose=0)
print('Train Loss : {}'.format(train_loss))
print('Train Accuracy : {}'.format(train_acc))
print('Train Top5 Accuracy : {}'.format(train_top5_acc))

# Validation 데이터로 평가하기
valid_loss, valid_acc , valid_top5_acc= model.evaluate(X_valid,y_valid,verbose=0)
print('Validation Loss : {}'.format(valid_loss))
print('Validation Accuracy : {}'.format(valid_acc))
print('Validation Top5 Accuracy : {}'.format(valid_top5_acc))<jupyter_output>Train Loss : 9.426516875839233
Train Accuracy : 0.009949999861419201
Train Top5 Accuracy : 0.04987499862909317
Validation Loss : 9.39263415222168
Validation Accuracy : 0.010499999858438969
Validation Top5 Accuracy : 0.051100000739097595<jupyter_text>Train / Validation Loss<jupyter_code># Train / Validation 데이터에 대해서 Loss 시각화
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1,len(loss)+1)

plt.plot(epochs,loss,label='Training Loss')
plt.plot(epochs,val_loss,label='Validation Loss')
plt.title('Training and Validation Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>Train / Validation Accuaracy<jupyter_code># Train / Validation 데이터에 대해서 Accuracy 시각화
acc = history.history['acc']
val_acc = history.history['val_acc']
epochs = range(1,len(acc)+1)

plt.plot(epochs,acc,label='Training Accuarcy')
plt.plot(epochs,val_acc,label='Validation Accuarcy')
plt.title('Training and Validation Accuarcy')
plt.xlabel('Epochs')
plt.ylabel('Accuarcy')
plt.legend()
plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>Train / Validation Top5 Accuracy<jupyter_code># Train / Validation 데이터에 대해서 Accuracy 시각화
acc = history.history['top_k_categorical_accuracy']
val_acc = history.history['val_top_k_categorical_accuracy']
epochs = range(1,len(loss)+1)

plt.plot(epochs,acc,label='Training Top5 Accuarcy')
plt.plot(epochs,val_acc,label='Validation Top5 Accuarcy')
plt.title('Training and Validation Top5 Accuarcy')
plt.xlabel('Epochs')
plt.ylabel('Accuarcy')
plt.legend()
plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>신경망 모델 검증하기<jupyter_code>test_loss, test_acc, test_top5_acc = model.evaluate(X_test,y_test,verbose=0)
print('Test Loss : {}'.format(test_loss))
print('Test Accuracy : {}'.format(test_acc))
print('Test Accuracy : {}'.format(test_top5_acc))<jupyter_output>Test Loss : 9.417670503234863
Test Accuracy : 0.009999999776482582
Test Accuracy : 0.05079999938607216<jupyter_text>RenNet으로 학습한 신경망이 과연 어떠한 이미지를 잘 못 예측했는지 확인해보자<jupyter_code># 재연성을 위하여 랜덤시드 고정
random.seed('intel')

# TEST 데이터 예측하기
predicted_result = model.predict(X_train)
predicted_labels = np.argmax(predicted_result, axis=1)

# TEST 데이터의 정답 가져오기
test_labels = np.argmax(y_test, axis=1)

# random 함수를 통해서 임의의 16개 데이터 가져오기
samples = random.choices(population=test_labels, k=16)

# CIFAR100를 그릴 Figure 준비
plt.figure(figsize=(12, 8))

# 16개의 이미지 시각화
for count, n in enumerate(samples, start=1):
    plt.subplot(4, 4, count)
    plt.imshow(X_test[n], interpolation='nearest')
    label = "Label:" + CIFAR100_CLASSES[test_labels[n]]
    pred = "Prediction:" + CIFAR100_CLASSES[predicted_labels[n]]
    plt.title(label+', '+pred)

plt.tight_layout()
plt.savefig('./img/result/cifar100_wrong_result.png')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13128252044142033, 'fraction_numerical': 0.05289038824403911, 'mean_word_length': 4.822237017310253, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 76, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '29972', 'license': 'MIT', 'n_tokens_mistral': 7996, 'n_tokens_neox': 8025, 'n_words': 1362, 'path': 'Lecture06 - CIFAR100.ipynb', 'repo_name': 'hoesung/intel-image-classification', 'size': '17494'}